Skip to content

Document Network Configurators #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 13, 2024
Merged

Document Network Configurators #363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9e859ec
Document nm-configurator
anmazzotti Aug 9, 2024
75121c2
Add network Configurators section
anmazzotti Aug 13, 2024
285849f
Collapse configurator sample configs
anmazzotti Aug 13, 2024
4dd6f53
Formatting
anmazzotti Aug 13, 2024
9536684
Mention nmstatectl
anmazzotti Aug 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/machineregistration-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,14 @@ Both yip and cloud-init syntax are supported. See the [Cloud Config Reference](c
#### config.network

Contains the Declarative Networking configuration, supporting integration with [CAPI IPAM Providers](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#ipam-provider).
Supports [nmstate](https://nmstate.io) syntax.
See the [Declarative Networking Reference](networking.md) for full information.

| Key | Type | Default value | Description |
|-------------------|-----------|---------------|---------------------------------------------------------------------------------------------------------------|
| configurator | string | nmc | The network configurator type to use (`nmc`, `nmstate`, or `nmconnections`) |
| ipAddresses | objRefMap | empty | A map of `IPPool` references. Map keys can be used for IPAddress substitution in the network config template. |
| config | obj | empty | The network config template. Syntax varies depending on the `configurator` in use. |

#### config.elemental.registration
Contains the configuration used for the connection and the initial registration to the <Vars name="elemental_operator_name" />.

Expand Down
64 changes: 60 additions & 4 deletions docs/networking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ title: ''
</head>

import RegistrationWithNetwork from "!!raw-loader!@site/examples/network/machineregistration.yaml"
import RegistrationWithNetworkNmc from "!!raw-loader!@site/examples/network/machineregistration-nmc.yaml"
import RegistrationWithNetworkNmstate from "!!raw-loader!@site/examples/network/machineregistration-nmstate.yaml"
import RegistrationWithNetworkNmconnections from "!!raw-loader!@site/examples/network/machineregistration-nmconnections.yaml"

## Network configuration with Elemental

Expand All @@ -20,8 +23,12 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne
- An IPAM Provider of your choice is installed on the Rancher management cluster.
For example the [InCluster IPAM Provider](https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster).

- [nmstatectl](https://github.com/nmstate/nmstate/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use.
Elemental provided images already include these dependencies, this only applies when building custom images.
- [NetworkManager](https://networkmanager.dev) needs to be installed on OS images and it can be directly configured using the `nmconnections` network [configurator](#configurators).
Already included in Elemental provided images.

- (optionally) [nmc](https://github.com/suse-edge/nm-configurator/releases) can be used with `nmc` network [configurator](#configurators).

- (optionally) [nmstatectl](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators).

### How to install the CAPI IPAM Provider

Expand Down Expand Up @@ -72,7 +79,7 @@ Note that this solution may eventually lead to conflicts with the applied CRDs a
The `network` section of the `MachineRegistration` allows users to define:

1. A map of IPPool references.
1. A nmstate configuration template.
1. A network config template (in this case `nmc` configurator is in use).

For example:

Expand Down Expand Up @@ -129,7 +136,7 @@ config:
enabled: false
```

The snippet above is almost 1:1 [nmstate syntax](https://nmstate.io/examples.html#nmstate-state-examples), with the only exception of the `{inventory-ip}` placeholder.
The snippet above is almost 1:1 [nm-configurator syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations), with the only exception of the `{inventory-ip}` placeholder.
During the installation or reset phases of Elemental machines, the `elemental-operator` will claim one IP Address from the referenced IP Pool, and substitute the `{inventory-ip}` placeholder with a real IP Address.

### Claimed IPAddresses
Expand Down Expand Up @@ -162,3 +169,52 @@ status:
Whenever a `MachineInventory` is deleted, the default (DHCP) network configuration will be restored and the IPs assigned will be released.

For more information and details on how troubleshoot issues, please consult the [documentation](./troubleshooting-network.md).

### Configurators

On the Elemental machine, `elemental-register` can configure the `NetworkManager` in different ways.
The configurator in use is defined in the [MachineRegistration.spec.network](./machineregistration-reference.md#confignetwork):

- [nmc](https://github.com/suse-edge/nm-configurator)
- [nmstate](https://nmstate.io/)
- [nmconnections](https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-settings-keyfile.html)

<Tabs>

<TabItem value="nmc" label="nmc" default>

The `nmc` configurator uses the [nm-configurator unified syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) to generate NetworkManager's connection files.

<details>
<summary>example MachineRegistration using nmc configurator</summary>
<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmc}</CodeBlock>
</details>

</TabItem>

<TabItem value="nmstate" label="nmstate" default>

The `nmstate` configurator uses [nmstate syntax](https://nmstate.io/examples.html) to generate NetworkManager's connection files.
Note that [nmstatectl](https://github.com/nmstate/nmstate/releases) needs to be installed on the Elemental system to use this configurator. This is not included by default in Elemental images, but can be installed when building a [custom image](./custom-images.md).

<details>
<summary>example MachineRegistration using nmstate configurator</summary>
<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmstate}</CodeBlock>
</details>

</TabItem>

<TabItem value="nmconnections" label="nmconnections" default>

The `nmconnections` configurator is the simplest option available and allows the user to directly write `nmconnection` files.
Defining these files for complex network setups may be challenging, but it's always possible to use [nmcli](https://networkmanager.dev/docs/api/latest/nmcli.html), or even [nmstate](https://nmstate.io), or [nm-configurator](https://github.com/suse-edge/nm-configurator), and use the generated `nmconnection` files as a template.
This configurator only needs `NetworkManager`, without any extra dependency.

<details>
<summary>example MachineRegistration using nmconnections configurator</summary>
<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmconnections}</CodeBlock>
</details>

</TabItem>

</Tabs>
12 changes: 6 additions & 6 deletions docs/troubleshooting-network.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Given the following sample registration:

<CodeBlock language="yaml" title="example MachineRegistration using Declarative Networking" showLineNumbers>{RegistrationWithNetwork}</CodeBlock>

We can expect each Elemental Machine to be configured using the defined `nmstate` template.
We can expect each Elemental Machine to be configured using the defined `nm-configurator` `_all.yaml` template.

At the very first boot, the `elemental-register` will try to contact the Rancher API to register a new `MachineInventory`.
At this stage the machine's network is not configured and will default to DHCP. It is a requirement that the machine is able to contact the Rancher API in this setup, otherwise the registration can not take place.
Expand Down Expand Up @@ -89,7 +89,7 @@ You will notice that the `MachineInventory` carries the same `network.config` as
inventory-ip: 192.168.122.150
```

This `inventory-ip` will then be substituted in the `nmstate` config whenever `{inventory-ip}` has been defined.
This `inventory-ip` will then be substituted in the `nm-configurator` config whenever `{inventory-ip}` has been defined.

Also note that the `MachineInventory` references and owns each `IPAddressClaim` associated with it. Each claim follow the predictable `$MachineIventoryName-$IPPoolRefKey` naming convention: `m-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c-inventory-ip`.
These claims will follow the lifecycle of the `MachineInventory` object and be deleted on cascade, for example during the [reset workflow](./reset.md).
Expand All @@ -98,7 +98,7 @@ If the `IPAddresses` can not be claimed, the `NetworkConfigReady` condition will

## On the machine side

During the installation phase, the `elemental-register` process running on the machine will receive the `nmstate` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nmstate` configuration:
During the installation phase, the `elemental-register` process running on the machine will receive the `nm-configurator` `_all.yaml` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nm-configurator` configuration:

```yaml
config:
Expand Down Expand Up @@ -129,10 +129,10 @@ config:
table-id: 254
```

The `elemental-register` will then invoke `nmstatectl apply` to apply this configuration into the running system.
The `elemental-register` will then invoke `nmc generate` and `nmc apply` to apply this configuration into the running system.
From this moment until reset, the machine will always use the applied configuration.

Also note that outside of installation and reset, `nmstate` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmstatectl` rather than the `nmstate` configuration itself.
Also note that outside of installation and reset, `nm-configurator` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmc` rather than the `nmc` configuration itself.

For example on any running system, you will find a [yip](https://github.com/rancher/yip) configuration file (`/oem/94_custom.yaml`) to apply the desired `nmconnections`, for example:

Expand Down Expand Up @@ -174,7 +174,7 @@ stages:
[proxy]

[user]
nmstate.interface.description=Main-NIC
nm-configurator.interface.description=Main-NIC
encoding: ""
ownerstring: ""
if: '[ ! -f /run/elemental/recovery_mode ]'
Expand Down
84 changes: 84 additions & 0 deletions examples/network/machineregistration-nmc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-inventory-pool
namespace: fleet-default
spec:
addresses:
- 192.168.122.150-192.168.122.200
prefix: 24
gateway: 192.168.122.1
---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-secondary-pool
namespace: fleet-default
spec:
addresses:
- 172.16.0.150-172.16.0.200
prefix: 24
gateway: 172.16.0.1
---
apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
name: fire-nodes
namespace: fleet-default
spec:
machineName: m-${System Information/UUID}
config:
network:
configurator: nmc
ipAddresses:
inventory-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-inventory-pool
secondary-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-secondary-pool
config:
dns-resolver:
config:
server:
- 192.168.122.1
search: []
routes:
config:
- destination: 0.0.0.0/0
next-hop-interface: enp1s0
next-hop-address: 192.168.122.1
metric: 150
table-id: 254
- destination: 172.16.0.1/24
next-hop-interface: enp8s0
next-hop-address: 172.16.0.1
metric: 150
table-id: 254
interfaces:
- name: enp1s0
type: ethernet
description: Main-NIC
state: up
ipv4:
enabled: true
dhcp: false
address:
- ip: "{inventory-ip}"
prefix-length: 24
ipv6:
enabled: false
- name: enp8s0
type: ethernet
description: Secondary-NIC
state: up
ipv4:
enabled: true
dhcp: false
address:
- ip: "{secondary-ip}"
prefix-length: 24
ipv6:
enabled: false
65 changes: 65 additions & 0 deletions examples/network/machineregistration-nmconnections.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-inventory-pool
namespace: fleet-default
spec:
addresses:
- 192.168.122.150-192.168.122.200
prefix: 24
gateway: 192.168.122.1
---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-secondary-pool
namespace: fleet-default
spec:
addresses:
- 172.16.0.150-172.16.0.200
prefix: 24
gateway: 172.16.0.1
---
apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
name: fire-nodes
namespace: fleet-default
spec:
machineName: test-${System Information/UUID}
config:
network:
configurator: "nmconnections"
ipAddresses:
inventory-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-inventory-pool
secondary-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-secondary-pool
config:
enp1s0: |
[connection]
id=Wired connection 1
type=ethernet
interface-name=enp1s0
[ipv4]
address1={inventory-ip}/24,192.168.1.1
dns=192.168.122.1;
method=manual
route1=0.0.0.0/0,192.168.122.1
[ipv6]
method=disabled
enp8s0: |
[connection]
id=Wired connection 2
type=ethernet
interface-name=enp8s0
[ipv4]
address1={secondary-ip}/24,172.16.0.1
method=manual
route1=172.16.0.0/24,172.16.0.1,150
[ipv6]
method=disabled
84 changes: 84 additions & 0 deletions examples/network/machineregistration-nmstate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-inventory-pool
namespace: fleet-default
spec:
addresses:
- 192.168.122.150-192.168.122.200
prefix: 24
gateway: 192.168.122.1
---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
name: elemental-secondary-pool
namespace: fleet-default
spec:
addresses:
- 172.16.0.150-172.16.0.200
prefix: 24
gateway: 172.16.0.1
---
apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
name: fire-nodes
namespace: fleet-default
spec:
machineName: m-${System Information/UUID}
config:
network:
configurator: nmstate
ipAddresses:
inventory-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-inventory-pool
secondary-ip:
apiGroup: ipam.cluster.x-k8s.io
kind: InClusterIPPool
name: elemental-secondary-pool
config:
dns-resolver:
config:
server:
- 192.168.122.1
search: []
routes:
config:
- destination: 0.0.0.0/0
next-hop-interface: enp1s0
next-hop-address: 192.168.122.1
metric: 150
table-id: 254
- destination: 172.16.0.1/24
next-hop-interface: enp8s0
next-hop-address: 172.16.0.1
metric: 150
table-id: 254
interfaces:
- name: enp1s0
type: ethernet
description: Main-NIC
state: up
ipv4:
enabled: true
dhcp: false
address:
- ip: "{inventory-ip}"
prefix-length: 24
ipv6:
enabled: false
- name: enp8s0
type: ethernet
description: Secondary-NIC
state: up
ipv4:
enabled: true
dhcp: false
address:
- ip: "{secondary-ip}"
prefix-length: 24
ipv6:
enabled: false
Loading
Loading