From 9e859ec63a5e5960cd04d82dbd4b862e352aae93 Mon Sep 17 00:00:00 2001 From: Andrea Mazzotti Date: Fri, 9 Aug 2024 14:36:11 +0200 Subject: [PATCH 1/5] Document nm-configurator Signed-off-by: Andrea Mazzotti --- docs/machineregistration-reference.md | 2 +- docs/networking.md | 6 +++--- docs/troubleshooting-network.md | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/machineregistration-reference.md b/docs/machineregistration-reference.md index f9506d035..75f821764 100644 --- a/docs/machineregistration-reference.md +++ b/docs/machineregistration-reference.md @@ -52,7 +52,7 @@ Both yip and cloud-init syntax are supported. See the [Cloud Config Reference](c #### config.network Contains the Declarative Networking configuration, supporting integration with [CAPI IPAM Providers](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#ipam-provider). -Supports [nmstate](https://nmstate.io) syntax. +Supports [nm-configurator](https://github.com/suse-edge/nm-configurator) syntax. See the [Declarative Networking Reference](networking.md) for full information. #### config.elemental.registration diff --git a/docs/networking.md b/docs/networking.md index 8b2a9cfad..40ae48412 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -20,7 +20,7 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne - An IPAM Provider of your choice is installed on the Rancher management cluster. For example the [InCluster IPAM Provider](https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster). -- [nmstatectl](https://github.com/nmstate/nmstate/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use. +- [nmc](https://github.com/suse-edge/nm-configurator/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use. Elemental provided images already include these dependencies, this only applies when building custom images. ### How to install the CAPI IPAM Provider @@ -72,7 +72,7 @@ Note that this solution may eventually lead to conflicts with the applied CRDs a The `network` section of the `MachineRegistration` allows users to define: 1. A map of IPPool references. -1. A nmstate configuration template. +1. A nm-configurator [unified configuration](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) template. For example: @@ -129,7 +129,7 @@ config: enabled: false ``` -The snippet above is almost 1:1 [nmstate syntax](https://nmstate.io/examples.html#nmstate-state-examples), with the only exception of the `{inventory-ip}` placeholder. +The snippet above is almost 1:1 [nm-configurator syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations), with the only exception of the `{inventory-ip}` placeholder. During the installation or reset phases of Elemental machines, the `elemental-operator` will claim one IP Address from the referenced IP Pool, and substitute the `{inventory-ip}` placeholder with a real IP Address. ### Claimed IPAddresses diff --git a/docs/troubleshooting-network.md b/docs/troubleshooting-network.md index 59595522b..c05fb112e 100644 --- a/docs/troubleshooting-network.md +++ b/docs/troubleshooting-network.md @@ -15,7 +15,7 @@ Given the following sample registration: {RegistrationWithNetwork} -We can expect each Elemental Machine to be configured using the defined `nmstate` template. +We can expect each Elemental Machine to be configured using the defined `nm-configurator` `_all.yaml` template. At the very first boot, the `elemental-register` will try to contact the Rancher API to register a new `MachineInventory`. At this stage the machine's network is not configured and will default to DHCP. It is a requirement that the machine is able to contact the Rancher API in this setup, otherwise the registration can not take place. @@ -89,7 +89,7 @@ You will notice that the `MachineInventory` carries the same `network.config` as inventory-ip: 192.168.122.150 ``` -This `inventory-ip` will then be substituted in the `nmstate` config whenever `{inventory-ip}` has been defined. +This `inventory-ip` will then be substituted in the `nm-configurator` config whenever `{inventory-ip}` has been defined. Also note that the `MachineInventory` references and owns each `IPAddressClaim` associated with it. Each claim follow the predictable `$MachineIventoryName-$IPPoolRefKey` naming convention: `m-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c-inventory-ip`. These claims will follow the lifecycle of the `MachineInventory` object and be deleted on cascade, for example during the [reset workflow](./reset.md). @@ -98,7 +98,7 @@ If the `IPAddresses` can not be claimed, the `NetworkConfigReady` condition will ## On the machine side -During the installation phase, the `elemental-register` process running on the machine will receive the `nmstate` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nmstate` configuration: +During the installation phase, the `elemental-register` process running on the machine will receive the `nm-configurator` `_all.yaml` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nm-configurator` configuration: ```yaml config: @@ -129,10 +129,10 @@ config: table-id: 254 ``` -The `elemental-register` will then invoke `nmstatectl apply` to apply this configuration into the running system. +The `elemental-register` will then invoke `nmc generate` and `nmc apply` to apply this configuration into the running system. From this moment until reset, the machine will always use the applied configuration. -Also note that outside of installation and reset, `nmstate` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmstatectl` rather than the `nmstate` configuration itself. +Also note that outside of installation and reset, `nm-configurator` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmc` rather than the `nmc` configuration itself. For example on any running system, you will find a [yip](https://github.com/rancher/yip) configuration file (`/oem/94_custom.yaml`) to apply the desired `nmconnections`, for example: @@ -174,7 +174,7 @@ stages: [proxy] [user] - nmstate.interface.description=Main-NIC + nm-configurator.interface.description=Main-NIC encoding: "" ownerstring: "" if: '[ ! -f /run/elemental/recovery_mode ]' From 75121c2bb678ea17e1632c8dc4fc2a9c1cfa5bf7 Mon Sep 17 00:00:00 2001 From: Andrea Mazzotti Date: Tue, 13 Aug 2024 14:39:39 +0200 Subject: [PATCH 2/5] Add network Configurators section Signed-off-by: Andrea Mazzotti --- docs/machineregistration-reference.md | 7 +- docs/networking.md | 53 +++++++++++- examples/network/machineregistration-nmc.yaml | 84 +++++++++++++++++++ .../machineregistration-nmconnections.yaml | 65 ++++++++++++++ .../network/machineregistration-nmstate.yaml | 84 +++++++++++++++++++ examples/network/machineregistration.yaml | 1 + 6 files changed, 290 insertions(+), 4 deletions(-) create mode 100644 examples/network/machineregistration-nmc.yaml create mode 100644 examples/network/machineregistration-nmconnections.yaml create mode 100644 examples/network/machineregistration-nmstate.yaml diff --git a/docs/machineregistration-reference.md b/docs/machineregistration-reference.md index 75f821764..c6b565ec2 100644 --- a/docs/machineregistration-reference.md +++ b/docs/machineregistration-reference.md @@ -52,9 +52,14 @@ Both yip and cloud-init syntax are supported. See the [Cloud Config Reference](c #### config.network Contains the Declarative Networking configuration, supporting integration with [CAPI IPAM Providers](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#ipam-provider). -Supports [nm-configurator](https://github.com/suse-edge/nm-configurator) syntax. See the [Declarative Networking Reference](networking.md) for full information. +| Key | Type | Default value | Description | +|-------------------|-----------|---------------|---------------------------------------------------------------------------------------------------------------| +| configurator | string | nmc | The network configurator type to use (`nmc`, `nmstate`, or `nmconnections`) | +| ipAddresses | objRefMap | empty | A map of `IPPool` references. Map keys can be used for IPAddress substitution in the network config template. | +| config | obj | empty | The network config template. Syntax varies depending on the `configurator` in use. | + #### config.elemental.registration Contains the configuration used for the connection and the initial registration to the . diff --git a/docs/networking.md b/docs/networking.md index 40ae48412..7957dbb5f 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -8,6 +8,9 @@ title: '' import RegistrationWithNetwork from "!!raw-loader!@site/examples/network/machineregistration.yaml" +import RegistrationWithNetworkNmc from "!!raw-loader!@site/examples/network/machineregistration-nmc.yaml" +import RegistrationWithNetworkNmstate from "!!raw-loader!@site/examples/network/machineregistration-nmstate.yaml" +import RegistrationWithNetworkNmconnections from "!!raw-loader!@site/examples/network/machineregistration-nmconnections.yaml" ## Network configuration with Elemental @@ -20,8 +23,12 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne - An IPAM Provider of your choice is installed on the Rancher management cluster. For example the [InCluster IPAM Provider](https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster). -- [nmc](https://github.com/suse-edge/nm-configurator/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use. - Elemental provided images already include these dependencies, this only applies when building custom images. +- [NetworkManager](https://networkmanager.dev) needs to be installed on OS images and it can be directly configured using the `nmconnections` network [configurator](#configurators). + Already included in Elemental provided images. + +- (optionally) [nmc](https://github.com/suse-edge/nm-configurator/releases) can be used with `nmc` network [configurator](#configurators). + +- (optionally) [nmstate](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators). ### How to install the CAPI IPAM Provider @@ -72,7 +79,7 @@ Note that this solution may eventually lead to conflicts with the applied CRDs a The `network` section of the `MachineRegistration` allows users to define: 1. A map of IPPool references. -1. A nm-configurator [unified configuration](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) template. +1. A network config template (in this case `nmc` configurator is in use). For example: @@ -162,3 +169,43 @@ status: Whenever a `MachineInventory` is deleted, the default (DHCP) network configuration will be restored and the IPs assigned will be released. For more information and details on how troubleshoot issues, please consult the [documentation](./troubleshooting-network.md). + +## Configurators + +On the Elemental machine, `elemental-register` can configure the `NetworkManager` in different ways. +The configurator in use is defined in the [MachineRegistration.spec.network](./machineregistration-reference.md#confignetwork): + +- [nmc](https://github.com/suse-edge/nm-configurator) +- [nmstate](https://nmstate.io/) +- [nmconnections](https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-settings-keyfile.html) + + + + + +The `nmc` configurator uses the [nm-configurator unified syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) to generate NetworkManager's connection files. + +{RegistrationWithNetworkNmc} + + + + + +The `nmstate` configurator uses [nmstate syntax](https://nmstate.io/examples.html) to generate NetworkManager's connection files. +Note that [nmstatectl](https://github.com/nmstate/nmstate/releases) needs to be installed on the Elemental system to use this configurator. This is not included by default in Elemental images, but can be installed when building a [custom image](./custom-images.md). + +{RegistrationWithNetworkNmstate} + + + + + +The `nmconnections` configurator is the simplest option available and allows the user to directly write `nmconnection` files. +Defining these files for complex network setups may be challenging, but it's always possible to use [nmcli](https://networkmanager.dev/docs/api/latest/nmcli.html), or even [nmstate](https://nmstate.io), or [nm-configurator](https://github.com/suse-edge/nm-configurator), and use the generated `nmconnection` files as a template. +This configurator only needs `NetworkManager`, without any extra dependency. + +{RegistrationWithNetworkNmconnections} + + + + diff --git a/examples/network/machineregistration-nmc.yaml b/examples/network/machineregistration-nmc.yaml new file mode 100644 index 000000000..d6187dcf7 --- /dev/null +++ b/examples/network/machineregistration-nmc.yaml @@ -0,0 +1,84 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: m-${System Information/UUID} + config: + network: + configurator: nmc + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + dns-resolver: + config: + server: + - 192.168.122.1 + search: [] + routes: + config: + - destination: 0.0.0.0/0 + next-hop-interface: enp1s0 + next-hop-address: 192.168.122.1 + metric: 150 + table-id: 254 + - destination: 172.16.0.1/24 + next-hop-interface: enp8s0 + next-hop-address: 172.16.0.1 + metric: 150 + table-id: 254 + interfaces: + - name: enp1s0 + type: ethernet + description: Main-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{inventory-ip}" + prefix-length: 24 + ipv6: + enabled: false + - name: enp8s0 + type: ethernet + description: Secondary-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{secondary-ip}" + prefix-length: 24 + ipv6: + enabled: false diff --git a/examples/network/machineregistration-nmconnections.yaml b/examples/network/machineregistration-nmconnections.yaml new file mode 100644 index 000000000..792ec07dd --- /dev/null +++ b/examples/network/machineregistration-nmconnections.yaml @@ -0,0 +1,65 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: test-${System Information/UUID} + config: + network: + configurator: "nmconnections" + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + enp1s0: | + [connection] + id=Wired connection 1 + type=ethernet + interface-name=enp1s0 + [ipv4] + address1={inventory-ip}/24,192.168.1.1 + dns=192.168.122.1; + method=manual + route1=0.0.0.0/0,192.168.122.1 + [ipv6] + method=disabled + enp8s0: | + [connection] + id=Wired connection 2 + type=ethernet + interface-name=enp8s0 + [ipv4] + address1={secondary-ip}/24,172.16.0.1 + method=manual + route1=172.16.0.0/24,172.16.0.1,150 + [ipv6] + method=disabled diff --git a/examples/network/machineregistration-nmstate.yaml b/examples/network/machineregistration-nmstate.yaml new file mode 100644 index 000000000..faad1909d --- /dev/null +++ b/examples/network/machineregistration-nmstate.yaml @@ -0,0 +1,84 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: m-${System Information/UUID} + config: + network: + configurator: nmstate + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + dns-resolver: + config: + server: + - 192.168.122.1 + search: [] + routes: + config: + - destination: 0.0.0.0/0 + next-hop-interface: enp1s0 + next-hop-address: 192.168.122.1 + metric: 150 + table-id: 254 + - destination: 172.16.0.1/24 + next-hop-interface: enp8s0 + next-hop-address: 172.16.0.1 + metric: 150 + table-id: 254 + interfaces: + - name: enp1s0 + type: ethernet + description: Main-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{inventory-ip}" + prefix-length: 24 + ipv6: + enabled: false + - name: enp8s0 + type: ethernet + description: Secondary-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{secondary-ip}" + prefix-length: 24 + ipv6: + enabled: false diff --git a/examples/network/machineregistration.yaml b/examples/network/machineregistration.yaml index 867bdf417..2a52ce623 100644 --- a/examples/network/machineregistration.yaml +++ b/examples/network/machineregistration.yaml @@ -18,6 +18,7 @@ spec: machineName: m-${System Information/UUID} config: network: + configurator: nmc ipAddresses: inventory-ip: apiGroup: ipam.cluster.x-k8s.io From 285849f029677c15585c9078c24ff9d90f00b0f1 Mon Sep 17 00:00:00 2001 From: Andrea Mazzotti Date: Tue, 13 Aug 2024 14:56:40 +0200 Subject: [PATCH 3/5] Collapse configurator sample configs Signed-off-by: Andrea Mazzotti --- docs/networking.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/networking.md b/docs/networking.md index 7957dbb5f..adab02238 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -185,7 +185,10 @@ The configurator in use is defined in the [MachineRegistration.spec.network](./m The `nmc` configurator uses the [nm-configurator unified syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) to generate NetworkManager's connection files. -{RegistrationWithNetworkNmc} +
+ example MachineRegistration using nmc configurator +{RegistrationWithNetworkNmc} +
@@ -194,7 +197,10 @@ The `nmc` configurator uses the [nm-configurator unified syntax](https://github. The `nmstate` configurator uses [nmstate syntax](https://nmstate.io/examples.html) to generate NetworkManager's connection files. Note that [nmstatectl](https://github.com/nmstate/nmstate/releases) needs to be installed on the Elemental system to use this configurator. This is not included by default in Elemental images, but can be installed when building a [custom image](./custom-images.md). -{RegistrationWithNetworkNmstate} +
+ example MachineRegistration using nmstate configurator +{RegistrationWithNetworkNmstate} +
@@ -204,7 +210,10 @@ The `nmconnections` configurator is the simplest option available and allows the Defining these files for complex network setups may be challenging, but it's always possible to use [nmcli](https://networkmanager.dev/docs/api/latest/nmcli.html), or even [nmstate](https://nmstate.io), or [nm-configurator](https://github.com/suse-edge/nm-configurator), and use the generated `nmconnection` files as a template. This configurator only needs `NetworkManager`, without any extra dependency. -{RegistrationWithNetworkNmconnections} +
+ example MachineRegistration using nmconnections configurator +{RegistrationWithNetworkNmconnections} +
From 4dd6f535627870f6606f56bcbef149d014452bc6 Mon Sep 17 00:00:00 2001 From: Andrea Mazzotti Date: Tue, 13 Aug 2024 17:15:44 +0200 Subject: [PATCH 4/5] Formatting Signed-off-by: Andrea Mazzotti --- docs/networking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networking.md b/docs/networking.md index adab02238..ece72f634 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -170,7 +170,7 @@ Whenever a `MachineInventory` is deleted, the default (DHCP) network configurati For more information and details on how troubleshoot issues, please consult the [documentation](./troubleshooting-network.md). -## Configurators +### Configurators On the Elemental machine, `elemental-register` can configure the `NetworkManager` in different ways. The configurator in use is defined in the [MachineRegistration.spec.network](./machineregistration-reference.md#confignetwork): From 9536684024d05f230fcda72d8a6720ea082e2ffc Mon Sep 17 00:00:00 2001 From: Andrea Mazzotti Date: Tue, 13 Aug 2024 17:24:24 +0200 Subject: [PATCH 5/5] Mention nmstatectl Signed-off-by: Andrea Mazzotti --- docs/networking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networking.md b/docs/networking.md index ece72f634..0c7ec9a76 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -28,7 +28,7 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne - (optionally) [nmc](https://github.com/suse-edge/nm-configurator/releases) can be used with `nmc` network [configurator](#configurators). -- (optionally) [nmstate](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators). +- (optionally) [nmstatectl](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators). ### How to install the CAPI IPAM Provider