diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 9613db2ec..b275e9866 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -13,6 +13,7 @@ - [Cluster Autoscaler (kubeadm)](./topics/flavors/cluster-autoscaler.md) - [k3s](./topics/flavors/k3s.md) - [rke2](./topics/flavors/rke2.md) + - [vpcless (kubeadm)](./topics/flavors/vpcless.md) - [Etcd](./topics/etcd.md) - [Backups](./topics/backups.md) - [Multi-Tenancy](./topics/multi-tenancy.md) @@ -21,6 +22,7 @@ - [Data Disks](./topics/disks/data-disks.md) - [Machine Health Checks](./topics/health-checking.md) - [Autoscaling](./topics/autoscaling.md) + - [VPC](./topics/vpc.md) - [Development](./developers/development.md) - [Releasing](./developers/releasing.md) - [Reference](./reference/reference.md) diff --git a/docs/src/developers/development.md b/docs/src/developers/development.md index 6a766841c..64c5c32a7 100644 --- a/docs/src/developers/development.md +++ b/docs/src/developers/development.md @@ -170,7 +170,7 @@ This can then be used with `clusterctl` by adding the following to `~/.clusterct ``` providers: - name: local-linode - url: ${HOME}/cluster-api-provider-linode/infrastructure-linode/v0.0.0/infrastructure-components.yaml + url: ${HOME}/cluster-api-provider-linode/infrastructure-local-linode/v0.0.0/infrastructure-components.yaml type: InfrastructureProvider ``` diff --git a/docs/src/topics/flavors/vpcless.md b/docs/src/topics/flavors/vpcless.md new file mode 100644 index 000000000..8f03266ac --- /dev/null +++ b/docs/src/topics/flavors/vpcless.md @@ -0,0 +1,28 @@ +# VPCLess + +This flavor supports provisioning k8s clusters outside of VPC. It uses kubeadm for +setting up control plane and uses cilium with VXLAN for pod networking. + +## Specification +| Control Plane | CNI | Default OS | Installs ClusterClass | IPv4 | IPv6 | +|---------------|--------|--------------|-----------------------|------|------| +| Kubeadm | Cilium | Ubuntu 22.04 | No | Yes | No | +## Prerequisites +[Quickstart](../getting-started.md) completed + +## Notes +This flavor is identical to the default flavor with the exception that it provisions +k8s clusters without VPC. Since it runs outside of VPC, native routing is not +supported in this flavor and it uses VXLAN for pod to pod communication. + +## Usage +1. Generate cluster yaml + ```bash + clusterctl generate cluster test-cluster \ + --infrastructure linode-linode \ + --flavor vpcless > test-cluster.yaml + ``` +2. Apply cluster yaml + ```bash + kubectl apply -f test-cluster.yaml + ``` diff --git a/docs/src/topics/getting-started.md b/docs/src/topics/getting-started.md index 618f86b38..3106db3a8 100644 --- a/docs/src/topics/getting-started.md +++ b/docs/src/topics/getting-started.md @@ -33,7 +33,7 @@ For Regions and Images that do not yet support Akamai's cloud-init datasource CA to provision the node. If you are using a custom image ensure the [cloud_init](https://www.linode.com/docs/api/images/#image-create) flag is set correctly on it ``` ```admonish warning -By default, clusters are provisioned within VPC. For Regions which do not have [VPC support](https://www.linode.com/docs/products/networking/vpc/#availability) yet, use the VPCLess[TODO] flavor to have clusters provisioned. +By default, clusters are provisioned within VPC. For Regions which do not have [VPC support](https://www.linode.com/docs/products/networking/vpc/#availability) yet, use the [VPCLess](./flavors/vpcless.md) flavor to have clusters provisioned. ``` ## Register linode as an infrastructure provider diff --git a/docs/src/topics/vpc.md b/docs/src/topics/vpc.md new file mode 100644 index 000000000..ca5f55e06 --- /dev/null +++ b/docs/src/topics/vpc.md @@ -0,0 +1,51 @@ +# VPC + +This guide covers how [VPC](https://www.linode.com/docs/products/networking/vpc/) is used with CAPL clusters. By default, CAPL clusters are provisioned within VPC. + +## Default configuration +Each linode within a cluster gets provisioned with two interfaces: +1. eth0 (for public and nodebalancer traffic) +2. eth1 (connected to VPC, for pod-to-pod traffic) + +Key facts about VPC network configuration: +1. VPCs are provisioned with a private subnet 10.0.0.0/8. +2. All pod-to-pod communication happens over the VPC interface (eth1). +3. We assign a pod CIDR of range 10.192.0.0/10 for pod-to-pod communication. +3. By default, cilium is configured with [native routing](https://docs.cilium.io/en/stable/network/concepts/routing/#native-routing) +4. [Kubernetes host-scope IPAM mode](https://docs.cilium.io/en/stable/network/concepts/ipam/kubernetes/) is used to assign pod CIDRs to nodes. We run [linode CCM](https://github.com/linode/linode-cloud-controller-manager) with [route-controller enabled](https://github.com/linode/linode-cloud-controller-manager?tab=readme-ov-file#routes) which automatically adds/updates routes within VPC when pod cidrs are added/updated by k8s. This enables pod-to-pod traffic to be routable within the VPC. +5. kube-proxy is disabled by default. + +## How VPC is provisioned +A VPC is tied to a region. CAPL generates LinodeVPC manifest which contains the VPC name, region and subnet information. By defult, VPC name is set to cluster name but can be overwritten by specifying relevant environment variable. + +```yaml +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 +kind: LinodeVPC +metadata: + name: ${VPC_NAME:=${CLUSTER_NAME}} + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} +spec: + region: ${LINODE_REGION} + subnets: + - ipv4: 10.0.0.0/8 + label: default +``` + +Reference to LinodeVPC object is added to LinodeCluster object which then uses the specified VPC to provision resources. + +## Troubleshooting +### If pod-to-pod connectivity is failing +If a pod can't ping pod ips on different node, check and make sure pod CIDRs are added to ip_ranges of VPC interface. + +```sh +curl --header 'Authorization: Bearer $LINODE_API_TOKEN' -X GET https://api.linode.com/v4/linode/instances/${LINODEID}/configs | jq .data[0].interfaces[].ip_ranges +``` + +```admonish note +CIDR returned in the output of above command should match with the pod CIDR present in node's spec `k get node -o yaml | yq .spec.podCIDRs` +``` + +### Running cilium connectivity tests +One can also run cilium connectivity tests to make sure networking works fine within VPC. Follow the steps defined in [cilium e2e tests](https://docs.cilium.io/en/stable/contributing/testing/e2e/) guide to install cilium binary, set the KUBECONFIG variable and then run `cilium connectivity tests`. \ No newline at end of file diff --git a/templates/addons/cilium/cilium-ipv6.yaml b/templates/addons/cilium/cilium-ipv6.yaml index f55b8a617..ab6ef24b4 100644 --- a/templates/addons/cilium/cilium-ipv6.yaml +++ b/templates/addons/cilium/cilium-ipv6.yaml @@ -4,9 +4,10 @@ metadata: name: cilium-ipv6 spec: clusterSelector: - matchLabels: - cni: cilium - ipv6: "true" + matchExpressions: + - {key: ipv6, operator: In, values: ['true']} + - {key: cni, operator: In, values: ['cilium']} + - {key: vxlan, operator: DoesNotExist} repoURL: https://helm.cilium.io/ chartName: cilium namespace: kube-system diff --git a/templates/addons/cilium/cilium-vxlan.yaml b/templates/addons/cilium/cilium-vxlan.yaml new file mode 100644 index 000000000..70d15029a --- /dev/null +++ b/templates/addons/cilium/cilium-vxlan.yaml @@ -0,0 +1,30 @@ +apiVersion: addons.cluster.x-k8s.io/v1alpha1 +kind: HelmChartProxy +metadata: + name: cilium-vxlan +spec: + clusterSelector: + matchExpressions: + - {key: vxlan, operator: In, values: ['true']} + - {key: cni, operator: In, values: ['cilium']} + - {key: ipv6, operator: DoesNotExist} + repoURL: https://helm.cilium.io/ + chartName: cilium + namespace: kube-system + version: ${CILIUM_VERSION:=1.15.0} + options: + waitForJobs: true + wait: true + timeout: 5m + valuesTemplate: | + bgpControlPlane: + enabled: true + ipam: + mode: kubernetes + k8s: + requireIPv4PodCIDR: true + hubble: + relay: + enabled: true + ui: + enabled: true diff --git a/templates/addons/cilium/cilium.yaml b/templates/addons/cilium/cilium.yaml index 509089ffd..e7117470c 100644 --- a/templates/addons/cilium/cilium.yaml +++ b/templates/addons/cilium/cilium.yaml @@ -6,6 +6,7 @@ spec: clusterSelector: matchExpressions: - {key: ipv6, operator: DoesNotExist} + - {key: vxlan, operator: DoesNotExist} - {key: cni, operator: In, values: ['cilium']} repoURL: https://helm.cilium.io/ chartName: cilium diff --git a/templates/addons/cilium/kustomization.yaml b/templates/addons/cilium/kustomization.yaml index eafa3a9d3..bc81f2113 100644 --- a/templates/addons/cilium/kustomization.yaml +++ b/templates/addons/cilium/kustomization.yaml @@ -3,3 +3,4 @@ kind: Kustomization resources: - cilium.yaml - cilium-ipv6.yaml + - cilium-vxlan.yaml diff --git a/templates/addons/provider-linode/kustomization.yaml b/templates/addons/provider-linode/kustomization.yaml index 8adc23586..46c269ec5 100644 --- a/templates/addons/provider-linode/kustomization.yaml +++ b/templates/addons/provider-linode/kustomization.yaml @@ -2,3 +2,4 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - linode-ccm.yaml + - linode-ccm-vpcless.yaml diff --git a/templates/addons/provider-linode/linode-ccm-vpcless.yaml b/templates/addons/provider-linode/linode-ccm-vpcless.yaml new file mode 100644 index 000000000..8ff959a66 --- /dev/null +++ b/templates/addons/provider-linode/linode-ccm-vpcless.yaml @@ -0,0 +1,22 @@ +apiVersion: addons.cluster.x-k8s.io/v1alpha1 +kind: HelmChartProxy +metadata: + name: linode-cloud-controller-manager-vpcless +spec: + clusterSelector: + matchLabels: + ccm: linode + vpcless: "true" + repoURL: https://linode.github.io/linode-cloud-controller-manager/ + chartName: ccm-linode + namespace: kube-system + version: ${LINODE_CCM_VERSION:=v0.4.4} + options: + waitForJobs: true + wait: true + timeout: 5m + valuesTemplate: | + secretRef: + name: "linode-token-region" + image: + pullPolicy: IfNotPresent diff --git a/templates/addons/provider-linode/linode-ccm.yaml b/templates/addons/provider-linode/linode-ccm.yaml index 64e57c9de..2fe8d5464 100644 --- a/templates/addons/provider-linode/linode-ccm.yaml +++ b/templates/addons/provider-linode/linode-ccm.yaml @@ -4,8 +4,9 @@ metadata: name: linode-cloud-controller-manager spec: clusterSelector: - matchLabels: - ccm: linode + matchExpressions: + - {key: vpcless, operator: DoesNotExist} + - {key: ccm, operator: In, values: ['linode']} repoURL: https://linode.github.io/linode-cloud-controller-manager/ chartName: ccm-linode namespace: kube-system @@ -16,7 +17,7 @@ spec: timeout: 5m valuesTemplate: | routeController: - vpcName: ${VPC_NAME:=${CLUSTER_NAME}} + vpcName: {{ .InfraCluster.spec.vpcRef.name }} clusterCIDR: 10.0.0.0/8 configureCloudRoutes: true secretRef: diff --git a/templates/common-init-files/secret.yaml b/templates/common-init-files/secret.yaml index 8afd927e4..fdf30bf7d 100644 --- a/templates/common-init-files/secret.yaml +++ b/templates/common-init-files/secret.yaml @@ -38,5 +38,7 @@ stringData: modprobe overlay modprobe br_netfilter sysctl --system - IPADDR=$(ip a s eth1 |grep 'inet ' |cut -d' ' -f6|cut -d/ -f1) - sed -i "s/kubeletExtraArgs:/kubeletExtraArgs:\n node-ip: $IPADDR/g" /run/kubeadm/kubeadm.yaml + if [ -d "/sys/class/net/eth1" ]; then + IPADDR=$(ip a s eth1 |grep 'inet ' |cut -d' ' -f6|cut -d/ -f1) + sed -i "s/kubeletExtraArgs:/kubeletExtraArgs:\n node-ip: $IPADDR/g" /run/kubeadm/kubeadm.yaml + fi diff --git a/templates/flavors/dual-stack/kustomization.yaml b/templates/flavors/dual-stack/kustomization.yaml index 5ba4d119c..1537925cc 100644 --- a/templates/flavors/dual-stack/kustomization.yaml +++ b/templates/flavors/dual-stack/kustomization.yaml @@ -1,7 +1,8 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - - ../default + # VPCs don't support ipv6. Hence using vpcless flavor to run outside of VPC + - ../vpcless patches: - target: @@ -25,6 +26,13 @@ patches: cidrBlocks: - 10.96.0.0/12 - fd03::/108 + - target: + group: cluster.x-k8s.io + version: v1beta1 + kind: Cluster + patch: |- + - op: remove + path: /metadata/labels/vxlan - target: group: controlplane.cluster.x-k8s.io version: v1beta1 diff --git a/templates/flavors/vpcless/kustomization.yaml b/templates/flavors/vpcless/kustomization.yaml new file mode 100644 index 000000000..b23c84785 --- /dev/null +++ b/templates/flavors/vpcless/kustomization.yaml @@ -0,0 +1,40 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - ../default + +patches: + - target: + kind: LinodeVPC + patch: |- + $patch: delete + apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 + kind: LinodeVPC + metadata: + name: ${VPC_NAME:=${CLUSTER_NAME}} + - target: + group: infrastructure.cluster.x-k8s.io + version: v1alpha1 + kind: LinodeCluster + patch: |- + - op: remove + path: /spec/vpcRef + - target: + group: controlplane.cluster.x-k8s.io + version: v1beta1 + kind: KubeadmControlPlane + patch: |- + - op: remove + path: /spec/kubeadmConfigSpec/initConfiguration/skipPhases + - target: + group: cluster.x-k8s.io + version: v1beta1 + kind: Cluster + patch: |- + apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + metadata: + name: ${CLUSTER_NAME} + labels: + vxlan: "true" + vpcless: "true"