Add vpcless flavor

docs/src/SUMMARY.md

@@ -13,6 +13,7 @@
       - [Cluster Autoscaler (kubeadm)](./topics/flavors/cluster-autoscaler.md)
       - [k3s](./topics/flavors/k3s.md)
       - [rke2](./topics/flavors/rke2.md)
+      - [vpcless (kubeadm)](./topics/flavors/vpcless.md)
     - [Etcd](./topics/etcd.md)
     - [Backups](./topics/backups.md)
     - [Multi-Tenancy](./topics/multi-tenancy.md)
@@ -21,6 +22,7 @@
       - [Data Disks](./topics/disks/data-disks.md)
     - [Machine Health Checks](./topics/health-checking.md)
     - [Autoscaling](./topics/autoscaling.md)
+    - [VPC](./topics/vpc.md)
 - [Development](./developers/development.md)
     - [Releasing](./developers/releasing.md)
 - [Reference](./reference/reference.md)

docs/src/developers/development.md

@@ -170,7 +170,7 @@ This can then be used with `clusterctl` by adding the following to `~/.clusterct
 ```
 providers:
   - name: local-linode
-    url: ${HOME}/cluster-api-provider-linode/infrastructure-linode/v0.0.0/infrastructure-components.yaml
+    url: ${HOME}/cluster-api-provider-linode/infrastructure-local-linode/v0.0.0/infrastructure-components.yaml
     type: InfrastructureProvider
 ```
 

docs/src/topics/flavors/vpcless.md

@@ -0,0 +1,28 @@
+# VPCLess
+
+This flavor supports provisioning k8s clusters outside of VPC. It uses kubeadm for
+setting up control plane and uses cilium with VXLAN for pod networking.
+
+## Specification
+| Control Plane | CNI    | Default OS   | Installs ClusterClass | IPv4 | IPv6 |
+|---------------|--------|--------------|-----------------------|------|------|
+| Kubeadm       | Cilium | Ubuntu 22.04 | No                    | Yes  | No   |
+## Prerequisites
+[Quickstart](../getting-started.md) completed
+
+## Notes
+This flavor is identical to the default flavor with the exception that it provisions
+k8s clusters without VPC. Since it runs outside of VPC, native routing is not
+supported in this flavor and it uses VXLAN for pod to pod communication.
+
+## Usage
+1. Generate cluster yaml
+    ```bash
+    clusterctl generate cluster test-cluster \
+        --infrastructure linode-linode \
+        --flavor vpcless > test-cluster.yaml
+    ```
+2. Apply cluster yaml
+    ```bash
+    kubectl apply -f test-cluster.yaml
+    ```

docs/src/topics/getting-started.md

@@ -33,7 +33,7 @@ For Regions and Images that do not yet support Akamai's cloud-init datasource CA
 to provision the node. If you are using a custom image ensure the [cloud_init](https://www.linode.com/docs/api/images/#image-create) flag is set correctly on it
 ```
 ```admonish warning
-By default, clusters are provisioned within VPC. For Regions which do not have [VPC support](https://www.linode.com/docs/products/networking/vpc/#availability) yet, use the VPCLess[TODO] flavor to have clusters provisioned.
+By default, clusters are provisioned within VPC. For Regions which do not have [VPC support](https://www.linode.com/docs/products/networking/vpc/#availability) yet, use the [VPCLess](./flavors/vpcless.md) flavor to have clusters provisioned.
 ```
 
 ## Register linode as an infrastructure provider

docs/src/topics/vpc.md

@@ -0,0 +1,51 @@
+# VPC
+
+This guide covers how [VPC](https://www.linode.com/docs/products/networking/vpc/) is used with CAPL clusters. By default, CAPL clusters are provisioned within VPC.
+
+## Default configuration
+Each linode within a cluster gets provisioned with two interfaces:
+1. eth0 (for public and nodebalancer traffic)
+2. eth1 (connected to VPC, for pod-to-pod traffic)
+
+Key facts about VPC network configuration:
+1. VPCs are provisioned with a private subnet 10.0.0.0/8.
+2. All pod-to-pod communication happens over the VPC interface (eth1).
+3. We assign a pod CIDR of range 10.192.0.0/10 for pod-to-pod communication.
+3. By default, cilium is configured with [native routing](https://docs.cilium.io/en/stable/network/concepts/routing/#native-routing)
+4. [Kubernetes host-scope IPAM mode](https://docs.cilium.io/en/stable/network/concepts/ipam/kubernetes/) is used to assign pod CIDRs to nodes. We run [linode CCM](https://github.com/linode/linode-cloud-controller-manager) with [route-controller enabled](https://github.com/linode/linode-cloud-controller-manager?tab=readme-ov-file#routes) which automatically adds/updates routes within VPC when pod cidrs are added/updated by k8s. This enables pod-to-pod traffic to be routable within the VPC.
+5. kube-proxy is disabled by default.
+
+## How VPC is provisioned
+A VPC is tied to a region. CAPL generates LinodeVPC manifest which contains the VPC name, region and subnet information. By defult, VPC name is set to cluster name but can be overwritten by specifying relevant environment variable.
+
+```yaml
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+kind: LinodeVPC
+metadata:
+  name: ${VPC_NAME:=${CLUSTER_NAME}}
+  labels:
+    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
+spec:
+  region: ${LINODE_REGION}
+  subnets:
+    - ipv4: 10.0.0.0/8
+      label: default
+```
+
+Reference to LinodeVPC object is added to LinodeCluster object which then uses the specified VPC to provision resources.
+
+## Troubleshooting
+### If pod-to-pod connectivity is failing
+If a pod can't ping pod ips on different node, check and make sure pod CIDRs are added to ip_ranges of VPC interface.
+
+```sh
+curl --header 'Authorization: Bearer $LINODE_API_TOKEN' -X GET https://api.linode.com/v4/linode/instances/${LINODEID}/configs | jq .data[0].interfaces[].ip_ranges
+```
+
+```admonish note
+CIDR returned in the output of above command should match with the pod CIDR present in node's spec `k get node <nodename> -o yaml | yq .spec.podCIDRs`
+```
+
+### Running cilium connectivity tests
+One can also run cilium connectivity tests to make sure networking works fine within VPC. Follow the steps defined in [cilium e2e tests](https://docs.cilium.io/en/stable/contributing/testing/e2e/) guide to install cilium binary, set the KUBECONFIG variable and then run `cilium connectivity tests`.

templates/addons/cilium/cilium-ipv6.yaml

@@ -4,9 +4,10 @@ metadata:
   name: cilium-ipv6
 spec:
   clusterSelector:
-    matchLabels:
-      cni: cilium
-      ipv6: "true"
+    matchExpressions:
+      - {key: ipv6, operator: In, values: ['true']}
+      - {key: cni, operator: In, values: ['cilium']}
+      - {key: vxlan, operator: DoesNotExist}
   repoURL: https://helm.cilium.io/
   chartName: cilium
   namespace: kube-system

templates/addons/cilium/cilium-vxlan.yaml

@@ -0,0 +1,30 @@
+apiVersion: addons.cluster.x-k8s.io/v1alpha1
+kind: HelmChartProxy
+metadata:
+  name: cilium-vxlan
+spec:
+  clusterSelector:
+    matchExpressions:
+      - {key: vxlan, operator: In, values: ['true']}
+      - {key: cni, operator: In, values: ['cilium']}
+      - {key: ipv6, operator: DoesNotExist}
+  repoURL: https://helm.cilium.io/
+  chartName: cilium
+  namespace: kube-system
+  version: ${CILIUM_VERSION:=1.15.0}
+  options:
+    waitForJobs: true
+    wait: true
+    timeout: 5m
+  valuesTemplate: |
+    bgpControlPlane:
+      enabled: true
+    ipam:
+      mode: kubernetes
+    k8s:
+      requireIPv4PodCIDR: true
+    hubble:
+      relay:
+        enabled: true
+      ui:
+        enabled: true

templates/addons/cilium/cilium.yaml

@@ -6,6 +6,7 @@ spec:
   clusterSelector:
     matchExpressions:
       - {key: ipv6, operator: DoesNotExist}
+      - {key: vxlan, operator: DoesNotExist}
       - {key: cni, operator: In, values: ['cilium']}
   repoURL: https://helm.cilium.io/
   chartName: cilium

templates/addons/cilium/kustomization.yaml

@@ -3,3 +3,4 @@ kind: Kustomization
 resources:
     - cilium.yaml
     - cilium-ipv6.yaml
+    - cilium-vxlan.yaml

templates/addons/provider-linode/kustomization.yaml

@@ -2,3 +2,4 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
     - linode-ccm.yaml
+    - linode-ccm-vpcless.yaml

templates/addons/provider-linode/linode-ccm-vpcless.yaml

@@ -0,0 +1,22 @@
+apiVersion: addons.cluster.x-k8s.io/v1alpha1
+kind: HelmChartProxy
+metadata:
+  name: linode-cloud-controller-manager-vpcless
+spec:
+  clusterSelector:
+    matchLabels:
+      ccm: linode
+      vpcless: "true"
+  repoURL: https://linode.github.io/linode-cloud-controller-manager/
+  chartName: ccm-linode
+  namespace: kube-system
+  version: ${LINODE_CCM_VERSION:=v0.4.4}
+  options:
+    waitForJobs: true
+    wait: true
+    timeout: 5m
+  valuesTemplate: |
+    secretRef:
+      name: "linode-token-region"
+    image:
+      pullPolicy: IfNotPresent

templates/addons/provider-linode/linode-ccm.yaml

@@ -4,8 +4,9 @@ metadata:
   name: linode-cloud-controller-manager
 spec:
   clusterSelector:
-    matchLabels:
-      ccm: linode
+    matchExpressions:
+      - {key: vpcless, operator: DoesNotExist}
+      - {key: ccm, operator: In, values: ['linode']}
   repoURL: https://linode.github.io/linode-cloud-controller-manager/
   chartName: ccm-linode
   namespace: kube-system
@@ -16,7 +17,7 @@ spec:
     timeout: 5m
   valuesTemplate: |
     routeController:
-      vpcName: ${VPC_NAME:=${CLUSTER_NAME}}
+      vpcName: {{ .InfraCluster.spec.vpcRef.name }}
       clusterCIDR: 10.0.0.0/8
       configureCloudRoutes: true
     secretRef:

templates/common-init-files/secret.yaml

@@ -38,5 +38,7 @@ stringData:
     modprobe overlay
     modprobe br_netfilter
     sysctl --system
-    IPADDR=$(ip a s eth1 |grep 'inet ' |cut -d' ' -f6|cut -d/ -f1)
-    sed -i "s/kubeletExtraArgs:/kubeletExtraArgs:\n    node-ip: $IPADDR/g" /run/kubeadm/kubeadm.yaml
+    if [ -d "/sys/class/net/eth1" ]; then
+        IPADDR=$(ip a s eth1 |grep 'inet ' |cut -d' ' -f6|cut -d/ -f1)
+        sed -i "s/kubeletExtraArgs:/kubeletExtraArgs:\n    node-ip: $IPADDR/g" /run/kubeadm/kubeadm.yaml
+    fi

templates/flavors/dual-stack/kustomization.yaml

@@ -1,7 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - ../default
+  # VPCs don't support ipv6. Hence using vpcless flavor to run outside of VPC
+  - ../vpcless
 
 patches:
   - target:
@@ -25,6 +26,13 @@ patches:
             cidrBlocks:
               - 10.96.0.0/12
               - fd03::/108
+  - target:
+      group: cluster.x-k8s.io
+      version: v1beta1
+      kind: Cluster
+    patch: |-
+      - op: remove
+        path: /metadata/labels/vxlan
   - target:
       group: controlplane.cluster.x-k8s.io
       version: v1beta1

templates/flavors/vpcless/kustomization.yaml

@@ -0,0 +1,40 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - ../default
+
+patches:
+  - target:
+      kind: LinodeVPC
+    patch: |-
+      $patch: delete
+      apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+      kind: LinodeVPC
+      metadata:
+        name: ${VPC_NAME:=${CLUSTER_NAME}}
+  - target:
+      group: infrastructure.cluster.x-k8s.io
+      version: v1alpha1
+      kind: LinodeCluster
+    patch: |-
+      - op: remove
+        path: /spec/vpcRef
+  - target:
+      group: controlplane.cluster.x-k8s.io
+      version: v1beta1
+      kind: KubeadmControlPlane
+    patch: |-
+      - op: remove
+        path: /spec/kubeadmConfigSpec/initConfiguration/skipPhases
+  - target:
+      group: cluster.x-k8s.io
+      version: v1beta1
+      kind: Cluster
+    patch: |-
+      apiVersion: cluster.x-k8s.io/v1beta1
+      kind: Cluster
+      metadata:
+        name: ${CLUSTER_NAME}
+        labels:
+            vxlan: "true"
+            vpcless: "true"