Flexible subnet configuration (#196)

giantswarm · Jan 19, 2023 · db03686 · db03686
1 parent 1f4eb1b
commit db03686
Show file tree

Hide file tree

Showing 9 changed files with 272 additions and 25 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,10 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Breaking Change
+
+- For private clusters, where `network.vpcMode` is set to `private`, the subnets property has changed. Instead of previously being a list of CIDR strings the property now include a more complex object providing more configuration options. For migration steps see the "Upgrading to `v0.21.0`" section in the readme.
+
 ### Added
 
+- More configuration options when defining subnets to be created
+- `controlPlane.subnetTags`, `bastion.subnetTags` and `machinePools[].subnetTags` to target specific subnets
 - Add icon to Chart.yaml
 
+### Changed
+
+- Subnets are now specified on the `AWSCluster` resource by default rather than relying on CAPA code to default them. The same sizing as the CAPA default have been used.
+
 ## [0.20.7] - 2023-01-12
 
 ### Changed

diff --git a/README.md b/README.md
@@ -1,3 +1,147 @@
 # cluster-aws
 
 `cluster-aws` is an app that helps create a CRs for a Cluster API AWS cluster for Giant Swarm platform.
+
+## Custom Subnet Layouts
+
+As of v0.21.0 it possible to specify more complex subnet layouts that allow using different sets of subnets for different grouping of machines.
+
+Subnet groupings can be defined by setting `.network.subnets`. For example, to have different subnets for control plane, worker and bastion nodes you might have something similar to the following:
+
+```yaml
+
+network:
+  availabilityZoneUsageLimit: 3
+  vpcCIDR: 10.0.0.0/16
+  subnets:
+  # Control plane nodes subnets
+  - cidrBlocks:
+    - cidr: 10.0.32.0/19
+      availabilityZone: a
+    - cidr: 10.0.64.0/19
+      availabilityZone: b
+    - cidr: 10.0.96.0/19
+      availabilityZone: c
+    isPublic: false
+    tags:
+      subnet.giantswarm.io/role: control-plane
+  # Worker nodes subnets
+  - cidrBlocks:
+    - cidr: 10.0.128.0/19
+      availabilityZone: a
+    - cidr: 10.0.160.0/19
+      availabilityZone: b
+    - cidr: 10.0.192.0/19
+      availabilityZone: c
+    isPublic: false
+    tags:
+      subnet.giantswarm.io/role: workers
+  # Bastion nodes subnets
+  - cidrBlocks:
+    - cidr: 10.0.0.0/24
+      availabilityZone: a
+    - cidr: 10.0.1.0/24
+      availabilityZone: b
+    - cidr: 10.0.2.0/24
+      availabilityZone: c
+    isPublic: true
+    tags:
+      subnet.giantswarm.io/role: bastion
+```
+
+The desired subnet can then be targetted by using the `subnetTags` value to set the AWS tags to match on. For example:
+
+```yaml
+
+bastion:
+  subnetTags:
+    - subnet.giantswarm.io/role: bastion
+
+controlPlane:
+  subnetTags:
+    - subnet.giantswarm.io/role: control-plane
+
+machinePools:
+- name: def00
+  subnetTags:
+    - subnet.giantswarm.io/role: workers
+```
+
+### API-server ELB subnets
+
+Currently it's not possible to target the subnets used for the api-server ELB using the `subnetTags` approach like other resources. The ELB will be associated with the first grouping of subnets defined in the `network.subnets` list.
+
+If you want the ELB to use its own subnets then you can add a new grouping **first** in the list specifically for the api-server ELB.
+
+### VPC Endpoints subnets
+
+You can have VPC Endpoints target specific subnets by using the `subnet.giantswarm.io/endpoints: "true"` on those subnets. E.g.
+
+```yaml
+# VPC endpoints
+- cidrBlocks:
+  - cidr: 10.233.19.0/24
+    availabilityZone: a
+  - cidr: 10.233.20.0/24
+    availabilityZone: b
+  - cidr: 10.233.21.0/24
+    availabilityZone: c
+  isPublic: true
+  tags:
+    subnet.giantswarm.io/role: attatchments
+    subnet.giantswarm.io/endpoints: "true"
+```
+
+If the `subnet.giantswarm.io/endpoints: "true"` tag isn't found on any subnets then it will default to using the first grouping of subnets.
+
+### Transit Gateway Attachment subnets
+
+> **Warning**
+> Currently not possible, see [giantswarm/roadmap#1865](https://github.com/giantswarm/roadmap/issues/1865)
+
+You can have Transit Gateway Attachments target specific subnets by using the `subnet.giantswarm.io/tgw-attachments: "true"` on those subnets. E.g.
+
+```yaml
+# TGW attachments
+- cidrBlocks:
+  - cidr: 10.233.19.0/24
+    availabilityZone: a
+  - cidr: 10.233.20.0/24
+    availabilityZone: b
+  - cidr: 10.233.21.0/24
+    availabilityZone: c
+  isPublic: true
+  tags:
+    subnet.giantswarm.io/role: attatchments
+    subnet.giantswarm.io/tgw-attachments: "true"
+```
+
+If the `subnet.giantswarm.io/tgw-attachments: "true"` tag isn't found on any subnets then it will default to using the first grouping of subnets.
+
+### Ingress subnets
+
+> **Warning**
+> Currently not possible, see [giantswarm/roadmap#1866](https://github.com/giantswarm/roadmap/issues/1866)
+
+## Upgrade Migrations
+
+### Upgrading to `v0.21.0`
+
+If your cluster previously has `network.vpcMode` set to private you will need to make a small change to your values when upgrading to this version.
+
+If using the default list of subnets you will need to set the following in your values:
+
+```yaml
+network:
+  subnets:
+  - cidrBlocks:
+    - cidr: 10.0.0.0/18
+      availabilityZone: a
+    - cidr: 10.0.64.0/18
+      availabilityZone: b
+    - cidr: 10.0.128.0/18
+      availabilityZone: c
+    isPublic: false
+```
+
+If you've specified your own CIDR blocks previous you'll need to convert those strings to the block structure like above. Be aware to make sure the correct availability zone is specified for each CIDR block.
diff --git a/helm/cluster-aws/templates/_aws_cluster.tpl b/helm/cluster-aws/templates/_aws_cluster.tpl
@@ -34,13 +34,20 @@ spec:
     vpc:
       availabilityZoneUsageLimit: {{ .Values.network.availabilityZoneUsageLimit }}
       cidrBlock: {{ .Values.network.vpcCIDR }}
-    {{- if (eq .Values.network.vpcMode "private") }}
     subnets:
-    {{- range $i, $subnet := .Values.network.subnets }}
-    - cidrBlock: "{{ $subnet.cidrBlock }}"
-      availabilityZone: "{{ include "aws-region" $ }}{{ add 97 $i | printf "%c" }}"
-    {{- end -}}
-    {{ end }}
+    {{- range $j, $subnet := .Values.network.subnets }}
+    {{- range $i, $cidr := $subnet.cidrBlocks }}
+    - cidrBlock: "{{ $cidr.cidr }}"
+      {{- if eq (len $cidr.availabilityZone) 1 }}
+      availabilityZone: "{{ include "aws-region" $ }}{{ $cidr.availabilityZone }}"
+      {{- else }}
+      availabilityZone: "{{ $cidr.availabilityZone }}"
+      {{- end }}
+      isPublic: {{ $subnet.isPublic | default false }}
+      tags:
+        {{- toYaml $subnet.tags | nindent 8 }}
+    {{- end }}
+    {{- end }}
   sshKeyName: ssh-key
   region: {{ include "aws-region" . }}
 {{ end }}
diff --git a/helm/cluster-aws/templates/_bastion.tpl b/helm/cluster-aws/templates/_bastion.tpl
@@ -17,15 +17,21 @@ template:
     imageLookupOrg: "{{ .Values.flatcarAWSAccount }}"
     publicIP: {{ if eq .Values.network.vpcMode "private" }}false{{else}}true{{end}}
     sshKeyName: ""
+    uncompressedUserData: true
     subnet:
       filters:
-      - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/role{{else}}sigs.k8s.io/cluster-api-provider-aws/role{{end}}
-        values:
-        - {{ if eq .Values.network.vpcMode "private" }}private{{else}}public{{end}}
-      - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/{{else}}sigs.k8s.io/cluster-api-provider-aws/cluster/{{end}}{{ include "resource.default.name" $ }}
-        values:
-        - owned
-    uncompressedUserData: true
+        - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/role{{else}}sigs.k8s.io/cluster-api-provider-aws/role{{end}}
+          values:
+          - {{ if eq .Values.network.vpcMode "private" }}private{{else}}public{{end}}
+        - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/{{else}}sigs.k8s.io/cluster-api-provider-aws/cluster/{{end}}{{ include "resource.default.name" $ }}
+          values:
+          - owned
+          - shared
+        {{- range $i, $tags :=  .Values.bastion.subnetTags }}
+        - name: tag:{{ keys $tags | first }}
+          values:
+          - {{ index $tags (keys $tags | first) }}
+        {{- end }}
 {{- end }}
 
 {{- define "bastion" }}

diff --git a/helm/cluster-aws/templates/_control_plane.tpl b/helm/cluster-aws/templates/_control_plane.tpl
@@ -31,6 +31,17 @@ template:
       type: gp3
     iamInstanceProfile: control-plane-{{ include "resource.default.name" $ }}
     sshKeyName: ""
+    subnet:
+      filters:
+        - name: tag:kubernetes.io/cluster/{{ include "resource.default.name" $ }}
+          values:
+          - shared
+          - owned
+        {{- range $i, $tags :=  .Values.controlPlane.subnetTags }}
+        - name: tag:{{ keys $tags | first }}
+          values:
+          - {{ index $tags (keys $tags | first) }}
+        {{- end }}
 {{- end }}
 
 {{- define "control-plane" }}

diff --git a/helm/cluster-aws/templates/_machine_pools.tpl b/helm/cluster-aws/templates/_machine_pools.tpl
@@ -39,14 +39,15 @@ spec:
   availabilityZones: {{ include "aws-availability-zones" . | nindent 2 }}
   subnets:
   - filters:
-    - name: tag:sigs.k8s.io/cluster-api-provider-aws/cluster/{{ include "resource.default.name" $ }}
+    - name: tag:kubernetes.io/cluster/{{ include "resource.default.name" $ }}
       values:
+      - shared
       - owned
-    - name: tag:sigs.k8s.io/cluster-api-provider-aws/role
+    {{- range $i, $tags := .subnetTags }}
+    - name: tag:{{ keys $tags | first }}
       values:
-      - private
-    - name: availabilityZone
-      values: {{ include "aws-availability-zones" . | nindent 6 }}
+      - {{ index $tags (keys $tags | first) }}
+    {{- end }}
   awsLaunchTemplate:
     {{- include "ami" $ | nindent 4 }}
     iamInstanceProfile: nodes-{{ .name }}-{{ include "resource.default.name" $ }}

diff --git a/helm/cluster-aws/templates/_registry-secret.yaml b/helm/cluster-aws/templates/_registry-secret.yaml
@@ -1,10 +1,12 @@
 {{- define "registry-secret" -}}
-{{- if and .Values.registry .Values.registry.configure -}}
+{{- if .Values.registry -}}
+{{- if .Values.registry.configure -}}
 apiVersion: v1
 kind: Secret
 metadata:
-  name: {{ include "resource.default.name" $ }}-registry-configuration 
+  name: {{ include "resource.default.name" $ }}-registry-configuration
 data:
   registry-config.toml: {{ tpl ($.Files.Get "files/etc/containerd/conf.d/registry-config.toml") $ | b64enc | quote }}
 {{- end -}}
 {{- end -}}
+{{- end -}}
diff --git a/helm/cluster-aws/values.schema.json b/helm/cluster-aws/values.schema.json
@@ -37,6 +37,12 @@
                 },
                 "replicas": {
                     "type": "integer"
+                },
+                "subnetTags": {
+                    "type": "array",
+                    "items": {
+                        "type": "object"
+                    }
                 }
             }
         },
@@ -63,6 +69,12 @@
                 },
                 "rootVolumeSizeGB": {
                     "type": "integer"
+                },
+                "subnetTags": {
+                    "type": "array",
+                    "items": {
+                        "type": "object"
+                    }
                 }
             }
         },
@@ -150,6 +162,12 @@
                     },
                     "rootVolumeSizeGB": {
                         "type": "integer"
+                    },
+                    "subnetTags": {
+                        "type": "array",
+                        "items": {
+                            "type": "object"
+                        }
                     }
                 }
             }
@@ -183,8 +201,25 @@
                     "items": {
                         "type": "object",
                         "properties": {
-                            "cidrBlock": {
-                                "type": "string"
+                            "cidrBlocks": {
+                                "type": "array",
+                                "items": {
+                                    "type": "object",
+                                    "properties": {
+                                        "cidr": {
+                                            "type": "string"
+                                        },
+                                        "availabilityZone": {
+                                            "type": "string"
+                                        }
+                                    }
+                                }
+                            },
+                            "isPublic": {
+                                "type": "boolean"
+                            },
+                            "tags": {
+                                "type": "object"
                             }
                         }
                     }