diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e697210..23ca80fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Breaking Change + +- For private clusters, where `network.vpcMode` is set to `private`, the subnets property has changed. Instead of previously being a list of CIDR strings the property now include a more complex object providing more configuration options. For migration steps see the "Upgrading to `v0.21.0`" section in the readme. + ### Added +- More configuration options when defining subnets to be created +- `controlPlane.subnetTags`, `bastion.subnetTags` and `machinePools[].subnetTags` to target specific subnets - Add icon to Chart.yaml +### Changed + +- Subnets are now specified on the `AWSCluster` resource by default rather than relying on CAPA code to default them. The same sizing as the CAPA default have been used. + ## [0.20.7] - 2023-01-12 ### Changed diff --git a/README.md b/README.md index d68e6e91..c0af83c2 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,147 @@ # cluster-aws `cluster-aws` is an app that helps create a CRs for a Cluster API AWS cluster for Giant Swarm platform. + +## Custom Subnet Layouts + +As of v0.21.0 it possible to specify more complex subnet layouts that allow using different sets of subnets for different grouping of machines. + +Subnet groupings can be defined by setting `.network.subnets`. For example, to have different subnets for control plane, worker and bastion nodes you might have something similar to the following: + +```yaml + +network: + availabilityZoneUsageLimit: 3 + vpcCIDR: 10.0.0.0/16 + subnets: + # Control plane nodes subnets + - cidrBlocks: + - cidr: 10.0.32.0/19 + availabilityZone: a + - cidr: 10.0.64.0/19 + availabilityZone: b + - cidr: 10.0.96.0/19 + availabilityZone: c + isPublic: false + tags: + subnet.giantswarm.io/role: control-plane + # Worker nodes subnets + - cidrBlocks: + - cidr: 10.0.128.0/19 + availabilityZone: a + - cidr: 10.0.160.0/19 + availabilityZone: b + - cidr: 10.0.192.0/19 + availabilityZone: c + isPublic: false + tags: + subnet.giantswarm.io/role: workers + # Bastion nodes subnets + - cidrBlocks: + - cidr: 10.0.0.0/24 + availabilityZone: a + - cidr: 10.0.1.0/24 + availabilityZone: b + - cidr: 10.0.2.0/24 + availabilityZone: c + isPublic: true + tags: + subnet.giantswarm.io/role: bastion +``` + +The desired subnet can then be targetted by using the `subnetTags` value to set the AWS tags to match on. For example: + +```yaml + +bastion: + subnetTags: + - subnet.giantswarm.io/role: bastion + +controlPlane: + subnetTags: + - subnet.giantswarm.io/role: control-plane + +machinePools: +- name: def00 + subnetTags: + - subnet.giantswarm.io/role: workers +``` + +### API-server ELB subnets + +Currently it's not possible to target the subnets used for the api-server ELB using the `subnetTags` approach like other resources. The ELB will be associated with the first grouping of subnets defined in the `network.subnets` list. + +If you want the ELB to use its own subnets then you can add a new grouping **first** in the list specifically for the api-server ELB. + +### VPC Endpoints subnets + +You can have VPC Endpoints target specific subnets by using the `subnet.giantswarm.io/endpoints: "true"` on those subnets. E.g. + +```yaml +# VPC endpoints +- cidrBlocks: + - cidr: 10.233.19.0/24 + availabilityZone: a + - cidr: 10.233.20.0/24 + availabilityZone: b + - cidr: 10.233.21.0/24 + availabilityZone: c + isPublic: true + tags: + subnet.giantswarm.io/role: attatchments + subnet.giantswarm.io/endpoints: "true" +``` + +If the `subnet.giantswarm.io/endpoints: "true"` tag isn't found on any subnets then it will default to using the first grouping of subnets. + +### Transit Gateway Attachment subnets + +> **Warning** +> Currently not possible, see [giantswarm/roadmap#1865](https://github.com/giantswarm/roadmap/issues/1865) + +You can have Transit Gateway Attachments target specific subnets by using the `subnet.giantswarm.io/tgw-attachments: "true"` on those subnets. E.g. + +```yaml +# TGW attachments +- cidrBlocks: + - cidr: 10.233.19.0/24 + availabilityZone: a + - cidr: 10.233.20.0/24 + availabilityZone: b + - cidr: 10.233.21.0/24 + availabilityZone: c + isPublic: true + tags: + subnet.giantswarm.io/role: attatchments + subnet.giantswarm.io/tgw-attachments: "true" +``` + +If the `subnet.giantswarm.io/tgw-attachments: "true"` tag isn't found on any subnets then it will default to using the first grouping of subnets. + +### Ingress subnets + +> **Warning** +> Currently not possible, see [giantswarm/roadmap#1866](https://github.com/giantswarm/roadmap/issues/1866) + +## Upgrade Migrations + +### Upgrading to `v0.21.0` + +If your cluster previously has `network.vpcMode` set to private you will need to make a small change to your values when upgrading to this version. + +If using the default list of subnets you will need to set the following in your values: + +```yaml +network: + subnets: + - cidrBlocks: + - cidr: 10.0.0.0/18 + availabilityZone: a + - cidr: 10.0.64.0/18 + availabilityZone: b + - cidr: 10.0.128.0/18 + availabilityZone: c + isPublic: false +``` + +If you've specified your own CIDR blocks previous you'll need to convert those strings to the block structure like above. Be aware to make sure the correct availability zone is specified for each CIDR block. diff --git a/helm/cluster-aws/templates/_aws_cluster.tpl b/helm/cluster-aws/templates/_aws_cluster.tpl index 8d1242ac..efff8afb 100644 --- a/helm/cluster-aws/templates/_aws_cluster.tpl +++ b/helm/cluster-aws/templates/_aws_cluster.tpl @@ -34,13 +34,20 @@ spec: vpc: availabilityZoneUsageLimit: {{ .Values.network.availabilityZoneUsageLimit }} cidrBlock: {{ .Values.network.vpcCIDR }} - {{- if (eq .Values.network.vpcMode "private") }} subnets: - {{- range $i, $subnet := .Values.network.subnets }} - - cidrBlock: "{{ $subnet.cidrBlock }}" - availabilityZone: "{{ include "aws-region" $ }}{{ add 97 $i | printf "%c" }}" - {{- end -}} - {{ end }} + {{- range $j, $subnet := .Values.network.subnets }} + {{- range $i, $cidr := $subnet.cidrBlocks }} + - cidrBlock: "{{ $cidr.cidr }}" + {{- if eq (len $cidr.availabilityZone) 1 }} + availabilityZone: "{{ include "aws-region" $ }}{{ $cidr.availabilityZone }}" + {{- else }} + availabilityZone: "{{ $cidr.availabilityZone }}" + {{- end }} + isPublic: {{ $subnet.isPublic | default false }} + tags: + {{- toYaml $subnet.tags | nindent 8 }} + {{- end }} + {{- end }} sshKeyName: ssh-key region: {{ include "aws-region" . }} {{ end }} diff --git a/helm/cluster-aws/templates/_bastion.tpl b/helm/cluster-aws/templates/_bastion.tpl index fda1e08e..9bcb3cd1 100644 --- a/helm/cluster-aws/templates/_bastion.tpl +++ b/helm/cluster-aws/templates/_bastion.tpl @@ -17,15 +17,21 @@ template: imageLookupOrg: "{{ .Values.flatcarAWSAccount }}" publicIP: {{ if eq .Values.network.vpcMode "private" }}false{{else}}true{{end}} sshKeyName: "" + uncompressedUserData: true subnet: filters: - - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/role{{else}}sigs.k8s.io/cluster-api-provider-aws/role{{end}} - values: - - {{ if eq .Values.network.vpcMode "private" }}private{{else}}public{{end}} - - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/{{else}}sigs.k8s.io/cluster-api-provider-aws/cluster/{{end}}{{ include "resource.default.name" $ }} - values: - - owned - uncompressedUserData: true + - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/role{{else}}sigs.k8s.io/cluster-api-provider-aws/role{{end}} + values: + - {{ if eq .Values.network.vpcMode "private" }}private{{else}}public{{end}} + - name: tag:{{ if eq .Values.network.vpcMode "private" }}github.com/giantswarm/aws-vpc-operator/{{else}}sigs.k8s.io/cluster-api-provider-aws/cluster/{{end}}{{ include "resource.default.name" $ }} + values: + - owned + - shared + {{- range $i, $tags := .Values.bastion.subnetTags }} + - name: tag:{{ keys $tags | first }} + values: + - {{ index $tags (keys $tags | first) }} + {{- end }} {{- end }} {{- define "bastion" }} diff --git a/helm/cluster-aws/templates/_control_plane.tpl b/helm/cluster-aws/templates/_control_plane.tpl index 7900530f..1ac7c19c 100644 --- a/helm/cluster-aws/templates/_control_plane.tpl +++ b/helm/cluster-aws/templates/_control_plane.tpl @@ -31,6 +31,17 @@ template: type: gp3 iamInstanceProfile: control-plane-{{ include "resource.default.name" $ }} sshKeyName: "" + subnet: + filters: + - name: tag:kubernetes.io/cluster/{{ include "resource.default.name" $ }} + values: + - shared + - owned + {{- range $i, $tags := .Values.controlPlane.subnetTags }} + - name: tag:{{ keys $tags | first }} + values: + - {{ index $tags (keys $tags | first) }} + {{- end }} {{- end }} {{- define "control-plane" }} diff --git a/helm/cluster-aws/templates/_machine_pools.tpl b/helm/cluster-aws/templates/_machine_pools.tpl index 6f74a318..f12d3374 100644 --- a/helm/cluster-aws/templates/_machine_pools.tpl +++ b/helm/cluster-aws/templates/_machine_pools.tpl @@ -39,14 +39,15 @@ spec: availabilityZones: {{ include "aws-availability-zones" . | nindent 2 }} subnets: - filters: - - name: tag:sigs.k8s.io/cluster-api-provider-aws/cluster/{{ include "resource.default.name" $ }} + - name: tag:kubernetes.io/cluster/{{ include "resource.default.name" $ }} values: + - shared - owned - - name: tag:sigs.k8s.io/cluster-api-provider-aws/role + {{- range $i, $tags := .subnetTags }} + - name: tag:{{ keys $tags | first }} values: - - private - - name: availabilityZone - values: {{ include "aws-availability-zones" . | nindent 6 }} + - {{ index $tags (keys $tags | first) }} + {{- end }} awsLaunchTemplate: {{- include "ami" $ | nindent 4 }} iamInstanceProfile: nodes-{{ .name }}-{{ include "resource.default.name" $ }} diff --git a/helm/cluster-aws/templates/_registry-secret.yaml b/helm/cluster-aws/templates/_registry-secret.yaml index c9ccfc85..8ba4f6ec 100644 --- a/helm/cluster-aws/templates/_registry-secret.yaml +++ b/helm/cluster-aws/templates/_registry-secret.yaml @@ -1,10 +1,12 @@ {{- define "registry-secret" -}} -{{- if and .Values.registry .Values.registry.configure -}} +{{- if .Values.registry -}} +{{- if .Values.registry.configure -}} apiVersion: v1 kind: Secret metadata: - name: {{ include "resource.default.name" $ }}-registry-configuration + name: {{ include "resource.default.name" $ }}-registry-configuration data: registry-config.toml: {{ tpl ($.Files.Get "files/etc/containerd/conf.d/registry-config.toml") $ | b64enc | quote }} {{- end -}} {{- end -}} +{{- end -}} diff --git a/helm/cluster-aws/values.schema.json b/helm/cluster-aws/values.schema.json index 0318eaf5..944f7d3d 100644 --- a/helm/cluster-aws/values.schema.json +++ b/helm/cluster-aws/values.schema.json @@ -37,6 +37,12 @@ }, "replicas": { "type": "integer" + }, + "subnetTags": { + "type": "array", + "items": { + "type": "object" + } } } }, @@ -63,6 +69,12 @@ }, "rootVolumeSizeGB": { "type": "integer" + }, + "subnetTags": { + "type": "array", + "items": { + "type": "object" + } } } }, @@ -150,6 +162,12 @@ }, "rootVolumeSizeGB": { "type": "integer" + }, + "subnetTags": { + "type": "array", + "items": { + "type": "object" + } } } } @@ -183,8 +201,25 @@ "items": { "type": "object", "properties": { - "cidrBlock": { - "type": "string" + "cidrBlocks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "cidr": { + "type": "string" + }, + "availabilityZone": { + "type": "string" + } + } + } + }, + "isPublic": { + "type": "boolean" + }, + "tags": { + "type": "object" } } } diff --git a/helm/cluster-aws/values.yaml b/helm/cluster-aws/values.yaml index 3364771f..8d17d9d3 100644 --- a/helm/cluster-aws/values.yaml +++ b/helm/cluster-aws/values.yaml @@ -46,15 +46,42 @@ network: # prefixListID is the ID of the Managed Prefix List to use when `mode` is set to `UserManaged`. prefixListID: "" + # subnets defines all the subnets for a cluster. subnets: - - cidrBlock: 10.0.0.0/18 - - cidrBlock: 10.0.64.0/18 - - cidrBlock: 10.0.128.0/18 + # The CAPA default subnet layout + - cidrBlocks: + - cidr: 10.0.0.0/20 + availabilityZone: a + - cidr: 10.0.16.0/20 + availabilityZone: b + - cidr: 10.0.32.0/20 + availabilityZone: c + isPublic: true + - cidrBlocks: + - cidr: 10.0.64.0/18 + availabilityZone: a + - cidr: 10.0.128.0/18 + availabilityZone: b + - cidr: 10.0.192.0/18 + availabilityZone: c + isPublic: false + + # # Our previous default for private clusters + # - cidrBlocks: + # - cidr: 10.0.0.0/18 + # availabilityZone: a + # - cidr: 10.0.64.0/18 + # availabilityZone: b + # - cidr: 10.0.128.0/18 + # availabilityZone: c + # isPublic: false bastion: enabled: true instanceType: t3.small replicas: 1 + # subnetTags: + # - subnet.giantswarm.io/role: bastion controlPlane: instanceType: m5.xlarge @@ -62,6 +89,8 @@ controlPlane: etcdVolumeSizeGB: 100 containerdVolumeSizeGB: 100 kubeletVolumeSizeGB: 100 + # subnetTags: + # - subnet.giantswarm.io/role: control-plane machinePools: - name: def00 # Name of node pool. @@ -76,6 +105,8 @@ machinePools: # - key: "" # value: "" # effect: "" # Valid effects are NoSchedule, PreferNoSchedule and NoExecute. + # subnetTags: + # - subnet.giantswarm.io/role: workers sshSSOPublicKey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM4cvZ01fLmO9cJbWUj7sfF+NhECgy+Cl0bazSrZX7sU vault-ca@vault.operations.giantswarm.io" flatcarAWSAccount: "075585003325"