Skip to content

Commit 3bc1f07

Browse files
komer3amold1
komer3
and
amold1
authored
Adding an E2E test that runs a full CAPL cluster test (default, k3s, & rke2) on merging to main (#266)
 * adeodhar: add try catch block --------- Co-authored-by: Khaja Omer <[email protected]> Co-authored-by: Amol Deodhar <[email protected]>
1 parent 8a3977d commit 3bc1f07

File tree

53 files changed

+1302
-477
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1302
-477
lines changed

‎.github/workflows/build_test_ci.yml

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ jobs:
7474
needs: [go-build-test, docker-build]
7575
runs-on: ubuntu-latest
7676
if: github.event.pull_request.draft == false
77+
env:
78+
GITHUB_TOKEN: ${{ secrets.github_token }}
79+
LINODE_TOKEN: ${{ secrets.LINODE_TOKEN }}
7780
steps:
7881
- name: Harden Runner
7982
uses: step-security/harden-runner@v2
@@ -85,6 +88,7 @@ jobs:
8588
api.github.com:443
8689
github.com:443
8790
gcr.io:443
91+
ghcr.io:443
8892
proxy.golang.org:443
8993
sum.golang.org:443
9094
*.githubusercontent.com:443
@@ -103,6 +107,8 @@ jobs:
103107
cloud.tilt.dev:443
104108
kubernetes-sigs.github.io:443
105109
charts.jetstack.io:443
110+
helm.cilium.io:443
111+
linode.github.io:443
106112
107113
- uses: actions/checkout@v4
108114

@@ -117,11 +123,24 @@ jobs:
117123
with:
118124
key: docker-${{ runner.os }}-${{ hashFiles('go.sum') }}
119125

120-
- name: E2E test
126+
- name: Complete E2E Test
127+
if: github.ref == 'refs/heads/main'
121128
run: make e2etest
122129
env:
123-
GITHUB_TOKEN: ${{ secrets.github_token }}
124-
LINODE_TOKEN: ${{ secrets.LINODE_TOKEN }}
130+
E2E_FLAGS: '--assert-timeout 15m0s'
131+
INSTALL_K3S_PROVIDER: true
132+
INSTALL_RKE2_PROVIDER: true
133+
LINODE_REGION: us-sea
134+
LINODE_CONTROL_PLANE_MACHINE_TYPE: g6-standard-2
135+
LINODE_MACHINE_TYPE: g6-standard-2
136+
CLUSTERCTL_CONFIG: /home/runner/work/cluster-api-provider-linode/cluster-api-provider-linode/e2e/gha-clusterctl-config.yaml
137+
138+
139+
- name: Quick E2E Test
140+
if: github.ref != 'refs/heads/main'
141+
run: make e2etest
142+
env:
143+
E2E_FLAGS: '--selector quick'
125144

126145
- name: Copy logs
127146
if: ${{ always() }}

‎Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -146,8 +146,8 @@ test: generate fmt vet envtest ## Run tests.
146146
rm cover.out.tmp
147147

148148
.PHONY: e2etest
149-
e2etest: generate local-deploy chainsaw
150-
GIT_REF=$(GIT_REF) $(CHAINSAW) test ./e2e
149+
e2etest: generate local-release local-deploy chainsaw
150+
GIT_REF=$(GIT_REF) $(CHAINSAW) test ./e2e $(E2E_FLAGS)
151151

152152
local-deploy: kind ctlptl tilt kustomize clusterctl
153153
@echo -n "LINODE_TOKEN=$(LINODE_TOKEN)" > config/default/.env.linode

‎docs/src/developers/testing.md

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -177,14 +177,51 @@ var _ = Describe("linode creation", func() {
177177
For e2e tests CAPL uses the [Chainsaw project](https://kyverno.github.io/chainsaw) which leverages `kind` and `tilt` to
178178
spin up a cluster with the CAPL controllers installed and then uses `chainsaw-test.yaml` files to drive e2e testing.
179179

180-
All test live in the e2e folder with a directory structure of `e2e/${CONTROLLER_NAME}/${TEST_NAME}`
180+
All test live in the e2e folder with a directory structure of `e2e/${COMPONENT}/${TEST_NAME}`
181181
### Running tests
182-
In order to run e2e tests run the following command
182+
In order to run e2e tests run the following commands:
183183
```bash
184+
# Required env vars to run e2e tests
185+
export INSTALL_K3S_PROVIDER=true
186+
export INSTALL_RKE2_PROVIDER=true
187+
export LINODE_REGION=us-sea
188+
export LINODE_CONTROL_PLANE_MACHINE_TYPE=g6-standard-2
189+
export LINODE_MACHINE_TYPE=g6-standard-2
190+
191+
# IMPORTANT: Set linode, k3s, and rke2 providers in this config file.
192+
# Find an example at e2e/gha-clusterctl-config.yaml
193+
export CLUSTERCTL_CONFIG=~/.cluster-api/clusterctl.yaml
194+
184195
make e2etest
185196
```
197+
*Note: By default `make e2etest` runs all the e2e tests defined under `/e2e` dir*
198+
199+
In order to run specific test, you need to pass flags to chainsaw by setting env var `E2E_FLAGS`
200+
201+
Example: Only running e2e tests for flavors *(default, k3s, rke2)*
202+
```bash
203+
make e2etest E2E_FLAGS='--selector flavors --assert-timeout 10m0s'
204+
```
205+
*Note: We need to bump up the assert timeout to 10 mins to allow the cluster to complete building and become available*
206+
207+
There are other selectors you can use to invoke specfic tests. Please look at the table below for all the selectors available:
208+
209+
| Tests | Selector |
210+
| ------------- | ------------- |
211+
| All Controllers | `quick` |
212+
| All Flavors (default, k3s, rke2) | `flavors` |
213+
| K3S Cluster | `k3s` |
214+
| RKE2 Cluster | `rke2` |
215+
| Default (kubeadm) Cluster | `default-cluster` |
216+
| Linode Cluster Controller | `linodecluster` |
217+
| Linode Machine Controller | `linodemachine` |
218+
| Linode Obj Controller | `linodeobj` |
219+
| Linode VPC Controller | `linodevpc` |
220+
221+
*Note: For any flavor e2e tests, please set the required env variables*
222+
186223
### Adding tests
187-
1. Create a new directory under the controller you are testing with the naming scheme of `e2e/${CONTROLLER_NAME}/${TEST_NAME}`
224+
1. Create a new directory under the controller you are testing with the naming scheme of `e2e/${COMPONENT}/${TEST_NAME}`
188225
2. Create a minimal `chainsaw-test.yaml` file in the new test dir
189226
```yaml
190227
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json

‎e2e/capl-cluster-flavors/default-CAPL-cluster/assert-capi-resources.yaml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: capi-controller-manager
5+
namespace: capi-system
6+
status:
7+
availableReplicas: 1
8+
---
9+
apiVersion: apps/v1
10+
kind: Deployment
11+
metadata:
12+
name: capl-controller-manager
13+
namespace: capl-system
14+
status:
15+
availableReplicas: 1
16+
---
17+
apiVersion: apps/v1
18+
kind: Deployment
19+
metadata:
20+
name: capi-kubeadm-bootstrap-controller-manager
21+
namespace: kubeadm-bootstrap-system
22+
status:
23+
availableReplicas: 1
24+
---
25+
apiVersion: apps/v1
26+
kind: Deployment
27+
metadata:
28+
name: capi-kubeadm-control-plane-controller-manager
29+
namespace: kubeadm-control-plane-system
30+
status:
31+
availableReplicas: 1
32+
---
33+
apiVersion: apps/v1
34+
kind: Deployment
35+
metadata:
36+
name: caaph-controller-manager
37+
namespace: caaph-system
38+
status:
39+
availableReplicas: 1

‎e2e/capl-cluster-flavors/default-CAPL-cluster/assert-child-cluster-resources.yaml

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
3+
kind: LinodeMachine
4+
metadata:
5+
labels:
6+
cluster.x-k8s.io/cluster-name: ($cluster)
7+
spec:
8+
region: (env('LINODE_REGION'))
9+
type: g6-standard-2
10+
status:
11+
ready: true
12+
instanceState: running
13+
---
14+
apiVersion: cluster.x-k8s.io/v1beta1
15+
kind: Machine
16+
metadata:
17+
labels:
18+
cluster.x-k8s.io/cluster-name: ($cluster)
19+
spec:
20+
clusterName: ($cluster)
21+
status:
22+
bootstrapReady: true
23+
infrastructureReady: true
24+
---
25+
apiVersion: cluster.x-k8s.io/v1beta1
26+
kind: MachineDeployment
27+
metadata:
28+
labels:
29+
cluster.x-k8s.io/cluster-name: ($cluster)
30+
spec:
31+
clusterName: ($cluster)
32+
replicas: 1
33+
status:
34+
readyReplicas: 1
35+
unavailableReplicas: 0
36+
availableReplicas: 1
37+
---
38+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
39+
kind: KubeadmControlPlane
40+
metadata:
41+
labels:
42+
cluster.x-k8s.io/cluster-name: ($cluster)
43+
status:
44+
readyReplicas: 1
45+
unavailableReplicas: 0
46+
ready: true
47+
---
48+
apiVersion: addons.cluster.x-k8s.io/v1alpha1
49+
kind: HelmReleaseProxy
50+
metadata:
51+
labels:
52+
cluster.x-k8s.io/cluster-name: ($cluster)
53+
status:
54+
conditions:
55+
- type: Ready
56+
status: "True"
57+
- type: ClusterAvailable
58+
status: "True"
59+
- type: HelmReleaseReady
60+
status: "True"
61+
status: deployed

‎e2e/capl-cluster-flavors/default-CAPL-cluster/chainsaw-test.yaml

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json
2+
apiVersion: chainsaw.kyverno.io/v1alpha1
3+
kind: Test
4+
metadata:
5+
creationTimestamp: null
6+
name: default-capl-cluster
7+
# Labels to allow the test to be triggered based on selector flag
8+
labels:
9+
default-cluster:
10+
flavors:
11+
spec:
12+
bindings:
13+
# A short identifier for the E2E test run
14+
- name: run
15+
value: (join('-', ['e2e', 'default-cluster', env('GIT_REF')]))
16+
- name: cluster
17+
# Format the cluster name
18+
value: (trim((truncate(($run), `32`)), '-'))
19+
template: true
20+
steps:
21+
- name: Check if CAPI provider resources exist
22+
try:
23+
- assert:
24+
file: assert-capi-resources.yaml
25+
- name: Generate cluster using clusterctl
26+
try:
27+
- script:
28+
env:
29+
- name: CLUSTER
30+
value: ($cluster)
31+
- name: NAMESPACE
32+
value: ($namespace)
33+
- name: CLUSTERCTL_CONFIG
34+
value: (env('CLUSTERCTL_CONFIG'))
35+
content: |
36+
set -e
37+
clusterctl generate cluster $CLUSTER -n $NAMESPACE \
38+
--kubernetes-version v1.29.1 \
39+
--infrastructure local-linode:v0.0.0 \
40+
--control-plane-machine-count 1 --worker-machine-count 1 \
41+
--config $CLUSTERCTL_CONFIG > default-cluster.yaml
42+
check:
43+
($error == null): true
44+
- name: Apply generated cluster yaml
45+
try:
46+
- apply:
47+
file: default-cluster.yaml
48+
- assert:
49+
file: assert-child-cluster-resources.yaml
50+
catch:
51+
- describe:
52+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
53+
kind: LinodeMachine
54+
- describe:
55+
apiVersion: cluster.x-k8s.io/v1beta1
56+
kind: Machine
57+
- describe:
58+
apiVersion: cluster.x-k8s.io/v1beta1
59+
kind: MachineDeployment
60+
- describe:
61+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
62+
kind: KubeadmControlPlane
63+
- describe:
64+
apiVersion: addons.cluster.x-k8s.io/v1alpha1
65+
kind: HelmReleaseProxy
66+
- name: Check if the linodes are created
67+
try:
68+
- script:
69+
env:
70+
- name: TARGET_API
71+
value: api.linode.com
72+
- name: TARGET_API_VERSION
73+
value: v4beta
74+
- name: URI
75+
value: linode/instances
76+
- name: FILTER
77+
value: (to_string({"tags":($cluster)}))
78+
content: |
79+
set -e
80+
curl -s \
81+
-H "Authorization: Bearer $LINODE_TOKEN" \
82+
-H "X-Filter: $FILTER" \
83+
-H "Content-Type: application/json" \
84+
"https://$TARGET_API/$TARGET_API_VERSION/$URI"
85+
check:
86+
($error): ~
87+
(json_parse($stdout)):
88+
results: 2
89+
- name: Delete child cluster
90+
try:
91+
- delete:
92+
ref:
93+
apiVersion: cluster.x-k8s.io/v1beta1
94+
kind: Cluster
95+
name: ($cluster)
96+
- delete:
97+
ref:
98+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
99+
kind: LinodeVPC
100+
name: ($cluster)
101+
- error:
102+
file: check-child-cluster-and-vpc-deleted.yaml
103+
- name: Check if the linodes are deleted
104+
try:
105+
- script:
106+
env:
107+
- name: TARGET_API
108+
value: api.linode.com
109+
- name: TARGET_API_VERSION
110+
value: v4beta
111+
- name: URI
112+
value: linode/instances
113+
- name: FILTER
114+
value: (to_string({"tags":($cluster)}))
115+
content: |
116+
set -e
117+
curl -s \
118+
-H "Authorization: Bearer $LINODE_TOKEN" \
119+
-H "X-Filter: $FILTER" \
120+
-H "Content-Type: application/json" \
121+
"https://$TARGET_API/$TARGET_API_VERSION/$URI"
122+
check:
123+
($error): ~
124+
(json_parse($stdout)):
125+
results: 0
126+
- name: Delete generated child cluster manifest yaml
127+
try:
128+
- script:
129+
content: |
130+
rm -f default-cluster.yaml
131+
check:
132+
($error == null): true

‎e2e/capl-cluster-flavors/default-CAPL-cluster/check-child-cluster-and-vpc-deleted.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
2+
kind: LinodeMachine
3+
metadata:
4+
labels:
5+
cluster.x-k8s.io/cluster-name: ($cluster)
6+
---
7+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
8+
kind: LinodeVPC
9+
metadata:
10+
name: ($cluster)

‎e2e/capl-cluster-flavors/k3s-capl-cluster/assert-capi-resources.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: capi-controller-manager
5+
namespace: capi-system
6+
status:
7+
availableReplicas: 1
8+
---
9+
apiVersion: apps/v1
10+
kind: Deployment
11+
metadata:
12+
name: capl-controller-manager
13+
namespace: capl-system
14+
status:
15+
availableReplicas: 1
16+
---
17+
apiVersion: apps/v1
18+
kind: Deployment
19+
metadata:
20+
name: capi-k3s-bootstrap-controller-manager
21+
namespace: capi-k3s-bootstrap-system
22+
status:
23+
availableReplicas: 1
24+
---
25+
apiVersion: apps/v1
26+
kind: Deployment
27+
metadata:
28+
name: capi-k3s-control-plane-controller-manager
29+
namespace: capi-k3s-control-plane-system
30+
status:
31+
availableReplicas: 1

0 commit comments

Comments
 (0)