Skip to content

Commit 23b8a6f

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #8146 from fabriziopandini/document-k8s-infra-gcp-gcve
Add more GCVE documentation
2 parents f97f0e2 + 95ee6e0 commit 23b8a6f

File tree

8 files changed

+526
-123
lines changed

8 files changed

+526
-123
lines changed

‎infra/gcp/terraform/k8s-infra-gcp-gcve/README.md

Lines changed: 27 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -1,87 +1,39 @@
1-
# Setup
1+
# Overview
22

3-
## Creation of GCVE
3+
The code in `k8s-infra-gcp-gcve` sets up the infra required to allow prow jobs to create VMs on vSphere, e.g. to allow testing of the [Cluster API provider vSphere (CAPV)](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere).
44

5-
```sh
6-
gcloud auth application-default login
7-
terraform init
8-
terraform apply
9-
```
5+
![Overview](./docs/images/overview.jpg)
106

11-
## Setup jumphost/vpn for further configuration
7+
Prow container settings are managed outside of this folder, but understanding high level components could
8+
help to understand how the `k8s-infra-gcp-gcve` is set up and consumed.
129

13-
See [maintenance-jumphost/README.md](./maintenance-jumphost/README.md).
10+
More specifically, to allow prow jobs to create VM on vSphere, a few resources are made available to a prow container, so as of today only in the `k8s-infra-prow-build` prow cluster.
1411

15-
## Manual creation of a user and other IAM configuration in vSphere
12+
- A secret, added via the `preset-gcve-e2e-config` [preset](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/cluster-api-provider-vsphere/cluster-api-provider-vsphere-presets.yaml), that provides vSphere URL and vSphere credentials
13+
- A set of Boskos resources of type `gcve-vsphere-project`, allowing access to:
14+
- a vSphere folder and a vSphere resources pool where to run VMs during a test.
15+
- a reserved IP range to be used for the test e.g. for the kube vip load balancer in a CAPV cluster (VM instead will get IPs via DHCP).
1616

17-
> **Note:**
18-
> The configuration described here cannot be done via terraform due to non-existing functionality.
17+
Also, the network of the prow container is going to be paired to the VMware engine network, thus
18+
allowing access to both the GCVE management network and the NSX-T network where all the VM are running.
1919

20-
First we generate a password for the user which will be used in prow and set it as environment variable:
20+
The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. There are ther main components of this infrastracture:
2121

22-
```sh
23-
export GCVE_PROW_CI_PASSWORD="SomePassword"
24-
```
22+
The terraform manifest in this folder, which is applied by test-infra automation (Atlantis), uses the GCP terraform provider for creating.
23+
- A VMware Engine instance
24+
- The network infrastructure required for vSphere and for allowing communication between vSphere and Prow container.
25+
- The network used is `192.168.0.32/21`
26+
- Usable Host IP Range: `192.168.32.1 - 192.168.39.254`
27+
- DHCP Range: `192.168.32.11 - 192.168.33.255`
28+
- IPPool for 40 Projects having 16 IPs each: `192.168.35.0 - 192.168.37.127`
29+
- The network infrastructure used for maintenance.
2530

26-
And set credentials for `govc`:
31+
See inline comments for more details.
2732

28-
```sh
29-
export GOVC_URL="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(vcenter.fqdn)')"
30-
export GOVC_USERNAME='[email protected]'
31-
export GOVC_PASSWORD="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc [email protected] --location=us-central1-a --format='get(password)')"
32-
```
33+
The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance pourposes. See
34+
- [maintenance-jumphost](./maintenance-jumphost/README.md)
3335

34-
Run the script to setup the user, groups and IAM in vSphere.
36+
The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templetes, folders,
37+
resource pools and other vSphere components required when running tests. See:
38+
- [vsphere](./vsphere/README.md)
3539

36-
```
37-
./vsphere/scripts/ensure-users-groups.sh
38-
```
39-
40-
Create relevant secrets in Secrets Manager
41-
42-
```sh
43-
gcloud secrets describe k8s-gcp-gcve-ci-url 2>/dev/null || echo "$GOVC_URL" | gcloud secrets create k8s-gcp-gcve-ci-url --data-file=-
44-
gcloud secrets describe k8s-gcp-gcve-ci-username 2>/dev/null || echo "[email protected]" | gcloud secrets create k8s-gcp-gcve-ci-username --data-file=-
45-
gcloud secrets describe k8s-gcp-gcve-ci-password 2>/dev/null || echo "${GCVE_PROW_CI_PASSWORD}" | gcloud secrets create k8s-gcp-gcve-ci-password --data-file=-
46-
gcloud secrets describe k8s-gcp-gcve-ci-thumbprint 2>/dev/null || echo "$(govc about.cert -json | jq -r '.thumbprintSHA256')" | gcloud secrets create k8s-gcp-gcve-ci-thumbprint --data-file=-
47-
```
48-
49-
* `k8s-gcp-gcve-ci-username` with value `[email protected]`
50-
* `k8s-gcp-gcve-ci-password` with value set above for `GCVE_PROW_CI_PASSWORD`
51-
* `k8s-gcp-gcve-ci-url` with value set above for `GOVC_URL`
52-
53-
> **Note:** Changing the GCVE CI user's password
54-
>
55-
> 1. Set GOVC credentials as above.
56-
> 2. Run govc command to update password: `govc sso.user.update -p "${GCVE_PROW_CI_PASSWORD}" prow-ci-user`
57-
> 3. Update secret `k8s-gcp-gcve-ci-password` in secrets-manager: `echo "${GCVE_PROW_CI_PASSWORD}" | gcloud secrets versions add k8s-gcp-gcve-ci-password --data-file=-`
58-
59-
## Configuration of GCVE
60-
61-
```sh
62-
export [email protected]
63-
export TF_VAR_vsphere_password="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc [email protected] --location=us-central1-a --format='get(password)')" # gcloud command
64-
export TF_VAR_vsphere_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(vcenter.fqdn)')"
65-
export TF_VAR_nsxt_user=admin
66-
export TF_VAR_nsxt_password="$(gcloud vmware private-clouds nsx credentials describe --private-cloud k8s-gcp-gcve-pc --location us-central1-a --format='get(password)')"
67-
export TF_VAR_nsxt_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(nsx.fqdn)')"
68-
export GOVC_URL="${TF_VAR_vsphere_server}"
69-
export GOVC_USERNAME="${TF_VAR_vsphere_user}"
70-
export GOVC_PASSWORD="${TF_VAR_vsphere_password}"
71-
```
72-
73-
```sh
74-
cd vsphere
75-
terraform init
76-
terraform apply
77-
./scripts/ensure-users-permissions.sh
78-
```
79-
80-
## Initialize Boskos resources with project information
81-
82-
The script [boskos-userdata.sh](vsphere/scripts/boskos-userdata.sh) calculates and initializes the Boskos resources required for the project.
83-
84-
```sh
85-
BOSKOS_HOST=""
86-
vsphere/scripts/boskos-userdata.sh
87-
```

‎infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Boskos
2+
3+
Boskos support resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources.
4+
5+
Boskos configuration is split in three parts:
6+
7+
- The definition of the resource type in the [boskos-reaper](https://github.com/kubernetes/k8s.io/blob/main/kubernetes/gke-prow-build/prow/boskos-reaper.yaml) Deployment
8+
- search for e.g. `gcve-vsphere-project`
9+
- A static list of resources in the [boskos-resources-configmap](https://github.com/kubernetes/k8s.io/blob/main/kubernetes/gke-prow-build/prow/boskos-resources-configmap.yaml)
10+
- As of today we have 40 Boskos resources (from `k8s-infra-e2e-gcp-gcve-project-001` tp `k8s-infra-e2e-gcp-gcve-project-040`)
11+
- Setting up user data for each resource.
12+
13+
The last step requires access to the Boskos instance running in prow.
14+
15+
Once you get access run the following script:
16+
17+
```sh
18+
BOSKOS_HOST=""
19+
vsphere/scripts/boskos-userdata.sh
20+
```
21+
22+
This script adds user data to each one of the above resources, e.g. for `k8s-infra-e2e-gcp-gcve-project-001` we are going to set following user data linking to some of the objects previously set up in vSphere for prow tests:
23+
- A vSphere folder, e.g. `/Datacenter/vm/prow/k8s-infra-e2e-gcp-gcve-project-001`
24+
- A vSphere resource pool, e.g. `/Datacenter/host/k8s-gcve-cluster/Resources/prow/k8s-infra-e2e-gcp-gcve-project-001`
25+
- An ipPool with 16 addresses, e.g. `192.168.35.0-192.168.35.15`, corresponding gateway, `192.168.32.1` and CIDR subnet mask prefix, e.g. `21`

0 commit comments

Comments
 (0)