Add more GCVE documentation #8146
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: fabriziopandini The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
sig/k8s-infra
chrischdi
reviewed
Jun 2, 2025
| @@ -0,0 +1,25 @@ | |||
| # Boskos | |||
|
|
|||
| Boskos support resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources. | |||
There was a problem hiding this comment.
Suggested change
| Boskos support resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources. | |
| Boskos resources of type `gcve-vsphere-project` allow each test run to use a subset of vSphere resources. |
|
|
||
| See [README.md](https://github.com/kubernetes/k8s.io/tree/main/infra/gcp/terraform) for a general intro about using terraform in k8s.io. | ||
|
|
||
| In order to apply terraform manifests you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. |
There was a problem hiding this comment.
Suggested change
| In order to apply terraform manifests you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. | |
| In order to apply terraform manifests you must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need. |
| docker run -it --rm -v $(pwd):/workspace --entrypoint=/bin/bash gcr.io/k8s-staging-infra-tools/k8s-infra:v20241217-f8b07a049 | ||
| ``` | ||
|
|
||
| From your local workstatin / from inside the terraform container: |
There was a problem hiding this comment.
Suggested change
| From your local workstatin / from inside the terraform container: | |
| From your local workstation / from inside the terraform container: |
| gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn' | ||
| ``` | ||
|
|
||
| Add those entry to `/etc/hosts`. |
There was a problem hiding this comment.
Suggested change
| Add those entry to `/etc/hosts`. | |
| Add those entries to `/etc/hosts`. |
|
|
||
| Wiregard is used to get a secure and convenient access through the maintenace jump host VM. | ||
|
|
||
| In order to use wiregard you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. |
There was a problem hiding this comment.
Suggested change
| In order to use wiregard you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. | |
| In order to use wiregard you must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need. |
| Note: [email protected] user is creating automatically in a VMware Engine instance; | ||
| we are using it to set up vSphere and create a dedicate user for prow CI (with limited permissions). | ||
|
|
||
| Also, the terraform manifests in this folder requires `/etc/hosts` entries for vSphere and NSX |
There was a problem hiding this comment.
Suggested change
| Also, the terraform manifests in this folder requires `/etc/hosts` entries for vSphere and NSX | |
| Also, the terraform manifests in this folder require `/etc/hosts` entries for vSphere and NSX |
|
|
||
| # Accessing vSphere and NSX UI. | ||
|
|
||
| If required for maintenance reasons, it is possible to access the vSphere UI via [wirequard](../docs/wiregard.md) / [jumphost VM](../maintenance-jumphost/README.md). |
There was a problem hiding this comment.
Suggested change
| If required for maintenance reasons, it is possible to access the vSphere UI via [wirequard](../docs/wiregard.md) / [jumphost VM](../maintenance-jumphost/README.md). | |
| If required for maintenance reasons, it is possible to access the vSphere UI via [wireguard](../docs/wiregard.md) / [jumphost VM](../maintenance-jumphost/README.md). |
|
|
||
| First we generate a password for the user which will be used in prow and set it as environment variable: | ||
| The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. There are ther main components of this infrastracture: |
There was a problem hiding this comment.
Suggested change
| The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. There are ther main components of this infrastracture: | |
| The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. These are the main components of this infrastructure: |
| export GOVC_USERNAME='[email protected]' | ||
| export GOVC_PASSWORD="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc [email protected] --location=us-central1-a --format='get(password)')" | ||
| ``` | ||
| The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance pourposes. See |
There was a problem hiding this comment.
Suggested change
| The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance pourposes. See | |
| The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance purposes. See |
|
|
||
| Run the script to setup the user, groups and IAM in vSphere. | ||
| The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templetes, folders, |
There was a problem hiding this comment.
Suggested change
| The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templetes, folders, | |
| The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templates, folders, |
chrischdi
reviewed
Jun 3, 2025
| ```sh | ||
| export GCVE_PROW_CI_PASSWORD="SomePassword" | ||
| ``` | ||
| The terraform manifest in this folder, which is applied by test-infra automation (Atlantis), uses the GCP terraform provider for creating. |
There was a problem hiding this comment.
I just learned that this does not work for us: #8148 (comment)
Should we add terraform instructions instead?
chrischdi
reviewed
Jun 3, 2025
| When ready: | ||
|
|
||
| ```sh | ||
| terraform init |
There was a problem hiding this comment.
Should we add tfswitch?
Suggested change
| terraform init | |
| tfswitch | |
| terraform init |
The tfswitch command is available in the docker container and ensures the right terraform version (set at the provider.tf file) is used.
|
/lgtm Addressed the comments in #8148 |
k8s-ci-robot
added
the
lgtm
|
@chrischdi thanks for addressing comments in another PR, I'm running low of bandwidth lately |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kudos to @chrischdi for the great work!
This adds a little bit of doc on top
/cc @sbueringer