Add support for TLS for xDS with Gloo Gateway

[sheidkamp]

Description

The helm setting global.glooMtls.enabled has been extended to configure gg2 proxies to use mtls for xds. Previously the setting only applied to Edge gateways.

API changes

none

Code changes

• deployer
  • Added helm values to store mTLS data and methods to populate from Kubernetes
    • Updated proxy-deployment template to create Secret when needed.
  • Added GlooMtlsEnabled and Namespace to ControlPlaneInfo
• ggv2setup
  • Added check if glooMtls is enabled and passing value to NewControllerBuilder
• gateway2 controller
  • Added watcher for gloo-mtls-certs secret (if gloo mTLS is enabled)
  • Populate Gateway Config with new ControlPlaneInfo values

CI changes

none

Docs changes

none

Context

Users wanted to be able to use mtls with xDS for gg2 proxies.

mTLS docs

Interesting decisions

• Extending the Gloo Edge approach (we generate the certs and manage their rotation) rather than implement a "better" solution.
  Decision captured here. An approach using one way TLS + JWT using a k8s projected service account token is being investigated, and wanted to continue this approach to ensure delivery of functionality.

• Using the global global.glooMtls.enabled flag
  In the gloo control plane, mtls for xDS is either on or off. Adding a field to GatewayParameters would have allowed us to control the use of mtls per-proxy, but if the settings did not match the global setting, the xDS connection would fail
  Discussion about using GatewayParameters to potentially allow mixed mode

• Copying secrets across namespaces
  Mtls is implemented by mounting secrets to sidecar containers. In classic Gloo Edge, the proxy was always in the same namespace as the gloo pod, so they were both able to access the secret.
  
  gg2 proxies can be deployed into many namespaces, and secrets can only be mounted into a container from the same namespace. Because of this, the secret needs to be available in the namespace.
  
  We chose the deployer as the mechanism to mirror the secrets, as it already has the responsibility of deploying Kubernetes resources to the proxy namespace based on changes to underlying resources.

Testing steps

• Setup a kubernetes cluster

./ci/kind/setup-kind.sh;

• Install gloo with mtls enabled

helm upgrade -i -n gloo-system gloo ./_test/gloo-1.0.0-ci1.tgz --create-namespace --set kubeGateway.enabled=true --values test/kubernetes/e2e/tests/manifests/profiles/full-gateway.yaml --set global.glooMtls.enabled=true

• Apply resources

kubectl apply -f test/kubernetes/e2e/features/upgrade/testdata/service-for-route.yaml
kubectl apply -f test/kubernetes/e2e/features/upgrade/testdata/route-with-service.yaml

• Port forward to the proxy

kubectl port-forward deploy/gloo-proxy-gw 8080:8080 &
export PID=$!

• Successfully curl

curl -v -H "Host: example.com" localhost:8080

• Validate that the proxy deployment has the SDS container

kubectl get deployment gloo-proxy-gw -o jsonpath='{.spec.template.spec.containers[*].name}'

• Validate that the gloo deployment has the SDS container

kubectl get deployment gloo -n gloo-system -o jsonpath='{.spec.template.spec.containers[*].name}'

• Get secret resource versions

kubectl get secret gloo-mtls-certs -o jsonpath='{.metadata.resourceVersion}'
kubectl get secret gloo-mtls-certs -n gloo-system -o jsonpath='{.metadata.resourceVersion}'

• Force secret rotation

kubectl apply -f test/kubernetes/e2e/features/gloomtls/testdata/certgen.yaml

• Validate that both secrets were updated (new resouceVersions)

kubectl get secret gloo-mtls-certs -o jsonpath='{.metadata.resourceVersion}'
kubectl get secret gloo-mtls-certs -n gloo-system -o jsonpath='{.metadata.resourceVersion}'

• Apply a change to validate xDS communication

kubectl patch HTTPRoute example-route -p '{"spec":{"hostnames":["example.org"]}}' --type=merge

• Curl the old host (should fail)

curl -v -H "Host: example.com" localhost:8080

• Curl the new host (should succeed)

curl -v -H "Host: example.org" localhost:8080

• When done, stop port-forwarding

kill $PID

Istio validation

Istio can be used with Gloo Gateway to provide autoMtls for upstreams, and is implemented using the same SDS container on the proxy deployment that the xDS mTLS uses. There shouldn't be any interference between these pieces of functionality, but we should validate.

Manual validation

Follow the steps outlined in https://github.com/solo-io/gloo/blob/main/projects/gateway2/README.md, and when installing Gloo, add --set global.glooMtls.enabled=true to the helm upgrade -i commands.

Validation with e2e tests

In this (since reverted) commit, glooMtls was enabled for the Istio kubernetes e2e tests and run through CI. Results - the tests that failed were checking for exactly one container in the proxy-deployment, which is not the case under MTLS.

Notes for reviewers

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[github-actions]

Visit the preview URL for this PR (updated for commit 0500463):

https://gloo-edge--pr10582-sheidkamp-k8s-gw-mtl-dm0tvwih.web.app

_{(expires Thu, 06 Feb 2025 16:31:20 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[solo-changelog-bot]

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/6210

[ryanrolds]

LGTM

[jenshu]

nit: can use IsEnvTruthy helper