fix: do not delete workspace pod on authz errors

[rquitales]

Proposed Changes

This PR updates the workspace controller to run the grpc WhoAmI command before proceeding with reconciliation. This change helps surface authentication issues early in the process.

If authentication fails, the workspace pod is retained rather than deleted, as it remains in a pristine state. This avoids unnecessary pod recreation and reduces the time required to spin up a new StatefulSet pod. Additionally, it prevents excessive workspace pod churn in the event of persistent authentication failures.

Example Log Output on Authentication Failure

2025-02-05T18:37:33.029Z	INFO	Applying StatefulSet	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112271", "hash": "ec3e2de049a62cdb9a133f4c141df826", "source": {"Generation":1,"ForceRequest":"","Git":{"URL":"https://github.com/pulumi/examples","Ref":"4ce6027439cc8c9cc60f5704abc7d2204a07d98e","Dir":"/kubernetes-ts-nginx","Shallow":false,"SSHPrivateKey":null,"Username":null,"Password":null,"Token":null},"Flux":null,"Local":null}}
2025-02-05T18:37:33.151Z	INFO	Connecting to workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112271", "addr": "nginx-k8s-stack-workspace.pulumi-kubernetes-operator:50051"}
2025-02-05T18:37:33.158Z	INFO	Connected to workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112271", "addr": "nginx-k8s-stack-workspace.pulumi-kubernetes-operator:50051"}
2025-02-05T18:37:33.158Z	INFO	Running whoami to ensure authentication is setup correctly with the workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112271"}
2025-02-05T18:37:33.164Z	ERROR	unable to authenticate; retaining the workspace pod to retry later	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112271", "error": "rpc error: code = Unauthenticated desc = TokenReview API is unavailable"}
...
2025-02-05T18:37:33.176Z	INFO	Status updated	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"nginx-k8s-stack","namespace":"pulumi-kubernetes-operator"}, "namespace": "pulumi-kubernetes-operator", "name": "nginx-k8s-stack", "reconcileID": "bca83f73-73aa-4004-bce0-14aa8c4bf48a", "revision": "112336", "observedGeneration": 1, "address": "nginx-k8s-stack-workspace.pulumi-kubernetes-operator:50051", "conditions": [{"type":"Ready","status":"False","observedGeneration":1,"lastTransitionTime":"2025-02-05T18:37:12Z","reason":"AuthenticationFailed","message":"Unable to authenticate with the workspace pod."}]}

Stored Workspace Status

When the wrong Pulumi Access token is provided:

status:
  address: nginx-stack-workspace.default:50051
  conditions:
  - lastTransitionTime: "2025-02-07T23:34:13Z"
    message: Invalid access token used to authenticate with Pulumi Cloud
    observedGeneration: 1
    reason: InvalidAccessToken
    status: "False"
    type: Ready
  observedGeneration: 1

When there is a K8s auth issue:

status:
  address: nginx-stack-workspace.default:50051
  conditions:
  - lastTransitionTime: "2025-02-07T23:34:13Z"
    message: TokenReview API is unavailable
    observedGeneration: 2
    reason: Unauthenticated
    status: "False"
    type: Ready
  observedGeneration: 2

Testing

• Added an end-to-end (e2e) test to verify that the workspace pod is not deleted when authentication fails.
• Manually validated on a GKE cluster.

Related Issues (Optional)

Fixes: #740

[rquitales]

This change is part of the following stack:

• feat: add WorkspaceReclaimPolicy to Stack spec #804
  • feat: report status when stack is locked #807
    • fix: do not delete workspace pod on authz errors #805 ◀

_{Change managed by git-spice.}

[EronWright]

Awesome

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 23 lines in your changes missing coverage. Please review.

Project coverage is 51.22%. Comparing base (43cd638) to head (6c2747a).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...r/internal/controller/auto/workspace_controller.go	0.00%	20 Missing ⚠️
agent/pkg/server/server.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #805      +/-   ##
==========================================
- Coverage   51.48%   51.22%   -0.27%     
==========================================
  Files          31       31              
  Lines        4296     4318      +22     
==========================================
  Hits         2212     2212              
- Misses       1895     1917      +22     
  Partials      189      189              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.