🐛 Rebuild AWS session per controller when principal credentials change

[UgurTheG]

What type of PR is this?

/kind bug

What this PR does / why we need it:

sessionForClusterWithRegion decides whether it can reuse a cached AWS session by checking whether every principal provider was already present in providerCache:

isChanged := false
for i, provider := range providers {
    ...
    cachedProvider, ok := providerCache.Load(providerHash)
    if ok {
        provider = cachedProvider.(identity.AWSPrincipalTypeProvider)
    } else {
        isChanged = true
        providerCache.Store(providerHash, provider)
    }
    ...
}

if !isChanged {
    if s, ok := sessionCache.Load(getSessionName(region, clusterScoper)); ok {
        entry := s.(*sessionCacheEntry)
        return entry.session, entry.serviceLimiters, nil
    }
}

providerCache is global, but sessionCache is keyed per controller:

func getSessionName(region string, clusterScoper cloud.SessionMetadata) string {
	return fmt.Sprintf("%s-%s-%s-%s", region, clusterScoper.ControllerName(), clusterScoper.InfraClusterName(), clusterScoper.Namespace())
}

So when the credentials behind an identity change, the first controller to reconcile misses providerCache, rebuilds its session correctly, and repopulates providerCache. Every other controller then hits providerCache, leaves isChanged false, and early-returns its own sessionCache entry, which was built from the previous credentials. The sessionCache.Delete() recovery path further down is never reached, because the early return happens first, so the stale session is served until the process restarts.

This PR records the provider hashes a session was built from in sessionCacheEntry and reuses the cached session only when those hashes still match. The decision becomes local to each controller's own cache entry instead of depending on a global cache that another controller may have already populated.

We hit this in production: a cluster was deleted and re-provisioned reusing the same name and namespace, with freshly minted static credentials (the previous IAM access key was removed). AWSCluster reconciled first and came up fully healthy (VPC, subnets, NAT gateways, LB all ready) while AWSMachine kept failing for ~10 hours with:

failed to query AWSMachine instance by tags: ... operation error EC2: DescribeInstances,
get identity: get credentials: failed to refresh cached credentials,
operation error STS: AssumeRole, https response error StatusCode: 403,
api error InvalidClientTokenId: The security token included in the request is invalid.

Same identity, same cluster, different controller: the machine controller was pinned to the pre-rotation session. Restarting capa-controller-manager was the only remedy, and the problem returned on the next re-provisioning.

Which issue(s) this PR fixes:

None filed. Happy to open one first if maintainers prefer that.

Special notes for your reviewer:

All tests pass on this branch (go test ./pkg/cloud/scope/... is green).

To confirm the new test is a genuine regression test rather than one that would pass either way, I reverted just the session.go change and re-ran it. With the fix reverted it fails as expected, because the second controller keeps serving the pre-rotation access key:

--- FAIL: TestSessionRebuiltPerControllerAfterCredentialRotation
    session_cache_test.go:160:
        Expected
            <string>: AKIAIOSFODNN7EXAMPLE
        to equal
            <string>: AKIAI44QH8DHBEXAMPLE

With the fix in place it passes. The test drives sessionForClusterWithRegion through two cloud.SessionMetadata stubs differing only in ControllerName(), rotates the secret behind an AWSClusterStaticIdentity, and asserts both controllers observe the new access key. It uses a static identity so no STS or network calls are involved.

This PR closes two distinct paths to the same symptom, because fixing only the first leaves the second reachable:

1. Cross-controller session reuse (session.go). Covered above.
2. Role provider hash did not cover the source credentials (identity.go). AWSRolePrincipalTypeProvider.Hash() gob-encodes the provider, and gob only encodes exported fields. sourceProvider is unexported, so the hash was determined solely by the AWSClusterRoleIdentity object. Rotating the source secret while leaving that object untouched produced an identical hash, providerCache returned the provider built from the previous credentials, and every caller kept using them. The source provider hash is now mixed in.
   Both are needed in practice. Our provisioning applies the identity objects and the credentials secret with create-or-update semantics, so a full teardown and recreate goes through path 1 (new object UIDs), while an in-place credential refresh goes through path 2 (object unchanged, secret rotated). Each has its own test that fails without the corresponding change.
   Also worth noting separately: neither sessionCache nor providerCache is ever evicted, and Hash() computes sha256.New() then hash.Sum(gobBytes), which returns gobBytes || sha256("") rather than a digest of the input, so cache keys are full gob payloads that accumulate per identity revision.

AI Usage:

This PR was produced with AI assistance: GitHub Copilot in agent mode (Claude Sonnet 4.5) via the JetBrains IDE integration. The AI performed the production incident investigation, read the CAPA source to identify the cache interaction, wrote the fix and the regression test, and drafted this description. All of it was verified against a real cluster and by running the test against both patched and unpatched code. Reviewed by me before submission.

Checklist:

• squashed commits (two commits, one per defect; happy to squash on request)
• includes documentation
• includes AI generated content
• includes emoji in title
• adds unit tests
• adds or updates e2e tests

Release note:

Fix AWS credentials going stale after an identity's credentials are rotated. Session reuse was gated on the global provider cache, so once one controller repopulated it every other controller kept serving a session built from the previous credentials. Separately, a role principal's cache hash did not cover the source credentials used to assume the role, so rotating them left the cached provider in place. Both required a manager restart to recover.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: UgurTheG / name: Ugur Guenduez (fc401f1)

[kubernetes-prow]

Welcome @UgurTheG!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[kubernetes-prow]

Hi @UgurTheG. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dlipovetsky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[richardcase]

/ok-to-test

[UgurTheG]

/retest