🐛 fix: Clear credential cache on InvalidClientTokenId to allow retry on next reconcile

[omargallob]

Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Fixes #6123

What this PR does

Clears the cached *aws.CredentialsCache in AWSRolePrincipalTypeProvider.Retrieve() when the error is InvalidClientTokenId, allowing the next controller-runtime reconcile loop to create a fresh credentials cache and retry the AssumeRole call.

Why this is needed

After the AWS SDK v1→v2 migration, aws.NewCredentialsCache caches failed AssumeRole results. When a freshly-created IAM access key has not yet propagated through AWS IAM eventual consistency, the first AssumeRole call fails with InvalidClientTokenId. Unlike SDK v1 where IsExpired() returned true after failure (triggering a fresh attempt on the next reconcile), SDK v2's CredentialsCache retains the error.

This causes the CAPA controller to be unable to recover from a transient IAM propagation delay for the lifetime of the provider instance (~50 minutes observed in production until KubeadmControlPlane timeout).

How it works

result, err := p.credentials.Retrieve(ctx)
if err != nil && IsInvalidClientTokenIDError(err) {
    p.credentials = nil  // next reconcile will recreate the cache
}
return result, err

This restores the pre-SDK-migration behavior with minimal change.

Testing

• Unit test added verifying cache is cleared on InvalidClientTokenId and subsequent Retrieve() succeeds
• Unit test for IsInvalidClientTokenIDError helper with typed and string-match cases

/kind bug

AI Usage:

1. Research & Analysis — Investigated CAPA source code (<v2.9 vs v2.11.1), identified the AWS SDK v1→v2 credential caching regression as root cause of the IAM key propagation issue
2. Code — Implemented the fix in pkg/cloud/identity/identity.go (added IsInvalidClientTokenIDError helper, modified Retrieve() to clear cache on transient error) and wrote unit tests in identity_test.go

Checklist:

• squashed commits
• includes documentation
• includes AI generated content
• includes emoji in title
• adds unit tests
• adds or updates e2e tests

Fix credential cache poisoning in AWSRolePrincipalTypeProvider when AssumeRole fails with InvalidClientTokenId due to IAM eventual consistency. The credential cache is now cleared on this transient error, allowing the controller to retry on the next reconcile instead of remaining stuck until restart.

[kubernetes-prow]

@omargallob: The label(s) area/provider/identity cannot be applied, because the repository doesn't have them.

Details

In response to this:

What this PR does

Clears the cached *aws.CredentialsCache in AWSRolePrincipalTypeProvider.Retrieve() when the error is InvalidClientTokenId, allowing the next controller-runtime reconcile loop to create a fresh credentials cache and retry the AssumeRole call.

Why this is needed

After the AWS SDK v1→v2 migration, aws.NewCredentialsCache caches failed AssumeRole results. When a freshly-created IAM access key has not yet propagated through AWS IAM eventual consistency, the first AssumeRole call fails with InvalidClientTokenId. Unlike SDK v1 where IsExpired() returned true after failure (triggering a fresh attempt on the next reconcile), SDK v2's CredentialsCache retains the error.

This causes the CAPA controller to be unable to recover from a transient IAM propagation delay for the lifetime of the provider instance (~50 minutes observed in production until KubeadmControlPlane timeout).

How it works

result, err := p.credentials.Retrieve(ctx)
if err != nil && IsInvalidClientTokenIDError(err) {
   p.credentials = nil  // next reconcile will recreate the cache
}
return result, err

This restores the pre-SDK-migration behavior with minimal change.

Testing

• Unit test added verifying cache is cleared on InvalidClientTokenId and subsequent Retrieve() succeeds
• Unit test for IsInvalidClientTokenIDError helper with typed and string-match cases

Fixes #6123

/kind bug
/area provider/identity

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: omargallob / name: omar gallo (7299254)

[kubernetes-prow]

Welcome @omargallob!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[kubernetes-prow]

Hi @omargallob. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andidog for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[omargallob]

/kind bug
/area provider/aws
/release-note

[kubernetes-prow]

@omargallob: the /release-note and /release-note-action-required commands have been deprecated.
Please edit the release-note block in the PR body text to include the release note. If the release note requires additional action include the string action required in the release note. For example:

```release-note
Some release note with action required.
```

Details

In response to this:

/kind bug
/area provider/aws
/release-note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nrb]

/ok-to-test

[omargallob]

what are the next steps?

[richardcase]

So we'd still get an error in the logs but at least the cache is cleared and on the next reconcile it would work.

[richardcase]

/test ?

[richardcase]

/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e

[richardcase]

/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e

[Schuorz]

I would like to propose a little adjustment: omargallob#2
This change should reduce log pollution, by using aws-sdk-go-v2 failure handling, in CAPA while keeping the overall cache invalidation logic.

Giving it a little bit more thought, I realized, while related my proposal is independent of this PR. I will create my own, to not polute this one.