Skip to content

🐛 Stop recreating EKS access entries when username is unset - #6176

Open
moko-poi wants to merge 1 commit into
kubernetes-sigs:mainfrom
moko-poi:fix/access-entry-username-recreate-loop
Open

🐛 Stop recreating EKS access entries when username is unset#6176
moko-poi wants to merge 1 commit into
kubernetes-sigs:mainfrom
moko-poi:fix/access-entry-username-recreate-loop

Conversation

@moko-poi

@moko-poi moko-poi commented Aug 6, 2026

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

An access entry whose spec omits username is deleted and recreated on every reconcile.

updateAccessEntry compares the spec username against the value reported by AWS. When username is omitted, accessEntry.Username is "" (a plain string with omitempty and no defaulting, and createAccessEntry skips setting it), while EKS generates a username server-side. The comparison therefore never holds and the entry is torn down and rebuilt on every cycle. Each DeleteAccessEntry drops the associated access policies, so principals authorized via accessPolicies intermittently lose access — this is how the bug becomes user-visible as sporadic Forbidden errors.

UpdateAccessEntryInput accepts Username and has no Type field, so type is the only genuinely immutable attribute. This PR:

  1. Narrows the recreate path to a type change, and corrects the comment claiming a recreate is required to change the username.
  2. Moves username to the update path, treating an empty spec value as unmanaged so the EKS generated username is not seen as drift.
  3. Gates UpdateAccessEntry on a flag. Previously only a KubernetesGroups difference triggered the call, so a username-only change would have been silently dropped.

Beyond stopping the loop, this removes the delete/create window for username changes entirely.

Which issue(s) this PR fixes:

Fixes #6003

Special notes for your reviewer:

This overlaps with #6007, which fixes the same issue from the Type side by normalizing an empty Type to STANDARD. That PR leaves accessEntry.Username != existingUsername untouched, so entries that set type explicitly and omit username keep looping. Both PRs modify the same condition and will conflict textually; I am happy to rebase whichever lands second, or to fold this into #6007 if the author prefers.

Since this narrows the recreate condition to type only, it also subsumes the Type half of #6007 for the recreate decision, though not the redundant AssociateAccessPolicy calls that #6007 also addresses.

Three existing test cases changed expectations from delete+create to an in-place update. One of them, username cleared requires recreate, asserted that emptying username in the spec resets the entry. That behaviour is dropped here: EKS offers no way to clear a username and revert to a generated one, so the only mechanism was the recreate that causes this bug, and it is not documented anywhere. Losing an undocumented reset seemed clearly preferable to an endless loop that breaks authorization.

On verification: I observed the delete/create loop and the resulting Forbidden errors on a live EKS cluster, which is what led me here. The fix itself is verified by unit tests only — I have not run a patched controller against a real cluster. make lint and the pkg/cloud/services/eks and controlplane/eks/... suites pass.

Not included: test/e2e/data/eks/cluster-template-eks-control-plane-only-with-accessentries.yaml already contains an entry that omits username and carries accessPolicies, so the e2e suite exercises the affected configuration but has no assertion on entry stability. Adding one seems worthwhile but felt out of scope, and I cannot run the EKS e2e suite locally.

AI Usage:

Claude Code (Claude Opus) was used throughout. I investigated the production symptom and directed the work; the model read the upstream source to identify the faulty comparison, confirmed via go doc that UpdateAccessEntryInput carries Username but not Type (which is what shifted the fix from suppressing the comparison to moving username into the update path), and drafted the code, tests, and this description. I reviewed every change, and verified the tests fail without the fix and pass with it.

Checklist:

  • squashed commits
  • includes documentation
  • includes AI generated content
  • includes emoji in title
  • tested manually
  • includes unit tests
  • adds or updates e2e tests
Fix EKS access entries being deleted and recreated on every reconcile when `username` is not set in the spec, which intermittently removed associated access policies. Username changes are now applied in place; only a `type` change recreates the entry. Note that clearing `username` in the spec no longer resets the entry to an EKS generated username.

An access entry whose spec omits username was deleted and recreated on
every reconcile: the spec value is "" while EKS generates a username
server-side, so the two never compared equal. Each delete dropped the
associated access policies, intermittently breaking authorization for
the principal.

UpdateAccessEntry accepts username, so only type actually requires a
recreate. Move username to the update path and treat an empty spec value
as unmanaged, leaving the EKS generated one in place. Gate the update
call on a flag so a username-only change is no longer skipped.

Signed-off-by: moko-poi <mokopoi44@gmail.com>
@kubernetes-prow kubernetes-prow Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. needs-priority cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 6, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Hi @moko-poi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 6, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from faiq and nrb August 6, 2026 03:14
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign richardcase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@damdo

damdo commented Aug 7, 2026

Copy link
Copy Markdown
Member

/assign @raykrueger

@kubernetes-prow

Copy link
Copy Markdown
Contributor

@damdo: GitHub didn't allow me to assign the following users: raykrueger.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @raykrueger

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@raykrueger

Copy link
Copy Markdown
Contributor

I should probably join the org, the bots keep telling me that :)

@raykrueger

Copy link
Copy Markdown
Contributor

@moko-poi I'm looking at cherry-picking your work into mine so you get credit. I also have an integration test to fix at the same time.

@raykrueger

Copy link
Copy Markdown
Contributor

Cherry-picked in, working on integration tests.

@richardcase

Copy link
Copy Markdown
Member

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 21, 2026
@raykrueger

Copy link
Copy Markdown
Contributor

The work from this Branch/PR was merged into #6007 I think we can close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-priority ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

accessEntries reconciliation causes continuous delete+recreate cycle, breaking auth

4 participants