accessEntries reconciliation causes continuous delete+recreate cycle, breaking auth

[vterdunov]

What happened

When accessEntries is defined in AWSManagedControlPlane, CAPA deletes and recreates the access entry on every reconcile loop (~7 seconds). The access policy association is lost during this cycle because the next delete occurs before the policy is successfully re-associated.

This results in intermittent Unauthorized and Forbidden errors for all users authenticating via the affected access entry, making the cluster unusable for human operators.

What I expected

CAPA should detect that the access entry already exists with the correct configuration and skip reconciliation (no delete+create).

How to reproduce

1. Create an EKS cluster via CAPA with the following accessEntries in AWSManagedControlPlane:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: my-cluster
spec:
  eksClusterName: my-cluster
  region: eu-central-1
  version: v1.35.1
  accessConfig:
    authenticationMode: api_and_config_map
  accessEntries:
    - principalARN: "arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/eu-central-1/AWSReservedSSO_AdministratorAccess_abc123def456"
      type: standard
      accessPolicies:
        - policyARN: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          accessScope:
            type: cluster
  endpointAccess:
    private: true
    public: false

2. Wait for the cluster to become Ready.
3. Observe that kubectl get ns intermittently returns Unauthorized or Forbidden.

Evidence

Access entry createdAt changes every ~7 seconds (entry is being deleted and recreated):

$ for i in 1 2 3; do aws eks describe-access-entry ... --query 'accessEntry.createdAt'; sleep 7; done
"2024-01-01T10:00:00.044000+00:00"
"2024-01-01T10:00:07.355000+00:00"
"2024-01-01T10:00:14.683000+00:00"

Access policy is empty after recreation — CAPA creates the entry but fails to associate the policy before the next reconcile deletes it:

$ aws eks list-associated-access-policies --cluster-name my-cluster --principal-arn "arn:aws:iam::..."
{
    "associatedAccessPolicies": []
}

Kubernetes events showed 707 reconciliations before event deduplication kicked in (count stopped incrementing while createdAt continued to change every 7s):

$ kubectl get events -n my-cluster --field-selector reason=SuccessfulReconcileAccessEntries
707    2024-01-01T10:00:00Z    2024-01-01T11:39:00Z

Stopping CAPA controller confirms it is the cause — after scaling the deployment to 0, createdAt stops changing and (once policy is manually re-associated) auth works stably.

CAPA controller logs do not explicitly log access entry create/delete operations at Info level, making this issue difficult to diagnose. The only indication is the Kubernetes event count and the changing createdAt timestamp.

Suspected root cause

The access entry reconciler detects drift on every cycle even when the entry matches the desired state, triggering a delete+create. The sequence is: detect drift → delete entry (removes policy association) → create entry → (should associate policy, but next reconcile deletes before association completes).

Environment

• CAPA version: v2.11.1
• EKS version: v1.35.1
• Kubernetes (management cluster): v1.35
• Auth mode: API_AND_CONFIG_MAP
• Endpoint: private only
• Principal ARN: SSO role with full IAM path (/aws-reserved/sso.amazonaws.com/...)

When accessEntries is not defined in the spec, CAPA logs "no access entries defined, skipping reconcile" and does not touch existing entries.

[raykrueger]

My AI minion and I have come up with the following.

When accessEntries with an unspecified type field are defined in AWSManagedControlPlane, the reconciler detects drift on every cycle because an empty Type is compared as "" against "STANDARD" (the EKS default) from the AWS API. This triggers a continuous delete-and-recreate loop every ~7 seconds, with each deletion removing the access policy association and the subsequent recreation racing against the next reconcile before the policy can be re-associated.

I'll have a PR shortly.

[richardcase]

/triage accepted
/lifecycle active
/assign raykrueger

[moko-poi]

There is a second, independent trigger for this loop that #6007 does not address: the Username comparison.

We hit the exact symptom described here, but with type: standard explicitly set, so the Type normalization in #6007 does not apply. The recreate decision in updateAccessEntry is:

cluster-api-provider-aws/pkg/cloud/services/eks/accessentry.go

Line 176 in 13af066

if *accessEntry.Type.APIValue() != *describeOutput.AccessEntry.Type || accessEntry.Username != existingUsername {

When username is omitted from the spec, accessEntry.Username is "" — it is a plain string with omitempty and no defaulting, and createAccessEntry skips setting it:

cluster-api-provider-aws/pkg/cloud/services/eks/accessentry.go

Lines 143 to 145 in 13af066

	if accessEntry.Username != "" {
	createInput.Username = &accessEntry.Username
	}

Meanwhile existingUsername is read verbatim from the AWS API:

cluster-api-provider-aws/pkg/cloud/services/eks/accessentry.go

Lines 170 to 174 in 13af066

	// EKS requires recreate when changing type or removing username
	existingUsername := ""
	if describeOutput.AccessEntry.Username != nil {
	existingUsername = *describeOutput.AccessEntry.Username
	}

EKS does not leave the username empty in that case. For an IAM role principal it generates one server-side, of the form arn:aws:sts::<account>:assumed-role/<role-name>/{{SessionName}}. So the comparison is "" against a non-empty generated value, and it holds on every reconcile, forever. There is no empty-string normalization for Username anywhere in the file, and #6007 leaves this half of the condition untouched — the diff only substitutes desiredType on the left-hand side.

This matches what we observe: access entries that omit username are recreated on every cycle, while otherwise identical entries that set username explicitly to the value EKS would have generated are left alone.

The presence of accessPolicies is not the discriminator, but it is what makes the bug user-visible. DeleteAccessEntry drops the policy associations, so principals authorized via accessPolicies lose access during each window, producing the intermittent Forbidden errors reported above. Principals authorized via kubernetesGroups would be unaffected even if their entries flapped.

A fix consistent with #6007 would be to treat an empty spec Username as unmanaged and exclude it from the recreate decision, along the lines of accessEntry.Username != "" && accessEntry.Username != existingUsername.

accessentry.go is byte-identical between v2.12.1, v2.13.0 and main, so this affects all current releases.

I am happy to open a follow-up PR for the Username branch if #6007 is intended to stay scoped to Type.