[flake] AWSManagedControlPlane spec changes can be silently dropped due to event coalescing

[damdo]

/kind bug

CI flake (upgrade policy mismatch: expected EXTENDED, but found STANDARD)

At the moment, there are a number of CI failures with same symptom: upgrade policy mismatch: expected EXTENDED, but found STANDARD, the controller never reconciled the UpgradePolicy spec change to EKS.

The [upgrade-policy] e2e test (pull-cluster-api-provider-aws-e2e-eks) has a ~50% flake rate due to this bug. Observed failures across recent CI runs:

Prow Job	PR	Result
2069407977344864256	#5857	FAILED
2069759758755172352	#5857	passed
2069788494531661824	#5857	passed
2070111813940285440	#5857	FAILED
2070152731791724544	#5857	passed
2070423170019495936	#5857	FAILED
2070448586679652352	#5857	FAILED
2070491476738969600	#5857	passed
2071593029604478976	#6084	FAILED
2071746726246486016	#6084	passed

So the following is an investigation to figure out why.

What steps did you take and what happened:

When multiple EKS clusters are reconciling concurrently (e.g. during e2e tests), the AWSManagedControlPlane controller's worker slots can all be occupied for 12+ minutes during initial EKS cluster creation. If a spec change (e.g. updating UpgradePolicy) is made shortly after the initial reconcile completes, the following race occurs:

1. The deferred scope.Close() patches status, generating a watch event that is enqueued in the controller-runtime work queue.
2. The spec change generates another watch event for the same object key.
3. Since all workers are busy, both events sit in the queue and are coalesced (controller-runtime deduplicates by key).
4. By the time a worker picks up the item, a later event (e.g. deletion) may have also been coalesced, and the spec change is never processed.

The controller returns reconcile.Result{}, nil on success with no RequeueAfter, so there is no mechanism to catch coalesced events.

This was observed as an intermittent failure in the [upgrade-policy] e2e test, where the UpgradePolicy spec change from STANDARD to EXTENDED was never reconciled to EKS.

What did you expect to happen:

Spec changes to AWSManagedControlPlane should always be reconciled, regardless of concurrent controller load or event timing.

Anything else you would like to add:

Root cause analysis from CI logs (Prow job 2071593029604478976 on PR #6084):

• The initial reconcile ran from 14:09:10 to 14:21:21 (~12 min), occupying one of 5 worker slots.
• scope.Close() patched status at 14:21:21, the test patched spec at 14:21:23.
• Zero AWSManagedControlPlane reconciles occurred between 14:21:21 and 14:33:17 (delete).
• All 5 worker slots were occupied by other long-running reconciles during this window (confirmed by a 6-minute gap with zero reconcile starts).

Environment:

• Cluster-api-provider-aws version: main (post v2.8.0)
• Kubernetes version: v1.33.x (e2e)
• OS: Linux (Prow CI)