Workload cluster upgrade fails when adding new control plane nodes with stacked etcd during GitOps upgrades (v0.23.3→v0.24.6 and v0.24.6→v0.25.2)

[jmorenoc-o]

What happened

We are running EKS Anywhere on vSphere with a dedicated management cluster and multiple workload clusters managed through GitOps.

Our upgrade path was:

• EKS Anywhere v0.23.3 + Kubernetes v1.33 → EKS Anywhere v0.24.6 + Kubernetes v1.34
• EKS Anywhere v0.24.6 + Kubernetes v1.34 → EKS Anywhere v0.25.2 + Kubernetes v1.35

We followed the upgrade strategy recommended in the documentation:

1. Upgrade one EKS Anywhere minor version at a time.
2. Upgrade one Kubernetes minor version at a time.
3. Upgrade the management cluster first.
4. Upgrade workload clusters afterwards.

The management cluster was upgraded using the CLI tool:

eksctl anywhere upgrade cluster

Workload clusters were upgraded through GitOps by updating the cluster specifications.

During both upgrade cycles, some workload clusters failed while replacing control plane nodes.

The failure occurs after the new VM is provisioned and after the node has already joined the cluster, but before it successfully completes the control plane join workflow.

The error observed on the new control plane node is:

Error running bootstrapper cmd: error joining controlplane:
Error running command:
/opt/bin/kubeadm -v9 join --skip-phases preflight,control-plane-prepare,kubelet-start,control-plane-join/etcd,kubelet-wait-bootstrap --config /tmp/kubeadm-join-config.yaml:
exit status 1

The upgrade then becomes blocked waiting for the new control plane node to become ready.

---

What you expected to happen

The newly provisioned control plane node should successfully complete the kubeadm join control plane workflow and become a healthy control plane member automatically, allowing the rolling upgrade to continue without manual intervention.

---

How to reproduce it (as minimally and precisely as possible)

Environment:

• Provider: vSphere
• Management cluster upgraded via CLI
• Workload clusters upgraded through GitOps
• Bottlerocket nodes

Observed behavior across clusters:

Cluster	Topology	Upgrade Method	Result
Management Cluster	3 Control Planes + 3 External ETCD + 3 Workers	CLI	Success
Workload w-01	3 Control Planes + 3 External ETCD + 3 Workers	GitOps	Success
Workload w-02	3 Control Planes (stacked/integrated etcd) + 3 Workers	GitOps	Failed
Workload w-03	1 Control Plane (stacked/integrated etcd) + 3 Workers	GitOps	Failed

The issue appears to affect only clusters using stacked/integrated etcd.

We have not observed this problem on clusters using external ETCD.

The same behavior was reproduced during both upgrade paths:

• v0.23.3 / Kubernetes 1.33 → v0.24.6 / Kubernetes 1.34
• v0.24.6 / Kubernetes 1.34 → v0.25.2 / Kubernetes 1.35

---

Anything else we need to know?

We were able to recover and complete the upgrade manually.

The newly created node had already joined the cluster as a Kubernetes node but never completed the control plane bootstrap process.

The following procedure consistently allowed us to recover the failed control plane node and complete the upgrade.

1. SSH into the new control plane node

ssh -i ~/.ssh/eksa-ed25519 ec2-user@172.16.119.116

sudo sheltie

2. Wait until the kubeadm bootstrap container fails

systemctl is-failed host-containers@kubeadm-bootstrap.service

Wait until the service reports:

failed

3. Retrieve the kubeadm-bootstrap container image

IMAGE_ID=$(apiclient get | apiclient exec admin jq -r '.settings["host-containers"]["kubeadm-bootstrap"].source')

ctr image pull ${IMAGE_ID}

4. Generate a new bootstrap token

The following command prints a valid join command that will be useful later to obtain:

• API Server endpoint
• Bootstrap token
• CA certificate hash

ctr run \
  --net-host \
  --mount type=bind,src=/var/lib/kubeadm,dst=/var/lib/kubeadm,options=rbind:rw \
  --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:rw \
  --mount type=bind,src=/tmp,dst=/tmp,options=rbind:rw \
  --mount type=bind,src=/proc,dst=/proc,options=rbind:ro \
  --rm ${IMAGE_ID} tmp-kubeadm-token-create \
  /opt/bin/kubeadm token create \
    --ttl 2h \
    --print-join-command \
    --kubeconfig /var/lib/kubeadm/admin.conf

Example output:

kubeadm join 172.16.119.16:6443 \
  --token 1435b7.2iexxxxxxxxxxxxxx \
  --discovery-token-ca-cert-hash sha256:3c90fxxxxxxxxxxxxxxxxxxxxxxxx

5. Obtain node-specific values

apiclient get settings.network.hostname

apiclient get settings.kubernetes.node-ip

6. Create a JoinConfiguration

Using the values collected previously, create:

cat <<EOF >/tmp/kubeadm-join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: 172.16.119.16:6443
    caCertHashes:
    - sha256:3c90f...
    token: 1435b7.2ie...
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  kubeletExtraArgs:
  - name: anonymous-auth
    value: "false"
  - name: cloud-provider
    value: external
  - name: read-only-port
    value: "0"
  - name: tls-cipher-suites
    value: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  name: w01-kdl2q
  taints: null
controlPlane:
  localAPIEndpoint:
    advertiseAddress: 172.16.119.116
    bindPort: 6443
EOF

7. Create the admin.conf symlink

ln -sf /var/lib/kubeadm/admin.conf /etc/kubernetes/admin.conf

8. Execute the control plane join manually

ctr run \
  --net-host \
  --mount type=bind,src=/var/lib/kubeadm,dst=/var/lib/kubeadm,options=rbind:rw \
  --mount type=bind,src=/var/lib/etcd,dst=/var/lib/etcd,options=rbind:rw \
  --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:rw \
  --mount type=bind,src=/etc/kubernetes/manifests,dst=/etc/kubernetes/manifests,options=rbind:rw \
  --mount type=bind,src=/tmp,dst=/tmp,options=rbind:rw \
  --mount type=bind,src=/proc,dst=/proc,options=rbind:ro \
  --rm ${IMAGE_ID} tmp-cert-renew \
  /opt/bin/kubeadm -v9 join \
    --skip-phases preflight,check-etcd,control-plane-prepare,kubelet-start,etcd-join,wait-control-plane,kubelet-wait-bootstrap \
    --config /tmp/kubeadm-join-config.yaml

After executing the command above, the node successfully joined as a control plane member and the upgrade process completed normally.

Additional notes

I am not a Cluster API expert, so I cannot guarantee that the selected --skip-phases list is the most appropriate one.

These are simply the phases that consistently worked for us across multiple clusters and during two separate upgrade cycles.

It is possible that there is a cleaner or more appropriate recovery procedure.

However, the fact that the issue only appears on clusters using stacked/integrated etcd and can be consistently bypassed through a manual kubeadm join makes me suspect there may be a problem in the automated control plane bootstrap/join workflow during workload cluster upgrades.

If someone encounters the same issue, I would recommend validating the proposed workaround in a non-production environment first.

---

Environment

• Infrastructure Provider: vSphere
• OS: Bottlerocket

Upgrade Path #1

• EKS Anywhere Release:

  • Source: v0.23.3
  • Target: v0.24.6

• EKS Distro Release:

  • Kubernetes v1.33
  • Kubernetes v1.34

Upgrade Path #2

• EKS Anywhere Release:

  • Source: v0.24.6
  • Target: v0.25.2

• EKS Distro Release:

  • Kubernetes v1.34
  • Kubernetes v1.35