While scale up master node the etcd job is getting failed

[anuya9]

Hello Team,

We have deployed kuberenetes cluster with 3 master and 3 worker nodes. We were trying to scale up the master node from 3 to 4 and deploy it through bosh-manifest.
The deployment completed with error mentioned below:

Error: Action Failed get_task: Task 1ac12df5-dc01-47d3-46b5-42d4d749cad0 result: 1 of 4 post-start scripts failed. Failed Jobs: etcd. Successful Jobs: bosh-dns, kube-apiserver, kubernetes-roles.

All the VMs were in running state.
We checked the health of cluster by running ./etcdctl cluster-health,so it was observed that the newly added master node [master-3] is unhealthy and health check failed with error mentioned below:
failed to check the health of member 56f3d8fb2ca0cde1 on https://master-3.xxxxx.xxxxx.internal:2379: Get https://master-3.xxxxx.xxxxx.internal:2379/health: net/http: TLS handshake timeout
member 56f3d8fb2ca0cde1 is unreachable: [https://master-3.xxxxx.xxxxx.internal:2379] are all unreachable
cluster is degraded.

We are deploying cluster through cfcr deployment over bosh.
Bosh Version: 268.8
Kubernetes Version: 1.15.5
Ubuntu Version: 16.04.6
AWS ec2 instance

Below is the link that we are using to deploy our Kubernetes cluster:
https://github.com/cloudfoundry-incubator/kubo-release
https://github.com/cloudfoundry-incubator/cfcr-etcd-release

Anyone has idea about this issue and how can it be fixed?

[svrc]

This can be mitigated by using the etcdctl command to remove the member and re-adding it.
It has been resolved in CFCR etcd release 1.12.4 by using an older version of the etcdctl CLI.