peer-finder fails DNS lookup on K8s cluster on Azure AKS

[moksamedia]

I'm working on a project using a JHipster-based microservice with K8s hosted on Azure AKS. It's based on this tutorial but deployed to Azure AKS instead of GCP.

The store-mongodb deployment keeps hanging on the boot init container. I've traced the problem to the peer-finder tool. It doesn't seem to be able to resolve the store-mongodb service. The headless service is configured in the cluster, however.

The only thing I'm seeing in the boot init container logs is this message from peer-finder.

2022/03/25 18:37:43 lookup store-mongodb on 10.0.0.10:53: no such host                                                                                                                     
2022/03/25 18:37:44 lookup store-mongodb on 10.0.0.10:53: no such host                                                                                                                    
2022/03/25 18:37:45 lookup store-mongodb on 10.0.0.10:53: no such host                                                                                                                     
2022/03/25 18:37:46 lookup store-mongodb on 10.0.0.10:53: no such host                                                                                                                     
2022/03/25 18:37:48 lookup store-mongodb on 10.0.0.10:53: no such host        
...                                                                                                             

Any ideas? Thanks -- Andrew

andrewcarterhughes@pop-os: kubectl describe -f store-k8s/
Name:                   store
Namespace:              demo
CreationTimestamp:      Fri, 25 Mar 2022 11:34:34 -0700
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=store,version=v1
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=store
           version=v1
  Init Containers:
   init-ds:
    Image:      busybox:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      while true
      do
        rt=$(nc -z -w 1 store-mongodb 27017)
        if [ $? -eq 0 ]; then
          echo "DB is UP"
          break
        fi
        echo "DB is not yet reachable;sleep for 10s before retry"
        sleep 10
      done
      
    Environment:  <none>
    Mounts:       <none>
  Containers:
   store-app:
    Image:      andrewcarterhughes/store
    Port:       8082/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   512Mi
    Liveness:   http-get http://:http/management/health/liveness delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/management/health/readiness delay=20s timeout=1s period=15s #success=1 #failure=6
    Environment:
      SPRING_PROFILES_ACTIVE:                 prod
      SPRING_CLOUD_CONFIG_URI:                http://admin:${jhipster.registry.password}@jhipster-registry.demo.svc.cluster.local:8761/config
      JHIPSTER_REGISTRY_PASSWORD:             <set to the key 'registry-admin-password' in secret 'registry-secret'>  Optional: false
      EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE:  http://admin:${jhipster.registry.password}@jhipster-registry.demo.svc.cluster.local:8761/eureka/
      SPRING_DATA_MONGODB_DATABASE:           store
      SPRING_DATA_MONGODB_URI:                mongodb://store-mongodb-0.store-mongodb.demo:27017/?waitQueueMultiple=1000,store-mongodb-1.store-mongodb.demo:27017/?waitQueueMultiple=1000,store-mongodb-2.store-mongodb.demo:27017/?waitQueueMultiple=1000
      SPRING_SLEUTH_PROPAGATION_KEYS:         x-request-id,x-ot-span-context
      JAVA_OPTS:                               -Xmx256m -Xms256m
      SERVER_SHUTDOWN:                        graceful
    Mounts:                                   <none>
  Volumes:                                    <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   store-d5c77575f (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  5m14s  deployment-controller  Scaled up replica set store-d5c77575f to 1


Name:         store-mongodb-config
Namespace:    demo
Labels:       <none>
Annotations:  <none>

Data
====
mongod.conf:
----
net:
  port: 27017
replication:
  replSetName: rs0
storage:
  dbPath: /data/db


BinaryData
====

Events:  <none>


Name:              store-mongodb
Namespace:         demo
Labels:            <none>
Annotations:       service.alpha.kubernetes.io/tolerate-unready-endpoints: true
Selector:          app=store-mongodb
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              peer  27017/TCP
TargetPort:        27017/TCP
Endpoints:         10.244.2.41:27017
Session Affinity:  None
Events:            <none>


Name:         store-mongodb-init
Namespace:    demo
Labels:       <none>
Annotations:  <none>

Data
====
on-start.sh:
----
script_name=${0##*/}

log() {
    local msg="$1"
    local timestamp
    timestamp=$(date --iso-8601=ns)
    echo "[$timestamp] [$script_name] $msg" >> /work-dir/log.txt
}

shutdown_mongo() {
    if [[ $# -eq 1 ]]; then
        args="timeoutSecs: $1"
    else
        args='force: true'
    fi
    log "Shutting down MongoDB ($args)..."
    mongo admin "${admin_creds[@]}" "${ssl_args[@]}" --eval "db.shutdownServer({$args})"
}

my_hostname=$(hostname)
log "Bootstrapping MongoDB replica set member: $my_hostname"

log "Reading standard input..."
while read -ra line; do
    log "Line: ${line}"
    if [[ "${line}" == *"${my_hostname}"* ]]; then
        log "service_name ${line}"
        service_name="$line"
        continue
    fi
    peers=("${peers[@]}" "$line")
done

log "Peers: ${peers[*]}"

log "Generating the cert"
# Generate the ca cert
ca_crt=/data/configdb/tls.crt
if [ -f "$ca_crt"  ]; then
    log "Generating certificate"
    ca_key=/data/configdb/tls.key
    pem=/work-dir/mongo.pem
    ssl_args=(--ssl --sslCAFile "$ca_crt" --sslPEMKeyFile "$pem")

# Move into /work-dir
pushd /work-dir

cat >openssl.cnf <<EOL
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = $(echo -n "$my_hostname" | sed s/-[0-9]*$//)
DNS.2 = $my_hostname
DNS.3 = $service_name
DNS.4 = localhost
DNS.5 = 127.0.0.1
EOL

    # Generate the certs
    openssl genrsa -out mongo.key 2048
    openssl req -new -key mongo.key -out mongo.csr -subj "/CN=$my_hostname" -config openssl.cnf
    openssl x509 -req -in mongo.csr \
        -CA "$ca_crt" -CAkey "$ca_key" -CAcreateserial \
        -out mongo.crt -days 3650 -extensions v3_req -extfile openssl.cnf

    rm mongo.csrl
    cat mongo.crt mongo.key > $pem
    rm mongo.key mongo.crt
fi


log "Peers: ${peers[*]}"

log "Starting a MongoDB instance..."
mongod --config /data/configdb/mongod.conf --dbpath="$DATA_PATH" --replSet="$REPLICA_SET" --port=$PORT "${auth_args[@]}" --bind_ip=0.0.0.0 >> /work-dir/log.txt 2>&1 &

log "Waiting for MongoDB to be ready..."
until mongo "${ssl_args[@]}" --eval "db.adminCommand('ping')"; do
    log "Retrying..."
    sleep 2
done

log "Initialized."

# try to find a master and add yourself to its replica set.
for peer in "${peers[@]}"; do
    if mongo admin --host "$peer" "${admin_creds[@]}" "${ssl_args[@]}" --eval "rs.isMaster()" | grep '"ismaster" : true'; then
        log "Found master: $peer"
        log "Adding myself ($service_name) to replica set..."
        if mongo admin --host "$peer" "${admin_creds[@]}" "${ssl_args[@]}" --eval "rs.add('$service_name')" | grep 'Quorum check failed'; then
            log 'Quorum check failed, unable to join replicaset. Exiting prematurely.'
            shutdown_mongo
            exit 1
        fi

        sleep 3

        log 'Waiting for replica to reach SECONDARY state...'
        until printf '.' && [[ $(mongo admin "${admin_creds[@]}" "${ssl_args[@]}" --quiet --eval "rs.status().myState") == '2' ]]; do
            sleep 1
        done

        log '✓ Replica reached SECONDARY state.'

        shutdown_mongo "60"
        log "Good bye."
        exit 0
    fi
done

# else initiate a replica set with yourself.
if mongo "${ssl_args[@]}" --eval "rs.status()" | grep "no replset config has been received"; then
    log "Initiating a new replica set with myself ($service_name)..."
    mongo "${ssl_args[@]}" --eval "rs.initiate({'_id': '$REPLICA_SET', 'members': [{'_id': 0, 'host': '$service_name'}]})"

    sleep 3

    log 'Waiting for replica to reach PRIMARY state...'
    until printf '.' && [[ $(mongo "${ssl_args[@]}" --quiet --eval "rs.status().myState") == '1' ]]; do
        sleep 1
    done

    log '✓ Replica reached PRIMARY state.'

    log "Done."
fi

shutdown_mongo
log "Good bye."


BinaryData
====

Events:  <none>


Name:               store-mongodb
Namespace:          demo
CreationTimestamp:  Fri, 25 Mar 2022 11:34:37 -0700
Selector:           app=store-mongodb
Labels:             <none>
Annotations:        <none>
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=store-mongodb
  Init Containers:
   config:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
    Args:
      -c
      set -e
      set -x
      cp /configdb-readonly/mongod.conf /data/configdb/mongod.conf
      
    Environment:  <none>
    Mounts:
      /configdb-readonly from config (rw)
      /data/configdb from configdir (rw)
      /work-dir from workdir (rw)
   install:
    Image:      k8s.gcr.io/mongodb-install:0.6
    Port:       <none>
    Host Port:  <none>
    Args:
      --work-dir=/work-dir
    Environment:  <none>
    Mounts:
      /work-dir from workdir (rw)
   boot:
    Image:      mongo:4.4.12
    Port:       <none>
    Host Port:  <none>
    Command:
      /work-dir/peer-finder
    Args:
      -on-start=/init/on-start.sh
      -service=store-mongodb
    Environment:
      POD_NAMESPACE:   (v1:metadata.namespace)
      REPLICA_SET:    rs0
      DATA_PATH:      /data/db
      PORT:           27017
    Mounts:
      /data/configdb from configdir (rw)
      /data/db from datadir (rw)
      /init from init (rw)
      /work-dir from workdir (rw)
  Containers:
   mongodb:
    Image:      mongo:4.4.12
    Port:       27017/TCP
    Host Port:  0/TCP
    Command:
      mongod
    Args:
      --config=/data/configdb/mongod.conf
      --dbpath=$(DATA_PATH)
      --replSet=$(REPLICA_SET)
      --port=$(PORT)
      --bind_ip=0.0.0.0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   512Mi
    Liveness:   exec [mongo --eval db.adminCommand('ping')] delay=30s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [mongo --eval db.adminCommand('ping')] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:   (v1:metadata.namespace)
      REPLICA_SET:    rs0
      DATA_PATH:      /data/db
      PORT:           27017
    Mounts:
      /data/configdb from configdir (rw)
      /data/db from datadir (rw)
      /work-dir from workdir (rw)
  Volumes:
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      store-mongodb-config
    Optional:  false
   workdir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   init:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      store-mongodb-init
    Optional:  false
   configdir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Volume Claims:
  Name:          datadir
  StorageClass:  
  Labels:        <none>
  Annotations:   <none>
  Capacity:      1Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age    From                    Message
  ----    ------            ----   ----                    -------
  Normal  SuccessfulCreate  5m13s  statefulset-controller  create Pod store-mongodb-0 in StatefulSet store-mongodb successful


Name:              store
Namespace:         demo
Labels:            app=store
Annotations:       <none>
Selector:          app=store
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.38.241
IPs:               10.0.38.241
Port:              http  8082/TCP
TargetPort:        8082/TCP
Endpoints:         
Session Affinity:  None
Events:            <none>