Kubectl drain errors on test

[paulgmiller]

We're using GracefulShutdownTimeout: -1 so our webhook should drain but either
It could be konnectivity or kube-proxy not updating fast enough. I think it its not connectivity because I see that being evicted well before the error.

So think I'll try holdign sigterm for a while so kube-proxy can update? But still a mystery here.

rror when evicting pods/"store-front-bfb8f484c-mwxd4" -n "pets": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF
error when evicting pods/"makeline-service-5f6dcd9d4d-rpv8t" -n "pets": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF
error when evicting pods/"controller-manager-779cf5b64b-nc6kz" -n "default": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF
error when evicting pods/"store-admin-8f6db4878-4ftjb" -n "pets": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF
error when evicting pods/"rabbitmq-0" -n "pets": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF
error when evicting pods/"mongodb-0" -n "pets": Internal error occurred: failed calling webhook "eviction.mydomain.com": failed to call webhook: Post "https://eviction-webhook.default.svc:443/validate-eviction?timeout=10s": EOF

[paulgmiller]

Turns out the konnectivity agent eviction seems to cause this. If I exclude it from a node drain I don't reproduce. (used #39 to prove this)

[paulgmiller]

Readiness now checks api sever connection
kubernetes-sigs/apiserver-network-proxy#491
kubernetes-sigs/apiserver-network-proxy#485

[paulgmiller]

Ah maybe here kubernetes-sigs/apiserver-network-proxy#566

[paulgmiller]

Probably can't fix this. Depriotized as we're moving away from webhook. but still a little sad that any eviction webhook would cause these failures in kubectl drain but they do work on retry.