CoreDNS v1.14.3 image on is amd64-only

[hakman]

registry.k8s.io/coredns/coredns:v1.14.3 was promoted as a single linux/amd64 image manifest instead of a multi-arch manifest list. On any non-amd64 host the CoreDNS container fails to start:

exec /coredns: exec format error

This breaks arm64/arm/ppc64le/s390x/riscv64 consumers of the image (kubeadm clusters, kOps, etc.) — CoreDNS goes CrashLoopBackOff and cluster DNS is down.

Evidence

Image	Media type	Platforms
registry.k8s.io/coredns/coredns:v1.14.3	manifest.v2+json (single)	amd64 only ❌
registry.k8s.io/coredns/coredns:v1.14.2	manifest.list.v2+json	all 6 ✅
gcr.io/k8s-staging-coredns/coredns:v1.14.3	manifest.list.v2+json	all 6 ✅
docker.io/coredns/coredns:1.14.3	manifest.list.v2+json	all 6 ✅

$ crane manifest registry.k8s.io/coredns/coredns:v1.14.3 | jq .mediaType
"application/vnd.docker.distribution.manifest.v2+json"        # single amd64 manifest

$ crane manifest gcr.io/k8s-staging-coredns/coredns:v1.14.3 | jq .mediaType
"application/vnd.docker.distribution.manifest.list.v2+json"   # correct multi-arch list

• Promoted (broken) digest: sha256:884b72dd6d2f7d367902af420605e0288dffedb0516ce29330423ae3f8f5c6fa — the linux/amd64 child manifest
• Correct manifest-list digest (staging + Docker Hub): sha256:b21d26b915e10acb5bc78715c1e8b6047ab2675389b2bcc18b3a6499d90e74c0

Root cause

The build/staging image is fine — the multi-arch list exists in gcr.io/k8s-staging-coredns. The promotion PR #9395 recorded the amd64 child digest in images.yaml rather than the index digest, so the promoter copied only the amd64 manifest.

Attempts to fix

The in-place fix was attempted in #9421 and closed, registry.k8s.io tags are immutable and the promoter will not re-point an already-promoted tag (per @dims). A new version tag is the only way to land a correct multi-arch image on registry.k8s.io.

[hakman]

I believe we should just point the staging registry to docker.io/coredns/coredns and stop promoting random images.

CC @upodroid @ameukam @dims

[dims]

@hakman one issue will be our ci systems pointing to docker.io which will turn into throttling/timeout issues etc.

[hakman]

@hakman one issue will be our ci systems pointing to docker.io which will turn into throttling/timeout issues etc.

@dims that's true, but how bad would it be? how often would it actually query docker.io and would it auto recover?

[dims]

@hakman scalability and other sensitive jobs .. are the issue

[ameukam]

I believe we should just point the staging registry to docker.io/coredns/coredns and stop promoting random images.

CC @upodroid @ameukam @dims

I don't think that's our decision to make ?

[dims]

I don't think that's our decision to make ?

agree!

[hakman]

Not disagreeing, but who would be the person/group to decide this?
Also, who can make a policies regarding registry mirrors via registry.k8s.io?

[neolit123]

we should revert k/k to 1.14.<old>

@tico88612 @yashsingh74

The promotion PR https://giththis ub.com//pull/9395 recorded the amd64 child digest in images.yaml rather than the index digest, so the promoter copied only the amd64 manifest.

can this be caught in presubmits on a PR?

[ameukam]

Also, who can make a policies regarding registry mirrors via registry.k8s.io?

k8s infra.

[hakman]

can this be caught in presubmits on a PR?

@neolit123 this should not be caught in presubmits, the source for the promotion should be canonical source, same as we do with etcd, and any error will be caught by presubmits:

k8s.io/registry.k8s.io/manifests/k8s-staging-etcd/promoter-manifest.yaml

Lines 10 to 12 in bffad75

	registries:
	- name: gcr.io/etcd-development
	src: true

k8s infra

Thanks @ameukam, so we can propose a policy to no accept copies of non-owned registries as source. Promoting something broken that cannot be changed later (once promoted, it's forever), with no control to release a new version to fix it is not exactly ideal.

[neolit123]

can we block somewhere so that a bogus / partial image is not promoted? at least for the "core" ones like coredns and etcd.

[hakman]

can we block somewhere so that a bogus / partial image is not promoted? at least for the "core" ones like coredns and etcd.

we can't because the promoter looks at the configured staging registry, which people manually update right now.

[upodroid]

This was one of the reasons why we updated the PR template for this repo reminding users to double check their checksums, ensure images are multi-arch and that the registry is immutable.

[neolit123]

there has to be a presubmit validation for similar problems...users don't read PR templates and/or forget to do their local checks.

we have an ancient e2e periodic that catches missing arches from the "core" image manifest list, inc coredns and etcd:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kubernetes-e2e-manifest-lists/2056662344716521472

required architectures:
amd64, arm, arm64, ppc64le, s390x

* getFromURL(): https://api.github.com/repos/kubernetes/kubernetes/releases
WARNING: could not covert 'Content-Length' "" to integer
* downloaded the following releases from GitHub:
1.36.1, 1.36.0, 1.36.0-rc.1, 1.36.0-rc.0, 1.36.0-beta.0, 1.36.0-alpha.2, 1.36.0-alpha.1, 1.35.5, 1.35.4, 1.35.3, 1.35.2, 1.35.1, 1.35.0, 1.35.0-rc.1, 1.35.0-rc.0, 1.34.8, 1.34.7, 1.34.6, 1.34.5, 1.34.4, 1.34.3, 1.33.12, 1.33.11, 1.33.10, 1.33.9, 1.33.8, 1.33.7, 1.32.13, 1.32.12, 1.32.11

* testing the following list of releases that should support manifest lists:
1.36.1, 1.36.0, 1.36.0-rc.1, 1.36.0-rc.0, 1.36.0-beta.0, 1.36.0-alpha.2, 1.36.0-alpha.1, 1.35.5, 1.35.4, 1.35.3, 1.35.2, 1.35.1, 1.35.0, 1.35.0-rc.1, 1.35.0-rc.0, 1.34.8, 1.34.7, 1.34.6, 1.34.5, 1.34.4, 1.34.3

we should revert k/k to 1.14.

looks like 1.14.3 was reverted:

• Revert "bump coredns to v1.14.3" kubernetes#138714

so we are good for now.