Example Task Steps referencing images by tag resulting in throttling during e2e tests

[aThorp96]

The end to end tests can be quite flaky, and often due to ImgPullBackoff errors. In the majority of cases though, the image is not unavailable, instead the image puller is being rate limited (either from the image server or client-side) and eventually fails before the throttling is overcome. Client and Server side throttling can be mitigated by ensuring Task Step images in our e2e tests (i.e., the /examples directory) are frozen to a particular digest. Server-side throttling can be mitigated by using a pull-through image proxy cache. However it's worth noting that github container registry does not document any server side rate limiting.

I see three ways we can address the above:

Option 1: Freeze all images to their current tag's digest or their current latest digest

Pros:

• Very simple
• Little work required to migrate
• Mitigates both server-side and client-side rate limiting

Cons:

• Maintenance: Image digests may need to be periodically updated as older images could be deprecated/deleted
• It may become unclear which images require a specific version vs which images are just frozen for optimization

Option 2: Run a pre-test script which resolves the image references before we execute e2e tests

(this would run in CI and the resolved references would not need to be committed into the repo)
Pros:

• No additional image update maintenance required
• Examples can continue to freeze their images when necessary or leave without tags when not
• Mitigates both server-side and client-side rate limiting
• Heckin cool

Cons:

• Added complexity to the CI, an additional failure mode
• Script would need to be maintained
• Implementation more involved

Option 3: Image proxy cache (either per e2e-test cluster or global)

Pros:

• No additional image update maintenance required
• Examples can continue to freeze their images when necessary or leave without tags when not
• A more standard solution
• Mitigates server-side throttling

Cons:

• Added complexity to CI setup, as the registry would need to be setup
• Would not mitigate client-side throttling
• Mainly valuable if all e2e test clusters can share the registry, which might be a challenge to configure with Github workflow job networking

[aThorp96]

cc @vdemeester

[afrittoli]

For option 3, I set that up for the catalog job, but it's one proxy per job, which means the benefit is rather limited.
The setup is rather straightforward though, since we already run a registry next to the kind cluster:

https://github.com/tektoncd/catalog/blob/daa6f8bae226266fb8508aac240da1a8ebdb7c11/hack/setup-kind.sh#L174-L176

[afrittoli]

This is a list of images extracted from the examples with a simple script (grin 'image: ' -d "**/no-ci/**" | sed -n 's/.*image: //p' | sort -u)

$(params.BUILDER_IMAGE)
$(params.image)
alpine
alpine:latest
bash:latest
busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0
cgr.dev/chainguard/busybox
docker
docker.io/alpine/git:v2.26.2
docker.io/amd64/ubuntu
docker@sha256:74e78208fc18da48ddf8b569abe21563730845c312130bd0f0b059746a7e10f5
gcr.io/cloud-builders/git
gcr.io/go-containerregistry/crane
ghcr.io/tektoncd-catalog/git-clone:v1.1.0
ghcr.io/tektoncd/catalog/upstream/tasks/skopeo-copy:latest
ghcr.io/tektoncd/pipeline/nop-8eac7c133edad5df719dc37b36b62482:latest
ghcr.io/tektoncd/plumbing/alpine-git-nonroot:latest
mcr.microsoft.com/windows/nanoserver:1809
mirror.gcr.io/alpine
mirror.gcr.io/bash
mirror.gcr.io/bash:5.1.4@sha256:c523c636b722339f41b6a431b44588ab2f762c5de5ec3bd7964420ff982fb1d9
mirror.gcr.io/busybox
mirror.gcr.io/node:lts-alpine3.20
mirror.gcr.io/perl:devel-bullseye
mirror.gcr.io/python:3.12.4
mirror.gcr.io/ubuntu
python:latest
ubuntu

Apart from the parametrised ones, which needs to be checked, a lot of images already don't come from dockerhub.
It probably possible to converge the various base, alpine and busybox images to a single image not ok docker hub, which should reduce the problem with the rate limiting.
For the golang tests, the grepping is not 100% effective, but still it gives a rough idea:

➜ grin 'image: ' -d "**/no-ci/**" | sed -n 's/.*image: //p' | sort -u
" + archSpecificImage)
"{{ .Values.image.repository }}:{{ .Values.image.tag }}"
"docker.io/gitea/gitea:1.17.1"
"mirror.gcr.io/ubuntu"
%s
%s", err)
%v
%v", err)
<name>" pattern.
alpine:3.15.1
busybox
docker.io/alpine/curl
docker.io/bitnami/memcached:1.6.9-debian-10-r114
docker.io/bitnami/postgresql:11.11.0-debian-10-r62
gcr.io/cloud-builders/curl
gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
gcr.io/tekton-releases/dogfooding/skopeo:latest
ghcr.io/spiffe/spiffe-csi-driver:nightly
ghcr.io/spiffe/spire-agent:1.1.1
ghcr.io/spiffe/spire-server:1.1.1
ko://github.com/tektoncd/pipeline/test/wait-task-beta/cmd/controller
ko://resolver_with_timeout/resolver
mcr.microsoft.com/powershell:nanoserver
mcr.microsoft.com/windows/nanoserver:1809
mirror.gcr.io/alpine
mirror.gcr.io/alpine/git:v2.26.2
mirror.gcr.io/bash
mirror.gcr.io/busybox
mirror.gcr.io/node:lts-alpine3.20
mirror.gcr.io/perl:devel-bullseye
mirror.gcr.io/python:3.12.4
mirror.gcr.io/ubuntu
quay.io/k8scsi/csi-node-driver-registrar:v2.0.1

[vdemeester]

I think we need a mix of solutions.
Even if the proxy is one per job, I think it could help (less external network access).
But making sure we use a smaller set of images would be good as well (especially running things locally).

In at some point we had images pre-fetched somewhere and we would populate it at start of e2e tests, so that it would make everything quicker and less dependent on network.