CachedImage not created

[bpoland]

Hey there, I have a weird situation where one of my images was getting rewritten but no CachedImage or Repository were getting automatically created for it. I have looked through the controller logs for any hints but am not seeing anything obvious. The CachedImage and Repository are being created for other images in the same repo but not this one.

The only thing I could think of is that the problematic image name and tag are longer than others, so I tried manually creating the Cachedimage and Repository to see if I got any errors, but it worked fine and now my Pods are starting.

Has anyone seen anything like this before? Are there specific log lines I can look at in the controller log to try to troubleshoot? Thank you!

[plaffitt]

Hello,

I would need more information in order to help you. Could you please provide:

• Relevant logs (specifically the pod controller logs, which is in charge of creating CachedImages). You can enable debugging logs with .controllers.verbosity: DEBUG to get more details.
• A minimal reproducible example of a Pod producing this behavior.
• Any custom configuration in your values.yaml.

Thanks for reporting!

[bpoland]

Ah thank you, I will enable the higher verbosity and try to reproduce in a test cluster in the next few days.

[bpoland]

Hi just want to circle back here, sorry for the delay! This continues to happen to us occasionally but when it does happen, enabling debug logs restarts the controllers and as soon as they restart then they notice there should be a CachedImage and create it. So I guess we could try leaving it in debug mode for a while to try to reproduce but that might be too noisy.

The latest time it was just a simple nginx image that was failing. When this happens, the pod goes into ImagePullBackOff because the rewrite happens to add localhost:7439 but then the local registry doesn't have the image and 404s :(

  Normal   Pulling    24s (x2 over 42s)  kubelet            Pulling image "localhost:7439/public.ecr.aws/nginx/nginx:stable-alpine"
  Warning  Failed     24s (x2 over 42s)  kubelet            Failed to pull image "localhost:7439/public.ecr.aws/nginx/nginx:stable-alpine": reading manifest stable-alpine in localhost:7439/public.ecr.aws/nginx/nginx: StatusCode: 404, ""
  Warning  Failed     24s (x2 over 42s)  kubelet            Error: ErrImagePull
  Normal   BackOff    12s (x2 over 42s)  kubelet            Back-off pulling image "localhost:7439/public.ecr.aws/nginx/nginx:stable-alpine"
  Warning  Failed     12s (x2 over 42s)  kubelet            Error: ImagePullBackOff

In terms of custom stuff in our helm values, I don't think there's anything too crazy:

cachedImagesExpiryDelay: 7

controllers:
  webhook:
    acceptedImages:
      - "docker.io/.*"
      - "public.ecr.aws/.*"
      ... some private repos
    ignorePullPolicyAlways: false
  resources:
    requests:
      cpu: "500m"
      memory: "500Mi"
    limits:
      cpu: "2"
      memory: "1Gi"

proxy:
  hostNetwork: true

registry:
  garbageCollection:
    schedule: "0 0 * * *" # daily
  imagePullSecrets:
    - name: regcred
  replicas: 3
  pdb:
    create: true

minio:
  enabled: true
  global:
    imagePullSecrets:
      - name: regcred
  metrics:
    enabled: true
  persistence:
    storageClass: high-io
    size: 50Gi
  statefulset:
    zones: 1