[BUG] Ongoing problems with Docker-Image Repo

[the-hotmann]

Description

Related Issues/PRs:

• Error 500 when pushing a docker multiarch image #33010
• [ContainerRegistry] Internal server error on docker push <image_name> #31215
• Error 1062 (23000): Duplicate entry when using 'docker buildx imagetools create' in Actions #29563
• Manifest push fails when using Gitea container registry with Docker BuildKit caching #28973
• Optimize docker registry response code #33053
• Fix container parallel upload bugs #32022

@wxiaoguang thanks for your fix here: #33010 (comment)
But apparently I can not test it as I already am on "nightly", which is basically "1.24-nightly" and therefore newer than "1.23-nightly".

Here some info about my situation:

1. I use the current nightly build 1.24.0+dev-179-gf6dbf0e7b3
2. I use Debian Testing as Host

This is my docker-build command: (censored)

docker buildx build -f Dockerfile --pull --push --provenance=false \
    \
    --build-arg ALPINE_TAG="$TAG" \
    --platform linux/amd64/v1,linux/amd64/v2,linux/amd64/v3,linux/arm64/v8 \
    -t sub.domain.tld/repo/image:v${VERSION} \
    -t sub.domain.tld/repo/image \
    .

This is the error I get:

 > exporting to image:
3.174 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/n5iojjlsxfxsnobkf5g9gaae3?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
3.174 retrying in 1s
5.005 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/b5xta9fiofskiwdd0yy2nalna?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
5.005 retrying in 2s
7.815 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/fvt2wi7tsrbtbpsmtv3dohkjn?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
7.815 retrying in 4s
12.61 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/6emxck8jbooxwgju3vvatlruo?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error

First when I got this error, I navigated to https://sub.domain.tld/-/admin/packages and deleted the unreferenced packages. Then the upload worked again. I have to do this all 4-5 times I PUSH to the repo and this is very annoying, but since I use Gitea for my own, this is not the biggest problem.

Genenrally speaking Gitea is the coolest, cleanest, resource-saving Repo-Manager, with at the same time, the worst Docker-Image Repo I have ever tested. Since I set it up about half a year ago I just had trouble with it and the Docker-Image Repo was not working stable out of the box.

I would appreciate a fix for

Gitea Version

1.24.0+dev-179-gf6dbf0e7b3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

Debian Testing (12.9)

How are you running Gitea?

I run Gitea in a selfhosted docker-container.

Database

PostgreSQL

[the-hotmann]

I also figured:

When I PUSH very small layers (test-files in a seperate layer) it takes way longer to run into this problem.

[the-hotmann]

I also noticed, that if I push to: sub.domain.tld/repo/image and get the error unexpected status from PUT request. I can just delete the WHOLE "image" from Gitea and push again, then it works.

The problem here: deleting an Image/Package must be done seperately for EVERY layer .. which is unconventional. Beeing able to delete (and directly purge) a package should be something that makes handeling Docker-Images in Gitea much better.
(but this could be a seperate Feature-Request)

[wxiaoguang]

@wxiaoguang thanks for your fix here: #33010 (comment)

That fix is in 1.23.1 and main-nightly (1.24) now.

Other problems might be new ones (there are still many TODOs in the container package registry)

[the-hotmann]

Since I am running on nightly I assume I already have the fix?

[wxiaoguang]

Since I am running on nightly I assume I already have the fix?

nightly are built from different commits, so it depends on which nightly you are using, you can check it by the version if it is an official build.

I use the current nightly build 1.24.0+dev-179-gf6dbf0e7b3

f6dbf0e7b3 is the latest nightly, so it includes that fix.

---

And if you see 500 errors, usually there should be backend logs, please provide the Gitea's logs, it would help a lot.

[the-hotmann]

If you help me where I can find them, I will provide them.
I assumed they are in /data/gitea/log/, but there is nothing:

gitea:/data/gitea# ls -l
total 16
drwxr-xr-x    2 git      git              6 Nov 26 10:28 actions_artifacts
drwxr-xr-x    2 git      git              6 Nov 26 10:28 actions_log
drwxr-xr-x    2 git      git              6 Nov 26 10:28 attachments
drwxr-xr-x    3 git      git           4096 Nov 29 23:04 avatars
drwxr-xr-x    2 git      git             21 Nov 26 08:23 conf
drwxr-xr-x    2 git      git             24 Nov 26 10:28 home
drwx------    3 git      git             26 Jan 15 01:17 indexers
drwxr-xr-x    2 git      git             25 Nov 26 10:28 jwt
drwxr-xr-x    2 git      git              6 Nov 26 08:23 log
drwxr-xr-x  259 git      git           8192 Dec 11 23:05 packages
drwxr-xr-x    3 git      git             20 Nov 26 10:28 queues
drwxr-xr-x    2 git      git              6 Nov 26 10:28 repo-archive
drwxr-xr-x    2 git      git              6 Nov 26 10:28 repo-avatars
drwx------   18 git      git            150 Nov 29 22:58 sessions
drwxr-xr-x    3 git      git             28 Dec 14 20:45 tmp
gitea:/data/gitea# ls -la log/
total 4
drwxr-xr-x    2 git      git              6 Nov 26 08:23 .
drwxr-xr-x   17 git      git           4096 Nov 26 10:28 ..
gitea:/data/gitea#

I run giteea with this compose:

  gitea:
    image: gitea/gitea:nightly
    container_name: gitea
    hostname: gitea
    environment:
      USER_UID: "850"
      USER_GID: "850"
      GITEA__database__DB_TYPE: "postgres"
      GITEA__database__HOST: "postgres_17:5432"
      GITEA__database__NAME: "gitea"
      GITEA__database__USER: "gitea"
      GITEA__database__PASSWD: "CENSORED"
      DISABLE_REGISTRATION: true
    volumes:
      - "/var/docker/data/gitea/:/data/:rw"
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
    ports:
      - "127.0.0.1:3000:3000"
#      - "222:22"
    depends_on:
      - postgres_17
    deploy:
      resources:
        limits:
          memory: 2G
    restart: unless-stopped

  postgres_17:
    image: postgres:17-alpine
    container_name: postgres_17
    hostname: postgres_17
    environment:
      POSTGRES_PASSWORD: "CENSORED"
      POSTGRES_USER: "admin"
    logging:
      driver: "none"
    volumes:
      - "/var/docker/data/postgres_17/:/var/lib/postgresql/data/:rw"
    deploy:
      resources:
        limits:
          memory: 1G
    restart: unless-stopped

Just noticed I forgot the healthcheck and dependency with healthcheck, but this is very likely not the problem.

[wxiaoguang]

Use docker logs to collect logs

https://docs.gitea.com/help/support

[the-hotmann]

Indeed .. there was the hint.
I already checked these logs, but since they print so much (as the logs here are very verbose) I missed the small hints:

2025/01/15 16:33:00 ...ntainer/container.go:92:apiError() [E] write /data/gitea/tmp/package-upload/pbsgu8gap03bmz9zbzrkf60v0: no space left on device

I moved this docker-mount to another volume, now everything was able to be pushed.

I still have a last question:
How can I go back from nightly to stable?

Is there a chance of doing so? I was hoping the tool is also able to migrate down to lower versions, but this seems not to be the case.

Anyway thanks for your great help!

[wxiaoguang]

How can I go back from nightly to stable?

At the moment, you could manually update the "version" table to the 1.23's version 312 and use a 1.23 release. I have checked that it should work, but do not do so in the future unless you have confirmed the database is safe to be used for an old release.

[the-hotmann]

Ok thanks, I will do so.