--mount=type=cache permissions issue with non-root users

[jonner]

I am trying to enable a cache mount in my container build where the container user is non-root, and I was hoping that I could write the containerfile in such a way that it would work with both docker and podman. I have constructed the following minimal example to show the issue:

./Containerfile

FROM registry.access.redhat.com/ubi9/go-toolset:1.21.11-9 AS builder
WORKDIR /hello-world
COPY --chown=1001:0 ./ ./
ENV GOCACHE=/go-cache
RUN --mount=type=cache,target=${GOCACHE} go build main.go

The file main.go is just a basic 'hello world' program written in go.
Building with podman results in the following:

$ podman build -f Containerfile .
[...SNIP ]
STEP 5/5: RUN --mount=type=cache,target=${GOCACHE} go build main.go
failed to initialize build cache at /go-cache: mkdir /go-cache/00: permission denied
Error: building at STEP "RUN --mount=type=cache,target=${GOCACHE} go build main.go": while running runtime: exit status 1

Building with docker results in the following error:

$ sudo docker build -f ./Containerfile .
[...SNIP]
 => ERROR [builder 4/4] RUN --mount=type=cache,target=/go-cache go build main.go                                                       0.1s
------                                                                                                                                      
 > [builder 4/4] RUN --mount=type=cache,target=/go-cache go build main.go:
0.117 failed to initialize build cache at /go-cache: mkdir /go-cache/00: permission denied
[...SNIP]

Try the uid option

No problem. I know that the registry.access.redhat.com/ubi9/go-toolset image specifies USER 1001, so the build step will be run as user 1001, so we need to make sure that the cache mount is accessible to this user. I notice that the --mount=type=cache has a uid option which claims to set the uid of the cache directory to the specified uid. So let's try that:

./Containerfile

FROM registry.access.redhat.com/ubi9/go-toolset:1.21.11-9 AS builder
WORKDIR /hello-world
COPY --chown=1001:0 ./ ./
ENV GOCACHE=/go-cache
RUN --mount=type=cache,target=${GOCACHE},uid=1001 go build main.go

Unfortunately, podman still fails with the same permission error:

$ podman build -f Containerfile .
[...SNIP]
STEP 5/5: RUN --mount=type=cache,target=${GOCACHE},uid=1001 go build main.go
failed to initialize build cache at /go-cache: mkdir /go-cache/00: permission denied
Error: building at STEP "RUN --mount=type=cache,target=${GOCACHE},uid=1001 go build main.go": while running runtime: exit status 1

I stumbled across this PR that indicates that the uid option might not be implemented in buildah/podman: containers/buildah#4178. But it was closed with a message that it needs a redesign and a new PR will be opened later, but I haven't been able to find a later PR related to the uid option.

But docker builds successfully with this option:

$ sudo docker build -f ./Containerfile .
[+] Building 2.8s (10/10) FINISHED                                                                                           docker:default
[...SNIP]
 => [builder 4/4] RUN --mount=type=cache,target=/go-cache,uid=1001 go build main.go                                                    2.4s
 => exporting to image                                                                                                                 0.0s
 => => exporting layers                                                                                                                0.0s
 => => writing image sha256:045932ddaa24401ced5d9185c515713edf8f17f8cf1dbe66bc79a33b5bfb4f84                                           0.0s

Try the U option

While searching on the web, I somehow stumbled across references to the fact that podman apparently has a U option to --mount=type=cache which tries to automatically map the cache to the user within the container. [side note: it's very difficult to find a definitive reference to what containerfile syntax podman supports]. Let's try the U option:

./containerfile

FROM registry.access.redhat.com/ubi9/go-toolset:1.21.11-9 AS builder
WORKDIR /hello-world
COPY --chown=1001:0 ./ ./
ENV GOCACHE=/go-cache
RUN --mount=type=cache,target=${GOCACHE},U go build main.go

Now podman builds the image successfully:

$ podman build -f Containerfile .
[...SNIP]
STEP 5/5: RUN --mount=type=cache,target=${GOCACHE},U go build main.go
COMMIT
--> 5a5331794798
5a5331794798d4c24b020db0cfcbcae321c6a89e64ade089f0a895a9a2f0818c

Unfortunately, this option doesn't seem to exist in docker:

$ sudo docker build -f ./Containerfile .
[+] Building 0.3s (3/3) FINISHED                                                                                             docker:default
 => [internal] load build definition from Containerfile                                                                                0.0s
 => => transferring dockerfile: 298B                                                                                                   0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi9/go-toolset:1.21.11-9                                                  0.3s
 => [internal] load .dockerignore                                                                                                      0.0s
 => => transferring context: 2B                                                                                                        0.0s
Containerfile:5
--------------------
   3 |     COPY --chown=1001:0 ./ ./
   4 |     ENV GOCACHE=/go-cache
   5 | >>> RUN --mount=type=cache,target=${GOCACHE},U go build main.go
   6 |     
--------------------
ERROR: failed to solve: invalid field 'U' must be a key=value pair

It sems to imply that it understands the U option but requires it to be a key=value pair, but if I try to set U=1 or U=true, it fails with the following error:
ERROR: failed to solve: unexpected key 'u' in 'U=true' (did you mean id?)

Solutions?

So is there any portable way to use a cache mount that will work with both podman and docker with the same Containerfile when the container user is non-root?

[nalind]

A cache directory is reused based on its "id", or "target" if an "id" isn't set, and the ownership on a cache directory is set only when it's created the first time, so even after adding the "uid" option, you'd likely still have been using the same directory that was originally created for the default user "0:0". You should be able to clear it with podman system prune while experimenting, or by using a unique "id" option to force a different cache to be created and used.

It does appear that in docker build, cache directories are isolated by that owner UID as well, which would have prevented this conflict, so we need to fix that.

[jonner]

Oh, that's exactly it. Thank you.