Running rootless podman in an unprivileged container in kubernetes

[loelu]

We're trying to get rootless podman running in an unprivileged container using user namespaces in kubernetes. However we're struggling to get it working due to our lack of understanding. We we're hoping to find a detailed guide but the best we found was https://www.redhat.com/en/blog/podman-inside-kubernetes which references running podman with root in k8s with user namespaces, but was written in 2021 before user namespaces in k8s were enabled by default and used more often. Also in the example the container is granted (among others) the SYS_ADMIN capability. But I understood that with user namespaces it should be possible to run rootless container builds without additional capabilities.

The "least-privileged" approach that we got working is with allowPrivilegeEscalation: true and the CAP_SETUID and CAP_SETGID capabilities:

apiVersion: v1
kind: Pod
metadata:
  name: podman-noroot
spec:
  containers:
  - command:
    - sh
    - -c
    - "sleep infinity"
    image: quay.io/podman/stable:v5
    name: podman
    securityContext:
      capabilities:
        add:
        - CAP_SETUID
        - CAP_SETGID
      privileged: false
      allowPrivilegeEscalation: true
      seccompProfile:
        type: Unconfined
      appArmorProfile:
        type: Unconfined
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  hostUsers: false

which results in the following podman info

host:
  arch: amd64
  buildahVersion: 1.42.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-2.fc43.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 97.94
    systemPercent: 0.93
    userPercent: 1.13
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "43"
  eventLogger: file
  freeLocks: 2046
  hostname: debug-podman-noroot
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.12.63
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 256651264
  memTotal: 8153870336
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.17.0-1.fc43.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: netavark-1.17.2-1.fc43.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.2
  ociRuntime:
    name: crun
    package: crun-1.25.1-1.fc43.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.25.1
      commit: 156ae065d4a322d149c7307034f98d9637aa92a2
      rundir: /tmp/storage-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20260120.g386b5f5-1.fc43.x86_64
    version: |
      pasta 0^20260120.g386b5f5-1.fc43.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /tmp/storage-run-1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 74h 45m 8.00s (Approximately 3.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 32143048704
  graphRootUsed: 7201828864
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/storage-run-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 5.7.1
  BuildOrigin: Fedora Project
  Built: 1765324800
  BuiltTime: Wed Dec 10 00:00:00 2025
  GitCommit: f845d14e941889ba4c071f35233d09b29d363c75
  GoVersion: go1.25.4 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.1

When running the pod without allowPrivilegeEscalation: true we get the following error:

ERRO[0000] running `/usr/bin/newuidmap 205 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Any help or pointers to get this working or even about the topics in general (how to configure uid mappings on the host or if privileges in namespaced container are secure) would be much appreciated!

[giuseppe]

have you granted cap_setfcap to the Kubernetes container?

[loelu]

Granting cap_setfcap with allowPrivilegeEscalation: false leads to the same result (Operation not permitted).

As I understand this capability is needed to "map user ID 0 in a new user namespace". However this operation actually works also without this cap and with allowPrivilegeEscalation: false. It's the other two mappings that fail:

[podman@debug-podman-noroot ~]$ unshare -U sleep 600 &
[1] 37
[podman@debug-podman-noroot ~]$ newuidmap $!  0 1000 1
[podman@debug-podman-noroot ~]$ unshare -U sleep 600 &
[2] 39
[podman@debug-podman-noroot ~]$ newuidmap $!  0 1000 1 1 1 999
newuidmap: write to uid_map failed: Operation not permitted
[podman@debug-podman-noroot ~]$ unshare -U sleep 600 &
[3] 41
[podman@debug-podman-noroot ~]$ newuidmap $!  0 1000 1 1000 1001 64535
newuidmap: write to uid_map failed: Operation not permitted

[nalind]

Setting allowPrivilegeEscalation: false prevents the usual kernel-side logic that would heed the newuidmap binary's setuid bit or file capabilities (one or the other, depending on how the container image sets things on it) and run newuidmap with CAP_SETUID in the namespace of the "podman" container.
The podman process (started as UID 1000) tries to use this to set up ID mappings for a second copy of the podman process that's started in a new user namespace, which the first podman process creates and is trying to set up so that the second can run as UID 0 with CAP_SYS_ADMIN inside of a user namespace.
This means that the newuidmap process, which called to set multiple mappings, can no longer satisfy the rather exacting requirements laid out in the "Defining user and group ID mappings: writing to uid_map and gid_map" section of user_namespaces(7), so the kernel doesn't allow it.

[loelu]

Thanks for the explanation, that's really helpful! What I still don't understand is that our container without any capabilities and allowPrivilegeEscalation: false is able to create a mapping of UID 0 to UID 1000 (see answer to @giuseppe ) which I thought should also not be possible since the process doesn't have CAP_SETUID at all.
Can you explain why creating this mapping works but the other ones fail in a completely unprivileged container?

[nalind]

The parent process has the full set of capabilities in the new namespace that it creates, which includes CAP_SETUID, and while the parent process does not have CAP_SETUID in the user namespace that the parent process is in, even without that capability, it fulfills the rest of the requirements for the kernel to allow it to map its current effective ID (and only that ID) into the new namespace.