ISO build fails during depsolve when enabled repo uses gpgkey=file://... from source image (Curl error 37)

[seedmonn]

Summary

Building an ISO with bootc-image-builder can fail during depsolve when an enabled RPM repo in the source bootc container image uses a local GPG key reference like gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-....

In our case (Bazzite / terra-mesa repo) depsolve fails with Curl error (37) because the depsolve environment cannot open the file:// path, even though the key exists inside the source image.

How we run it (via ublue image-template)

We build and then run BIB using the ublue-os/image-template Justfile:

• rebuild-iso ultimately calls:
  bootc-image-builder --type iso --use-librepo=True --rootfs=btrfs ... <image>:<tag>

(See Justfile recipe: rebuild-iso -> _build-bib type "iso".)

Expected behavior

If the GPG key file exists inside the source container image at the referenced path, depsolve should not fail trying to read it, or bootc-image-builder should detect this situation and fail early with an actionable error.

Actual behavior

Depsolve fails with:
Errors during downloading metadata for repository 'terra-mesa':

• Curl error (37): Could not read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-terra43-mesa
  [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-terra43-mesa]
  RepoError: Failed to retrieve GPG key for repo 'terra-mesa': Curl error (37) ...

Reproduction (generic)

1. Use/build a bootc container image that has an enabled repo with:
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-example and the key file exists in the image at /etc/pki/rpm-gpg/RPM-GPG-KEY-example.
2. Run bootc-image-builder for ISO:

sudo podman run --rm -it --privileged --security-opt label=type:unconfined_t \
-v ./output:/output -v /var/lib/containers/storage:/var/lib/containers/storage \
quay.io/centos-bootc/bootc-image-builder:latest --type iso <image-ref>

3. Observe depsolve failing with Curl error (37) reading the file:// key path.

Why this happens (hypothesis)

Depsolve runs in an isolated build environment; file:// URLs are resolved against the depsolve filesystem, not the source
image filesystem. Therefore absolute paths like /etc/pki/rpm-gpg/... may not exist there.

[supakeen]

cc @lzap since you're already working on the related bug

[supakeen]

Just as a note, we would prefer if people try to move from the RPM based installer to the container based installer: the bootc-installer image type (and the future bootc-generic-iso which makes even less assumptions).

This doesn't make this bug invalid but I thought I'd mention it 🙂

[achilleas-k]

Depsolving works a bit differently in the bootc case. The depsolve request doesn't include repository configurations like in other cases but instead defines a root_dir that points to the root of the (mounted) base container's root, which is passed on to dnf to autodiscover repository configurations under $root_dir/etc/yum.repos.d/. This is a bit fiddly though because it requires translating any file paths within those configurations (certs, keys, etc). The depsolver handles the translation for the certs, but we neglected to consider paths for GPG keys.

My guess is that's the issue here. I suspect if we add the same treatment (a call to modify_rootdir_path()) for the gpg key it would work.

[achilleas-k]

But this line should be handling the gpg keys: https://github.com/osbuild/osbuild/blob/700012c9159ca76e670f338ae28ea47830087fa4/osbuild/solver/dnf.py#L105

There's probably more to this story then. This will need some deeper digging.