Build FCOS using podman build

[cgwalters]

Describe the enhancement

This tracking issue is the FCOS side of https://gitlab.com/fedora/bootc/tracker/-/issues/32

The way we use treefiles and git submodules is not container native and not something I'd like to support widely.

Effectively the goal here is that FCOS container build becomes:

# This defines the base image content and configuration
FROM quay.io/fedora/fedora:rawhide as config

# The version of this base image defines the build process
FROM quay.io/fedora/fedora-bootc:41 as builder

# Copy in the source code
COPY . /src
WORKDIR /src

RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared \
    --mount=type=bind,from=config,src=/,dst=/config <<EORUN
#!/bin/bash
x-image-builder init /config /target-rootfs
dnf --installroot /target-rootfs install zincati ...
# And other mutations here
EORUN
# Copy in our overlay content
COPY --from=overlays/* /target-rootfs
RUN x-image-builder chunked-oci /target-rootfs /buildcontext/output.oci

FROM oci:./output.oci
# Need to reference builder here to force ordering. But since we have to run
# something anyway, we might as well cleanup after ourselves.
RUN  --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared \
      rm /buildcontext/output.oci -rf

Also, coreos-assembler (or the FCOS build pipeline) would need to switch to podman build (or equivalent).

System details

No response

Additional information

No response

[jlebon]

Ongoing discussions about this in https://gitlab.com/fedora/bootc/tracker/-/issues/32 and coreos/rpm-ostree#5221. The other approach being explored is rechunking a derived container. So it would look something like this:

# build our rootfs
FROM quay.io/fedora/fedora-bootc:rawhide as rootfs
RUN --mount=type=bind,target=/run/src rpm-ostree compose apply /run/src/manifest.yaml

# rechunk
FROM quay.io/fedora/fedora-bootc:rawhide as builder
RUN --mount=type=bind,rw=true,dst=/run/src,bind-propagation=shared \
    --mount=from=rootfs,dst=/rootfs \
        rpm-ostree compose build-chunked-oci --ostree --rootfs=/rootfs --output /run/src/out.oci

# output rechunked OCI
FROM oci:./out.oci
LABEL containers.bootc 1
# <any other container-native metadata here>
# Need to reference builder here to force ordering. But since we have to run
# something anyway, we might as well cleanup after ourselves.
RUN --mount=type=bind,from=builder,src=.,target=/var/tmp \
    --mount=type=bind,rw=true,dst=/buildcontext,bind-propagation=shared \
        rm -f /buildcontext/out.oci

If we can do coreos/rpm-ostree#5221 (comment), then it reduces to:

# build our rootfs
FROM quay.io/fedora/fedora-bootc:rawhide as rootfs
RUN --mount=type=bind,rw=true,dst=/run/src,bind-propagation=shared \
  rpm-ostree compose apply /run/src/manifest.yaml && \
  rpm-ostree compose build-chunked-oci --ostree --rootfs=/ --output /run/src/out.oci

# output rechunked OCI
FROM oci:./out.oci
LABEL containers.bootc 1
# <any other container-native metadata here>
# Need to reference rootfs here to force ordering. But since we have to run
# something anyway, we might as well cleanup after ourselves.
RUN --mount=type=bind,from=rootfs,src=.,target=/var/tmp \
    --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared \
        rm -f /buildcontext/out.oci

[dustymabe]

We discussed this during the community meeting today.

Overall our understanding is that the goal of this issue/request is to change our build process for FCOS to use podman completely for base FCOS container creation. To that end we settled on:

• AGREED: FCOS is generally on board with changing FCOS builds to use podman build with multi-stage builds in the future. (@dustymabe:matrix.org, 17:30:05)

Further discussion happened about some nuance on how to implement this summarized by @jlebon as:

The "move to podman build" at this point is uncontroversial (at least in those upstream discussions). The conversation at this point is mostly around "what interface/mechanism should we expose to users for them to customize the base images". and there are two broad approaches to that:

1. another base image build, where you're actually composing from scratch. then the image gets chunked as usual, and it just looks like any base image.
2. do a derivation, which is what you usually do when you e.g. add packages on top of FROM fedora for example. This works, but sometimes you really do want it to "look like a base image" at the end because it results in more efficient images and can fix things up. And that's the "build-chunked-oci" magic that happens in those Containerfiles.

from which we're still digesting the options, but we're generally agreed that podman build satisfies all of these cases.

[jlebon]

So when deriving, one major area that'll need to be reworked is lockfile handling. For the base tier-x image, we can pin by digest and bump that. For the layered packages, we'll still need lockfiles. Unfortunately, lockfile support in dnf5 is not ready yet, so we'll probably have to re-implement this ourselves for now (see also rpm-software-management/libpkgmanifest#6 (comment)), e.g. as part of treefile-apply.

[cgwalters]

Note a large intersection here is Fedora konflux and the Konflux rpm lockfiles as are used by e.g. centos-bootc today; however right now, those aren't honored by podman build which is obviously a big thing to fix (well, the compose-pinned repo files are used, but obviously this is a bit circular)

[jlebon]

Note a large intersection here is Fedora konflux and the Konflux rpm lockfiles as are used by e.g. centos-bootc today; however right now, those aren't honored by podman build which is obviously a big thing to fix (well, the compose-pinned repo files are used, but obviously this is a bit circular)

Is there a story around honouring that in a local dev environment as well? Whatever we land on has to work in both Konflux and locally.

[dustymabe]

We discussed this during the community meeting today.

Overall our understanding is that the goal of this issue/request is to change our build process for FCOS to use podman completely for base FCOS container creation. To that end we settled on:

• AGREED: FCOS is generally on board with changing FCOS builds to use podman build with multi-stage builds in the future. (@dustymabe:matrix.org, 17:30:05)

Note that the future viability of workflow needed for this (as detailed in the description) is in question so we'll need to resolve that before continuing down this path.

[cgwalters]

We are investing in supporting a post-processing flow to add rechunking as an optional secondary step, so I don't think it should be considered a blocker for switching to podman build; we aren't taking it as a blocker for the bootc base image.