Build FCOS hermetically in Konflux

[joelcapitao]

The development has already been started but I'm creating this for the further discussion on this topic i.e: how to handle overrides PR (pin and fast-track).

Below the different patches:

• cmd-fetch: Derive konflux lockfiles from rpm-ostree coreos-assembler#4298
• src/konflux-rpm-lockfile: pull $releasever from manifest coreos-assembler#4344
• bump-lockfile: Add hermeto lockfile generation fedora-coreos-pipeline#1245
• config-bot: don't clobber rpms.lock.yaml fedora-coreos-releng-automation#218
• handle overrides PR (pin and fast-track)
• konflux: enable hermetic builds fedora-coreos-config#3723

[joelcapitao]

coreos/fedora-coreos-config@339a3f5 is the first commit that updated the rpms.lock.yaml file.

[joelcapitao]

With coreos/fedora-coreos-config#3829 we realized that we are missing the logic to test the overrides PR.

Below a proposal to address that:

• create a Konflux pipeline for each -on-pull-request and -on-push events that would trigger for the overrides PR i.e: when a change occurs on manifest-lock.overrides.yaml only.
• hermetic will be disable in -on-pull-request pipeline
• we add the same logic as in [1] in our Tekton prepare-build-context task. That logic would be executed only in the override specific pipeline.
• change the pipeline order in order to have prepare-build-context before prefetch-dependencies.
• update the override/pin workflow code, to update rpms.lock.yaml accordingly.

When submitting a PR, the Konflux pipeline would test the packages set non-hermetically. The overrides RPMs would be pulled from the overrides/rpm repo.
When merging the PR, coreos-koji-tagger would tag the RPMs to coreos-pool tag in Koji. We'd have to add some logic in prepare-build-context to ensure the Koji operation is complete before proceeding to the prefetch-dependencies task. Konflux will then pull the RPMs rpms.lock.yaml when running prefetch-dependencies task e.g: downloading https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/latest/x86_64/Packages/i/ignition-2.24.0-1.fc42.x86_64.rpm.

What do you think ?

[1] https://github.com/coreos/fedora-coreos-config/blob/339a3f52e90b520e791e51d5ad6668014f798e4f/.github/workflows/container-native.yml#L30-L47

[dustymabe]

One problem I see with coreos/coreos-assembler#4298 is that the rpms.lock.yaml include the links to the rpm as it was discovered in that run of cosa fetch. The problem with this is that the repos change over time, and what was in fedora-updates today, might not be in fedora-updates tomorrow (i.e. we may now have to go to the coreos-pool for some of those RPMs that are "out of date" but still locked).

[joelcapitao]

The problem with this is that the repos change over time, and what was in fedora-updates today, might not be in fedora-updates tomorrow

That's right, one way to address that is to run bump-lockfiles job more frequently than the nightly composes.
Another way is to proceed like bootc does i.e pulling compose repos c.f https://gitlab.com/fedora/bootc/compose-images and https://gitlab.com/fedora/bootc/base-images/-/issues/51#note_2543578650. So basically, the bootc images are rebased on top of "compose images" which are just base Fedora images + compose repos in /etc/yum.repos.d/. So they don't track mirror URL (= latest compose image). Though, they do that for reproducibility only for now, they are not enabling hermetic builds.
I double checked the retention time window of the compose repos c.f https://kojipkgs.fedoraproject.org/compose/updates/ looks like Fedora releng only keeps N-10 repos, N being the latest one.

I'm more inclined to reproduce bootc approach, but I'm not sure if there are other limitation/constraints specific to CoreOS that might prevent this.

[jbtrystram]

The problem with this is that the repos change over time, and what was in fedora-updates today, might not be in fedora-updates tomorrow (i.e. we may now have to go to the coreos-pool for some of those RPMs that are "out of date" but still locked).

This script favors addresses from the coreos-pool repo if the RPM is available there to address exactly that. Not sure if that's robust enough though.

So basically, the bootc images are rebased on top of "compose images" which are just base Fedora images + compose repos in /etc/yum.repos.d/. So they don't track mirror URL (= latest compose image).

Doesn't this means that we would end up with an SBOM containing all the RPMs available in that compose rather than the subset we actually picked ?
had a talk oob with Joel and I am wrong : the compose image contains only a repofile pointing to said compose, not the RPMs