Upgrade dracut module: Update /usr mounting solution

[pirat89]

Originally we had implemented our own mount_usr.sh script, which took care about mounting the /usr when it is present on separate partition / mountpoint. It took care also about LVM activation.

However, it has been problematic in various cases (e.g. when device needed more time for initialisation - e.g. when connected using FC). Let's use instead existing system solutions, starting the upgrade.target after initrd-fs.target (instead of just basic.target).

Jira-ref: RHEL-3344, RHEL-35446

Cooperating with: prajnoha, lnykryn, pvalena.

do_not_merge: just exprimenting right now. if we find working solution, I will update yet commit msg at minimum.

Technical description of the solution

1. LVM Autoactivation

The device that is mounted at /usr might be an LVM partition, so the proposed solution needs to handle LVM activation (the old implementation already did so). To this end, we include the 69-dm-lvm.rules udev rule into the upgrade initramfs. These rules autoactivate LVM volumes when a device is discovered by the kernel and the udev daemon is notified about them. This mechanism is borrowed from a standard boot process. For more details see lvmautoactivation (7).

2. Mounting the filesystem according to /etc/fstab

The goal is to use a mounting mechanism that is as close as possible to an ordinary boot process of a RHEL system. Inside the initramfs, the boot process looks as follows (taken from bootup):

                               : (beginning identical to above)
                               :
                               v
                         basic.target
                               |                       emergency.service
        ______________________/|                               |
       /                       |                               v
       |            initrd-root-device.target          emergency.target
       |                       |
       |                       v
       |                  sysroot.mount
       |                       |
       |                       v
       |             initrd-root-fs.target
       |                       |
       |                       v
       v            initrd-parse-etc.service
(custom initrd                 |
 services...)                  v
       |            (sysroot-usr.mount and
       |             various mounts marked
       |               with fstab option
       |              x-initrd.mount...)
       |                       |
       |                       v
       |                initrd-fs.target
       \______________________ |
                              \|
                               v
                          initrd.target
                               |
                               v
                     initrd-cleanup.service
                          isolates to
                    initrd-switch-root.target
                               |
                               v
        ______________________/|
       /                       v
       |        initrd-udevadm-cleanup-db.service
       v                       |
(custom initrd                 |
 services...)                  |
       \______________________ |
                              \|
                               v
                   initrd-switch-root.target
                               |
                               v
                   initrd-switch-root.service
                               |
                               v
                     Transition to Host OS

One of the most important parts is initrd-parse-etc, which calls systemd-fstab-generator to parse /etc/fstab and generate mount units that mount the filesystem tree as described by /etc/fstab when possible (device is available, mount target exists, etc.). However, triggering the service in initramfs will not help us, since the /etc/fstab in initramfs is different than the one used on the source system. Moreover, we a slightly different behavior than the one described by source system's fstab: instead of mounting device X at /usr, we want to mount it at /sysroot/usr.

The solution proposed by this PR is to manually run systemd-fstab-generator to generate corresponding mount units before reboot. These units are modified according to our needs (/usr -> /sysroot/usr), and, then we include them in our upgrade initramfs. These mount units are then mounted, since they are marked as a dependency of local-fs.target.

Notes for reviewers

Modify the created upgrade boot entry by running the following command.

grubby --update-kernel=/boot/vmlinuz-upgrade.x86_64 --args rd.upgrade.break=leapp-pre-upgrade

After rebooting, you will be dropped into a shell that allows you to check the state of the system before proceeding with the upgrade. Check that everything is mounted as expected under /sysroot, and that the created mount units are active without any problems:

mount | grep sysroot
systemctl status '*.mount'  # Everything should be green
systemctl status '*.swap'  # Everything should be green

Additional notes

• discovered that we need to use initrd-parse-etc.service, but the inside the unit file:

  ExecStart=/usr/bin/systemctl --no-block start initrd-cleanup.service
  

  The initrd-cleanup service includes also switch-root, which we do not want to happen at all and due to isolation it basically kills out upgrade process.
• the service has been changed in various RHEL systems, so creating own service would mean we should most likely copy couple of them (per each change in RHEL systems) and watch whether it's changed in future
• going now with the original workaround mount /usr: Implement try-sleep loop (SAN + FC) #1218 . We are going to continue to investigate the proper solution for future releases in IPU 8 -> 9 )and newer upgrade paths)

Update:

• Discussed the issue with systemd guys, we have decided to override the initrd-cleanup service inside the upgrade initramfs. Several initial tests seems to provide positive results. (Just discovered a problem, I will investigate it later)
• Seems I forget to apply a change I did in the past locally as I could not get to the state where I get when I discovered the problem that should be fixed now. It's again in the state that only /usr is handled but not the rest of fstab. I need to check it one more time what is wrong with the upgrade service definition.

[github-actions]

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

• review please @oamg/developers to notify leapp developers of the review request
• /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp, e.g. from PR#42, use /packit test oamg/leapp#42
Note that first time contributors cannot run tests automatically - they will be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

• /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
• /packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

[pirat89]

/packit build

[MichalHe]

/packit build

[MichalHe]

@pirat89 I have tried this patch with a VM that uses a separate virtio block device for /usr and it worked fine. I have verified that /usr is mounted, and that initrd-parse-etc.service was triggered successfully before upgrading (using rd.upgrade.break=leapp-pre-upgrade).

Do you happen to have any recollection on the conditions when this patch was not working?

[pirat89]

@MichalHe I have reproduced the problem with:

• fstab

#
# /etc/fstab
# Created by anaconda on Thu May  2 18:40:01 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=fc227809-1070-4d86-b676-54cb81a5455b /boot                   xfs     defaults        0 0
/dev/mapper/rhel01-usr  /usr                    xfs     defaults        0 0
/dev/mapper/rhel00-var  /var                    xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0

• partitioning:

# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part pv.1349 --fstype="lvmpv" --ondisk=vda --size=7172
part pv.146 --fstype="lvmpv" --ondisk=vda --size=6148
part /boot --fstype="xfs" --ondisk=vda --size=1024
part pv.726 --fstype="lvmpv" --ondisk=vda --size=8196
volgroup rhel00 --pesize=4096 pv.726
volgroup rhel01 --pesize=4096 pv.1349
volgroup rhel --pesize=4096 pv.146
logvol /  --fstype="xfs" --size=3072 --name=root --vgname=rhel
logvol /var  --fstype="xfs" --size=7168 --name=var --vgname=rhel00
logvol /usr  --fstype="xfs" --size=6144 --name=usr --vgname=rhel01
logvol swap  --fstype="swap" --size=2048 --name=swap --vgname=rhel

• lsblk

[root@localhost ~]# lsblk
NAME           MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda            252:0    0  30G  0 disk 
├─vda1         252:1    0   1G  0 part /boot
├─vda2         252:2    0   8G  0 part 
│ └─rhel00-var 253:3    0   7G  0 lvm  /var
├─vda3         252:3    0   7G  0 part 
│ └─rhel01-usr 253:2    0   6G  0 lvm  /usr
├─vda4         252:4    0   1K  0 part 
└─vda5         252:5    0   6G  0 part 
  ├─rhel-root  253:0    0   3G  0 lvm  /
  └─rhel-swap  253:1    0   2G  0 lvm  [SWAP]
vdb            252:16   0  20G  0 disk 

• screenshot with error:
  

note that for the screenshot I had to re-run it, that's why you see an extra msg about .leapp_upgrade_failed file and previous failure.

[MichalHe]

@pirat89 By default, systemd will mount only / and /usr from /sysroot/etc/fstab. If we want to have any other device mounted early by systemd, we need to add the x-initrd.mount option to corresponding fstab entries. Any such a unit will be picked up and mounted automatically. So, in the reproducer you have shared, I need to modify

/dev/mapper/rhel00-var  /var                    xfs     defaults        0 0

into

/dev/mapper/rhel00-var  /var                    xfs     defaults,x-initrd.mount        0 0

Also, if the device is an LVM's logical volume and the source system is using rd.lvm.lv cmdline args, we need to add anything we will be mounting the the list. So, I needed to add rd.lvm.lv=rhel00/var to the upgrade boot entry in order for the corresponding /dev/mapper/... entry to be present so that the unit generated by systemd could be activated. Alternatively, we can just remove all rd.lvm.lv cmdline args and dracut will make all lvm volumes available.

Otherwise, the upgrade went OK.

[MichalHe]

Looking at when systemd-fstab-generator is supposed to run: it is supposed to run before any services are run, or when PID 1 is reloaded. Now, initrd does not contain a valid /etc/fstab, so if we run systemd-fstab-generator; systemd daemon-reload it should have no effect whatsoever. Manually creating /etc/fstab inside initrd will not work either - the generated units should mimic fstab, and, thus, they would mount, e.g., /var at /var not /var /sysroot/var, hence they would not help. So we would have to essentially do switchroot, if we want to use systemd-fstab-generator the way normal boot uses it, imho.

What we could do in theory is to run systemd-fstab-generator before reboot and try keeping the generated .mount units. We would then modify them by prefixing mount targets with /sysroot, add them into the upgrade initramfs, and make them our dependencies. This could work, I but have not tried it.

[MichalHe]

So, the issue is essentially twofold:
1) We stop the standard boot process early, and, therefore, we might be stranded without necessary LVs. For example, if one uses rd.lvm.lv=.. to instruct dracut to only activate volumes for / and /usr, and there is a LV for /var no one will take care of this for us. Luckily, the lvmautoactivation manpage does a great job in explaining how the process works. So, to obtain some magic that will autoactivate LVM for us, we just need:

LEAPP_DRACUT_INSTALL_FILES="/usr/sbin/pvscan /usr/sbin/vgchange /usr/lib/udev/rules.d/69-dm-lvm.rules"

To tell dracut to include key LVM binaries, and an udev rule that takes care of triggering these commands. I have tried booting into the initramfs, and the lv for /var from @pirat89's example is activated, although the rd.lvm.lv= options are used for / and `/usr/

2) We want to ditch running /mount -a in a loop and, instead, use systemd mechanism
We hijack systemd-fstab-generator, keeping the units, but modifying their mount targets to be prefixed with /sysroot. So, I executed systemd-fstab-generator as

# systemd-fstab-generator fstab-mounts fstab-mounts fstab-mounts

resulting in the mount units being placed in /fstab-mounts. Then I deleted units for / (-.mount) and /usr (usr.mount) as these will be already mounted by dracut, leaving me only with var.mount and boot.mount. Then I modified both of these units, prefixing /sysroot to their mount targets. For example, in var.mount we change

Where=/var

into

Where=/sysroot/var

Now systemd is very strict how mount units should be named, so we need to rename the mount units to reflect the new mount target. For example, for /var we have var.mount renamed to sysroot-var.mount and similarly for boot.mount. Finally, systemd-fstab-generator also generates fstab-mounts/local-fs.target.requires directory. The contents of this directory act as if they were listed in Requires= of the local-fs-target. We have changed the names of the mount units, so the symlinks generated by systemd-fstab-generator are broken. Hence:

# cd /fstab-mounts/local-fs.target.requires
# rm ./*
# ln -s ../sysroot-var.mount . 
# ln -s ../sysroot-boot.mount .

Finally, we copy the contents of /fstab-mount to /usr/lib/systemd/system and tell dracut to include these units in our upgrade initramfs:

LEAPP_DRACUT_INSTALL_FILES="$LEAPP_DRACUT_INSTALL_FILES /usr/lib/systemd/system/sysroot-var.mount /usr/lib/systemd/system/local-fs.target.requires/sysroot-var.mount /usr/lib/systemd/system/sysroot-boot.mount /usr/lib/systemd/system/local-fs.target.requires/sysroot-boot.mount"

And we have an upgrade initramfs that a) autoactivates LVM, b) uses systemd to mount entries from /etc/fstab

😁

[prajnoha]

@MichalHe, thanks for the great in-depth analysis. Just to understand the issue better - so the problem is if we're trying to access anything else than / and /usr (like the /var) with the leapp upgrade script inside initrd, right?

Is this just a new use case we need to support now (having var on a separate device)? Otherwise, how did this work before? Was leapp using only / and /usr before (without  /var being a separate mount point)? I'm a bit confused now...

[pirat89]

@MichalHe, thanks for the great in-depth analysis. Just to understand the issue better - so the problem is if we're trying to access anything else than / and /usr (like the /var) with the leapp upgrade script inside initrd, right?

Is this just a new use case we need to support now (having var on a separate device)? Otherwise, how did this work before? Was leapp using only / and /usr before (without  /var being a separate mount point)? I'm a bit confused now...

Previously, we activated lvm and all other mountpoints have been mounted later by mount -a - everything in /etc/fstab needs to be mounted. it's hard to upgrade the system when all system partitions are not mounted to perform all RPM operation.

And that's basically why we started these discussions, as mount -a does not work when the storage is not properly initialized before the cmd execution.

[prajnoha]

OK, I see - so dracut/systemd tandem in initrd only cares about / and usr and nothing else, because they are the only things needed to get the rootfs in the end and switch to that. Well, the solution that Michal commented about is actually how LVM autoactivates volumes when we are switched already from initrd to rootfs - which would mean bypassing and replacing dracut's hooks completely as a matter of fact. So the question now is whether we want to follow this path or whether we should update dracut itself to also care about other mount points than / and /usr.

[prajnoha]

Side note: the reason that we're not using the usual autoactivation in dracut is historical. At first, we didn't have any event-based autoactivation, so dracut needed to come up with its own way (not that ideal, because it contains a kind of a loop, but it didn't matter much since we were interested only in the VG/LV on which the rootfs sit). Then we added native event-based autoactivation to LVM, but only with a helper lvmetad daemon and we didn't want to include a daemon inside the dracut to only support autoactivation of a single VG/few VGs (with the LVs where / and /usr sit) and the LVM dracut hook worked, so there was no reason to replace it anyway. Now (since 2020 or 2021 I think), we removed lvmetad and we can do autoactivation just with the access to /run where we store helper files to track incoming PVs so we can autoactivate the VG once all needed PVs are present even without a helper daemon.

We could also try what happens if we have MD (or other storage virtualization technology) in the device stack instead of LVM - but I assume it would be very similar situation.

[MichalHe]

@prajnoha Thanks. Anything that gets the job done is fine by me. I would just note that in case this change is implemented in dracut, we would be introducing code into dracut's codebase (a maintenance cost) that would be used by no one else but us (upgrades). How much work do you think would it be to get this into dracut? Is there a reason why is it a bad idea to bypass dracut's mechanisms?

@pirat89 Do you see any issues in case that this would be implemented in dracut w.r.t. to how leapp releases its RHEL builds? It would not be that we require dracut, we would have to require dracut with some minimal version, otherwise we would have to do some weird stuff when we detect the dracut version and use leapp's old mechanism or something.

I am also worried a bit about what if we decide, e.g., that we will support SANs. I know a little to nothing about having a SAN attached to the system, so I blindly wonder whether a LVM could be set up on top of a SAN. Would implementing the solution in dracut be flexible enough? Our use case requires us to mount everything in fstab - is there a chance that a LVM activation requires some nontrivial systemd dependencies?

Re-reading this message, I get the feeling that I am a proponent of going the "LVM-autoactivation" route. Please, do not be mistaken, I just want to make an informed decision 😁

[pirat89]

Summary from current manual testing:

• tested 30+times on AWS (20+ for 9 -> 10): ✔️
• tests with encrypted storage (8->9): ✔️
• VMs (8 ->9->10) with LVM: ✔️
• VMs (8->9->10): without LVM ✔️
• VM (8->9->10): rootfs + /boot: ✔️
• single MP (rootfs only, /boot is part of rootfs): 🚫 (skipped for now)
• Azure: 📆 (update tomorrow)

I will do yet a test on Azure due to specific (behaviour) storage there and if that one will pass as well, we can jump to merging. Possibly I will update yet the commit msg in the first commit to drop TBD.

Almost there!

[pirat89]

error on azure:

Risk Factor: high (error)
Title: Actor mount_unit_generator unexpectedly terminated with exit code: 1
Summary: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 75, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/actors/__init__.py", line 319, in run
    self.process(*args)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/actor.py", line 22, in process
    mount_unit_generator_lib.setup_storage_initialization()
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/libraries/mount_unit_generator.py", line 245, in setup_storage_initialization
    mount_unit_files = copy_units_into_system_location(upgrade_container_ctx, workspace_path)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/libraries/mount_unit_generator.py", line 184, in copy_units_into_system_location
    shutil.copytree(src_path, dst_path, symlinks=True)
  File "/usr/lib64/python3.6/shutil.py", line 321, in copytree
    os.makedirs(dst)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/lib/leapp/el9userspace/usr/lib/systemd/system/remote-fs.target.wants'

• I executed it couple of times on the machine so it's possible that this is reproducidble on multiple machines, but reproduced on azure RHEL 8

[matejmatuska]

error on azure:

Risk Factor: high (error)
Title: Actor mount_unit_generator unexpectedly terminated with exit code: 1
Summary: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 75, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/actors/__init__.py", line 319, in run
    self.process(*args)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/actor.py", line 22, in process
    mount_unit_generator_lib.setup_storage_initialization()
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/libraries/mount_unit_generator.py", line 245, in setup_storage_initialization
    mount_unit_files = copy_units_into_system_location(upgrade_container_ctx, workspace_path)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/initramfs/mount_units_generator/libraries/mount_unit_generator.py", line 184, in copy_units_into_system_location
    shutil.copytree(src_path, dst_path, symlinks=True)
  File "/usr/lib64/python3.6/shutil.py", line 321, in copytree
    os.makedirs(dst)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/lib/leapp/el9userspace/usr/lib/systemd/system/remote-fs.target.wants'

• I executed it couple of times on the machine so it's possible that this is reproducidble on multiple machines, but reproduced on azure RHEL 8

@pirat89 and I did some debugging, I cannot write the exact details because I don't have access to the machine right now:

Cause

As the error say /var/lib/leapp/el9userspace/usr/lib/systemd/system/remote-fs.target.wants is indeed present at the time we copy our generated mount units to target userspace. The file is owned by the systemd RPM.

The error however happens because we try to copy the contents of our generated units directory (the temporary dir) to /usr/lib/systemd/system. That means that the temp dir also contains remote-fs.target.wants. There is a mountpoint on the Azure machine for some ...resource_part1 (can't remember the exact name) mounted on /mnt (related to cloud-init). The mount options include nofail and according to man systemd.mount:

When reading /etc/fstab a few special mount options are understood by systemd which influence how dependencies are created for mount points. systemd will create a dependency of type Wants= or Requires=
(see option nofail below), from either local-fs.target or remote-fs.target, depending whether the file system is local or remote.

Therefore besides remote-fs.target.wants, also remote-fs.target.requires, local-fs.target.wants, local-fs.target.requires could exist.

Fix

We need to prefix the units units in the directories about with "sysroot" too.
To get around the above error, we need to remove these directories from the target userspace before copying our units over. copy our content into them, but preserve existing content.

[pirat89]

Retesting summary (in-progress) (swap is always included in following tests)

• AWS both arch (8 -> 9): ✔️

• AWS both arch (9 -> 10): ✔️

• Azure (8->9): ✔️

• Azure (9->10): ✔️

• GCP (@matejmatuska ) (8 -> 9): ✔️

• tests with encrypted storage (8->9): ✔️

• tests with encrypted storage (9->10): ✔️

• VMs with LVM - simple (root, boot) (8->9): ✔️

• VMs with LVM - simple (root, boot) (9->10): ✔️

• VMs with LVM (root, boot, usr, var) (8->9): ✔️

• VMs with LVM (root, boot, usr, var) (9->10): ✔️

• VMs without LVM (8 ->9): ✔️

• VMs without LVM (9 ->10): 🚫
  ** isn't that covered actually by one of clouds?

• single MP (rootfs only, /boot is part of rootfs) 8 -> 9: ✔️

• single MP (rootfs only, /boot is part of rootfs) 9 -> 10: 🚫

---

** ✔️ works
** ❌ broken
** 🚫 skipped
** nothing - not tested yet

[matejmatuska]

All tests, passed. Rebase only squashed the fixup commit. Approving, will merge once the tests pass again (there was no change, but we are not in hurry).

[matejmatuska]

LGTM, extensively tested on different setups.