Skip to content

NIX_CHANNEL=nixos-26.05 silently produces an unreachable, half-converted system (NixOS 26.05 removed NIXOS_LUSTRATE support) #276

Description

@fredcy

Summary

Running nixos-infect with NIX_CHANNEL=nixos-26.05 completes without error,
but the resulting system is broken: NixOS 26.05 boots with systemd stage-1
(boot.initrd.systemd.enable = true) by default, and its init has no
NIXOS_LUSTRATE support — so the old distro is never swept into
/old-root. The first boot runs a NixOS kernel/init against the old
distro's intact /etc, and systemd picks up the old distro's unit files
(which can't execute on NixOS). The machine typically comes up with no
networking and appears dead.

From the current NixOS manual (Installation > "Installing from another
Linux distribution"):

The lustrate process will not work if the boot.initrd.systemd.enable
option is set to true, which is now the default. Setting this to false is
deprecated and scheduled for removal in NixOS 26.11, along with
NIXOS_LUSTRATE.

The fixes are already pending as #267 (pin boot.initrd.systemd.enable = false) and #269 (lustrate module for systemd stage-1, with
NixOS/nixpkgs#509036), but until one merges, every stock run against
nixos-26.05 quietly builds a brick — and there's no issue describing the
failure for users to find. Hence this report.

Environment

  • Hetzner Cloud CX23 (x86_64, BIOS boot), Ubuntu 24.04 image
    (also reproduced on Hetzner's Ubuntu 26.04 image)
  • curl .../nixos-infect | PROVIDER=hetznercloud NIX_CHANNEL=nixos-26.05 bash
  • 2026-07-09, script at current master

Symptoms

  • Script exits 0; after reboot the host never becomes reachable (no ssh,
    no ping, either address family).
  • Console (if watched): GRUB → NixOS kernel messages → minutes-later
    failures of old-distro units, e.g.
    Failed to start Download data for packages that failed at package install time
    (Ubuntu's update-notifier) and Failed to start system activity accounting tool
    (sysstat).
  • Journal shows NixOS activation unable to claim /etc
    (/etc/systemd/system directory contains user files. Symlinking may fail.)
    and systemd crash-looping the old distro's binaries:
    Could not start dynamically linked executable: /usr/lib/systemd/systemd-networkd
    (NixOS stub-ld).
  • On-disk evidence: no /old-root; /etc/NIXOS_LUSTRATE present and
    unconsumed; old distro tree fully intact.
  • Confirming the mechanism: the built closure contains no lustration code —
    strings $(readlink /nix/var/nix/profiles/system)/init | grep -ci lustrat
    → 0, and same for the stage-2 scripts in the closure.

Suggested interim mitigation (until #267 or #269 merges)

Fail hard (or at least warn loudly) in the script when the requested
channel is ≥ 26.05, and note the incompatibility in the README's channel
examples. A one-line version-gate would have saved us a full
reimage-and-rescue cycle; the current failure mode gives no hint at any
stage.

Recovery for already-bitten systems (worked for us)

Boot the provider's rescue system, mount the disk, and lustrate manually:

mount /dev/sda1 /mnt && cd /mnt
mkdir -m 0755 -p old-root
for d in *; do
  case "$d" in nix|boot|old-root|lost+found) ;; *) mv "$d" old-root/ ;; esac
done
mkdir -m 0755 -p etc/ssh root
cp -a old-root/etc/nixos etc/
cp -a old-root/etc/resolv.conf etc/
cp -a old-root/etc/ssh/ssh_host_* etc/ssh/
cp -a old-root/root/.nix-defexpr root/
touch etc/NIXOS
mkdir -m 0755 -p proc sys dev run mnt var srv home media opt
mkdir -m 1777 -p tmp

then reboot. (We also had to add the IPv4 default route by hand afterwards —
that part is #271, the defaultGateway string form no longer applying an
interface route on 26.05.)

Possibly related

  • Broken on hetzner trixie #245 — the "Didn't reboot into a system with SSH" follow-up comments on
    Debian 13 look like this same root cause (recent channels), independent
    of the tmpfs-swap symptom in the original report.
  • While debugging we also noticed provider autodetection never fires:
    if [ ! -v $PROVIDER ] (note the $) tests the literal string -v when
    PROVIDER is unset, so autodetectProvider is skipped exactly when it's
    needed. if [ ! -v PROVIDER ] (or [ -z "${PROVIDER:-}" ]) fixes it.
    Happy to split this into its own issue/PR if useful.

Debugged interactively with an AI assistant (Claude); report written from
the actual disk/journal forensics and reviewed by a human before filing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions