Qubes packages (Its alive!) #341215
Qubes packages (Its alive!) #341215
Conversation
github-actions
bot
added
6.topic: python
6.topic: nixos
There was a problem hiding this comment.
The python files need a meta attribute set. Since a lot of the Qubes tools are GPLv2 and have the https://qubes-os.org homepage, we could deduplicate the meta attributes in a common file that is inherited and // updated as needed.
|
|
||
| qubes-core-qubesdb = toPythonModule (pkgs.qubes-core-qubesdb.override { | ||
| inherit python; | ||
| withPython = true; |
There was a problem hiding this comment.
Might be better to make the main derivation have withPython ? true, and drop the override.
There was a problem hiding this comment.
Well, we still need to build this package using specific python version, thus override is inevitable
| gtk3 | ||
| ]; | ||
|
|
||
| # Should also be packaged as vm-only? |
There was a problem hiding this comment.
A split output package sounds good here IMO.
There was a problem hiding this comment.
I think current approach with passthru is okayish, as those packages are tightly bound.
Proper splitting will require a qubes package set, which I want to avoid.
| mv $out/usr/* $out/ | ||
| rm -rf $out/usr | ||
| # Scripts: qubes-rpc-multiplexer | ||
| cp ../lib/* $out/bin/ |
There was a problem hiding this comment.
| cp ../lib/* $out/bin/ | |
| cp ../lib/* $out/libexec/ |
Very optional. If it breaks something, revert to $out/bin. Having executable libraries on libexec would make this more cohesive with the xen package.
There was a problem hiding this comment.
It is installed to bin, it is just located in lib in source code.
I want however to split some of sbin binaries to libexec
SigmaSquadron
added
2.status: work-in-progress
|
Pushed other batch of qubes fixes, now you can actually somehow manage VMs using qubes-manager, given that you have installed templates somehow (E.g you can copy VMs from other qubes installation). Standard qvm-template doesn't work, because it uses RPM to install such packages. @SigmaSquadron xen needs python module, do you think it is possible to split it from xen package? At least separate output will do, but ideally it needs to be built fully independently, since xen is a large package. |
|
Forgot to cherry-pick some changes from my workstation nixpkgs. Sorry for the dirty git history, I will rebase and clean it up after more polishing. |
I'm all for splitting the Python stuff from the main package, but I have no idea how to do that or how circular dependencies are even supposed to work in the first place.
Feel free to dump everything in the draft PR. We can always beat the git history into a shape later. |
Well, they don't, derivation circular dependencies needs to be resolved somehow. Some can be merged together, some can be updated to be loosely coupled (E.g use PATH or specific runtime environment variables), but they need to be fixed, derivation can't depend on itself (Except for $out, but it is only known at runtime), because derivation hash = hash of its inputs. Right now, both of discovered dependency cycles are resolved using absolute system installation paths, and I'm not sure of a proper solution. |
|
Some more information for the module: Qubes' boot parameters: virtualisation.xen.bootParams = [ "smt=off" "gnttab_max_frames=2048" "gnttab_max_maptrack_frames=4096" ];Use the C-based XenStore (as Qubes doesn't use virtualisation.xen.store.path = "${pkgs.qubes-vmm-xen}/bin/xenstored";Also, any systemd services brought in by the module need to have their |
Yeah, of course, this is just a rough draft of how it should look, nothing work in this nixos module yet. BTW, qubesd has non-critical crash on service stop when using python 3.11 due to new python features being used (QubesOS/qubes-core-admin@23be00c), and I can't upgrade it to 3.12 because of meson regression in Xen, do you have idea on how to fix that? #253751 |
The hacky solution is to |
|
Thrown even more broken code into the pile, now it is able to generate menu entries for VMs, and recognize PCI device names. Nixos module is still broken, but now it might start most of the required things by itself. |
This hacky solution sounds good enough for now, I don't know how many of the bugs are caused by python version older than upstream wants. |
CertainLach
changed the title
CertainLach
changed the title
|
Latest changes make it run in KDE Wayland session. There are still many bugs, but basic functionality works perfectly fine. |
ofborg
bot
added
the
2.status: merge conflict
| # Bug: | ||
| # Loopback vchan connection not supported | ||
| # FATAL: vchan initialization failed | ||
| # Solution: Xenstore has no set domid | ||
| # It can be set manually using | ||
| # `run0 xenstore-write "/domid" 0` | ||
| # However it should always be set by default by something, | ||
| # and I am not sure how/where. | ||
| # FIXME: Is this still relevant? I don't remember where I have encountered this error, | ||
| # but I no longer have this problem. | ||
|
|
||
| # Bug: | ||
| # Not enough memory during start of VM | ||
| # Solution: | ||
| # Qubes memory manager overrides max memory of dom0 with | ||
| # total memory available, you need to set memory limit for it. | ||
| # I have no idea what writes this memory limit in QubesOS, but | ||
| # it can be set manually: | ||
| # `run0 xenstore-write /local/domain/0/memory/static-max 100000000` | ||
| # Note that the standard `xl mem-set` doesn't work for that, as it | ||
| # uses /vm/uuid/memory field instead. | ||
| # FIXME: Permanent solution needed. |
There was a problem hiding this comment.
Both of those looks like missing xen-init-dom0 call (normally a systemd service of this name, calling /usr/libexec/xen/bin/xen-init-dom0) - part of xen package
There was a problem hiding this comment.
First problem is fixed even without domid key.
I believe it was caused by hostname being not dom0, thus causing qubes-core-admin-client to misbehave, because it can't find domain according to hostname (qubes expects adminvm to have name == dom0 with no ability to change it, and if dom0 hostname is not dom0 - then it can't detect that current domain is dom0)
This was fixed by changing default dom0 hostname to dom0, and by setting qubesdb /name key to dom0 (Which is used as an alternative method of hostname detection) for the cases where you can't use dom0 as hostname.
There was a problem hiding this comment.
qubes expects adminvm to have name == dom0 with no ability to change it
Right... I know it's stupid, but unfortunately it's hardcoded in too many places right now...
There was a problem hiding this comment.
qubesdb /name workaround works well for everything
|
This branch is a little bit outdated, I'm debugging issues occured after nixos rebase. For some reason, installation of StandaloneVM stopped working, qemu window appears, but it is always empty. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
CertainLach
force-pushed
the
qubes-packages
branch
from
9697e80 to
172bd09
Compare
|
Figured out StandaloneVM - this is caused by changes that were not included in the previous version of this PR, but in my private nixpkgs fork ++ lib.lists.optional withSeaBIOS "--with-system-seabios=${seabios}/share/seabios"This is incorrect, as @SigmaSquadron PTAL |
ofborg
bot
requested review from
globin,
hehongbo,
lovesegfault,
SigmaSquadron and
fpletz
There was a problem hiding this comment.
The build failures can probably be fixed with a rebase once #345171, #345192 and #341429 are merged.
Result of nixpkgs-review pr 341215 run on x86_64-linux 1
2 packages blacklisted:
- nixos-install-tools
- tests.nixos-functions.nixos-test
65 packages failed to build:
- appvm
- collectd
- diffoscope
- diffoscope.dist
- diffoscope.man
- docker-machine-kvm2
- gnome-boxes
- guestfs-tools
- htcondor
- libguestfs
- librenms
- libvirt
- libvirt-glib
- libvirt-glib.dev
- libvirt-glib.devdoc
- libvmi
- libvmi.dev
- libvmi.lib
- mgmt
- minikube
- multipass
- perl536Packages.SysVirt
- perl536Packages.SysVirt.devdoc
- perl538Packages.SysVirt
- perl538Packages.SysVirt.devdoc
- python311Packages.guestfs
- python311Packages.guestfs.dist
- python311Packages.libvirt
- python311Packages.libvirt.dist
- python312Packages.guestfs
- python312Packages.guestfs.dist
- python312Packages.libvirt
- python312Packages.libvirt.dist
- qemu_xen_4_17
- qemu_xen_4_17.debug
- qemu_xen_4_17.ga
- rubyPackages.ruby-libvirt (rubyPackages_3_3.ruby-libvirt)
- rubyPackages_3_1.ruby-libvirt
- rubyPackages_3_2.ruby-libvirt
- vagrant
- virt-manager
- virt-manager-qt
- virt-manager.dist
- virt-top
- virt-viewer
- xen (xenPackages.xen_4_19)
- xen.boot (xenPackages.xen_4_19.boot)
- xen.dev (xenPackages.xen_4_19.dev)
- xen.doc (xenPackages.xen_4_19.doc)
- xen.man (xenPackages.xen_4_19.man)
- xenPackages.xen_4_17
- xenPackages.xen_4_17-slim
- xenPackages.xen_4_17-slim.boot
- xenPackages.xen_4_17-slim.dev
- xenPackages.xen_4_17-slim.doc
- xenPackages.xen_4_17-slim.man
- xenPackages.xen_4_17.boot
- xenPackages.xen_4_17.dev
- xenPackages.xen_4_17.doc
- xenPackages.xen_4_17.man
- xenPackages.xen_4_18
- xenPackages.xen_4_18.boot
- xenPackages.xen_4_18.dev
- xenPackages.xen_4_18.doc
- xenPackages.xen_4_18.man
66 packages built:
- python311Packages.qubes-app-linux-usb-proxy
- python311Packages.qubes-core-admin
- python311Packages.qubes-core-admin-client
- python311Packages.qubes-core-admin.man
- python311Packages.qubes-core-libvirt
- python311Packages.qubes-core-libvirt.dist
- python311Packages.qubes-core-qrexec
- python311Packages.qubes-core-qubesdb
- python311Packages.qubes-core-qubesdb.dist
- python311Packages.qubes-desktop-linux-common
- python311Packages.qubes-desktop-linux-common.dist
- python311Packages.qubes-imgconverter
- python311Packages.qubes-imgconverter.dist
- qubes-vmm-xen (python311Packages.qubes-vmm-pyxen)
- qubes-vmm-xen.boot (python311Packages.qubes-vmm-pyxen.boot)
- qubes-vmm-xen.dev (python311Packages.qubes-vmm-pyxen.dev)
- qubes-vmm-xen.doc (python311Packages.qubes-vmm-pyxen.doc)
- qubes-vmm-xen.man (python311Packages.qubes-vmm-pyxen.man)
- python312Packages.qubes-app-linux-usb-proxy
- python312Packages.qubes-core-admin
- python312Packages.qubes-core-admin-client
- python312Packages.qubes-core-admin.man
- python312Packages.qubes-core-libvirt
- python312Packages.qubes-core-libvirt.dist
- python312Packages.qubes-core-qrexec
- python312Packages.qubes-core-qubesdb
- python312Packages.qubes-core-qubesdb.dist
- python312Packages.qubes-desktop-linux-common
- python312Packages.qubes-desktop-linux-common.dist
- python312Packages.qubes-imgconverter
- python312Packages.qubes-imgconverter.dist
- qemu_qubes
- qemu_qubes.debug
- qemu_qubes.ga
- qemu_xen (qemu_xen_4_19)
- qemu_xen.debug (qemu_xen_4_19.debug)
- qemu_xen.ga (qemu_xen_4_19.ga)
- qemu_xen_4_18
- qemu_xen_4_18.debug
- qemu_xen_4_18.ga
- qubes-artwork
- qubes-core-admin-linux
- qubes-core-agent-linux
- qubes-core-agent-linux.man
- qubes-core-libvirt
- qubes-core-qubesdb
- qubes-core-vchan-xen
- qubes-desktop-linux-kde
- qubes-gui-common
- qubes-gui-daemon
- qubes-linux-utils
- qubes-manager
- qubes-manager.dist
- qubes-seabios
- qubes-vmm-stubdom-linux
- xen-guest-agent
- xen-slim (xenPackages.xen_4_19-slim)
- xen-slim.boot (xenPackages.xen_4_19-slim.boot)
- xen-slim.dev (xenPackages.xen_4_19-slim.dev)
- xen-slim.doc (xenPackages.xen_4_19-slim.doc)
- xen-slim.man (xenPackages.xen_4_19-slim.man)
- xenPackages.xen_4_18-slim
- xenPackages.xen_4_18-slim.boot
- xenPackages.xen_4_18-slim.dev
- xenPackages.xen_4_18-slim.doc
- xenPackages.xen_4_18-slim.man
|
god bless you seriously 🙇 |
|
Another rebase, this time StandaloneVM display is broken again? I don't even understand why seabios/xen are affecting this... As far as I can see, this is just a normal VM window (...however, without colored borders, for some reason?), created by this qubes qemu component: https://github.com/QubesOS/qubes-gui-agent-xen-hvm-stubdom, which is installed in stubdom (Extracted from qubes RPMs, it is too hard to build NixOS-based stubdom right now). |
wegank
added
the
2.status: merge conflict
|
I have made some progress on QubesOS builders and building stubdom for NixOS-Qubes, however I believe it would be much easier to handle those tasks after base QubesOS packages are already merged in NixOS, even if they have suboptimal implementation (Looking at stubdom, which is extracted from QubesOS RPMs) sometimes. Another way to simplify management of this package set would be extracting them to separate flake... Thoughts? For now, I have updated all of packages to latest versions, rebased to latest master, and populated meta attributes as much as possible, to mark it ready for review. Please ignore formatting deficiencies for now (Some of patches not present here might break on reformatting, and I would like to avoid too much rebase conflicts), there is enough of crimes against humanity here without that. |
|
StandaloneVM seems to be fixed again, but I would expect that it might break again due to unknown reasons, I believe this might be fixed after unification of package versions between dom0 and stubdom, which will only happen in the future, as building our own stubdom is too complicated for now. |
|
this is awesome work @CertainLach! I've been working on Qubes packaging as well, coming at it from the angle of just creating a NixOS template vm that can be used under vanilla Qubes. I love the idea of using NixOS in dom0 as well. as I'm fairly new to Nix I packaged it as a standalone flake to get feedback on how to best upstream it: if what I've created is complementary it'd be great to collaborate on merging the two. |
|
Do all of the qubes packages integrate well with nixos? I have used Qubes and there has been some jank because of the way it's got to do things |
|
As it became too painful to maintain this patchset as a part of nixos, I decided to split it into different git repo for now. This repo follows the same naming as NixOS, so patches from it should be also appliable to NixOS, but for quicker iteration it exposes itself as an overlay+flake. After all the mandatory functionality will become useable for normal user without too much tinkering (ideally, enabling qubes option should be enough) - all patches from this repo will be applied to NixOS in a new pull request. |
|
@evq I like how you packaged many domU things, it would be great to merge our work, because my packaging of many things will only work for dom0, I would like to collaborate. I see that you managed to use qubes-builder, this solves a great PITA for me, as I have tried to package systems using pure nix builder, and it didn't work well. Maybe a path forward for dom0 qubes would be to use qubes-builder too, and then re-package resulting rpm, as qubes-stubdom-vmm-xen derivation does. |
As far as dom0 goes, I haven't encountered any problems. I even have prototype of update daemon, which delivers updates from nixos cache to dom0 using updatevm. However, I haven't worked with NixOS as TemplateVM, and I think there might be some problems with store. |
|
Best of luck. I'll start messing with it once things start working |
|
Qubes in nixos-qubes repo is working. Quick-start guide: # Add flake input
nixos-qubes = {
url = "github:CertainLach/nixos-qubes";
inputs.nixpkgs.follows = "nixpkgs";
};
# Add nixos module
modules = [
inputs.nixos-qubes.nixosModules.default
];
# Enable qubes
virtualisation.qubes = {
dom0.enable = true;
# Current qubes module implementation is not as secure as QubesOS, this option is mandatory to clarify that.
secure = false;
# Insert your primary username here, qubes is single-user system. Only this user will be able to use qubes utilities.
user = USERNAME;
# All those options are opt-outs, I.e only keep options here that you want to be active
# Configuration I provide here is supposed to keep your system running as-is for now,
# I.e you will have USB, internet access, and qubes module will not mess with your system configuration too much.
optOutRecommendedConfiguration = [
"disable-smt" # Do not disable SMT
"hostname-dom0" # Use current machine hostname, do not reset it to dom0
"dom0-restricted-usb" # Do not block USB devices in dom0
"dedicated-sys-usb" # Allow USB devices in dom0
"dedicated-sys-net" # Allow networking in dom0
];
}; |
Description of changes
You can initialize qubes database files using
qubes-createfromqubes-core-admin, and then you need to install some virtual machines - this can be done by running qvm-backup on QubesOS system, and qvm-backup-restore in NixOS.After that, basic functionality will work.
Be careful with netvms, as dom0 updates will not work with them. Some loosening up of qubes security needs to be implemented, at least having "qvm-dom0-network-via-netvm" script (Which was present in QubesOS before, and now needs to be written from scratch?) would be a good start.
At this point, you're welcome for testing, but be aware that it is not finished yet.
Cc: @SigmaSquadron
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.