Support for FreeBSD guests ?

[afbjorklund]

Description

There was some question in Slack, about supporting also FreeBSD guests.

I guess it would be "freema" or something, and not just "lima", but it is doable...

There are qcow2 images for all architectures, but it does involve adding OS.

I was just doing some experiments, so thought I would raise the question first.

[afbjorklund]

Example:

os: "FreeBSD"

images:
- location: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/amd64/Latest/FreeBSD-14.0-CURRENT-amd64.qcow2.xz
  arch: "x86_64"
- location: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/aarch64/Latest/FreeBSD-14.0-CURRENT-arm64-aarch64.qcow2.xz
  arch: "aarch64"

video:
  display: "default" 

[afbjorklund]

It is supposed to have support for OCI runtime:

https://github.com/samuelkarp/runj

There should be support, since containerd 1.5:

https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/

[jay7x]

Hi! I was the original requestor on Slack. My need is to run a Puppet module acceptance tests on a FreeBSD. Lima is really attractive here because it's headless and scriptable. People are using VirtualBox (with or w/o vagrant) to do this usually. But after I got the M1-based Macbook I lost the ability to do this. In my case there is no real urgency but it'd be really nice to have. I'll try it on a coming week and will share the experience here.

[afbjorklund]

The support for linux containers (on freebsd) seems to have merged as well, in 1.7.

https://productionwithscissors.run/2022/09/04/containerd-linux-on-freebsd/

[afbjorklund]

Short list of portability issues:

• There is no /run, only /var/run (added variable)
• There is no /bin/bash, only /usr/local/bin/bash. (added variable)

And of course: s/linux/freebsd/

The LocalPorts support for guestagent can be implemented using sockstat.
Note: the output of the sockstat command is subtly different, between the OS.

• Investigating guestagent on FreeBSD, for containerd with runj #1509

The sshfs options for "fusefs" does not have allow_other, unfortunately:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215797

Not sure if -o slave works, but there is some -o passive flag instead ?
Also the kernel module is not loaded by default, needs /boot/loader.conf

But otherwise, it seems to be working:

   sudo mkdir -p /mnt/lima-cidata
   sudo mount -t cd9660 /dev/cd0 /mnt/lima-cidata
   sudo cp /mnt/lima-cidata/meta-data /run/lima-ssh-ready
   sudo install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
   sudo -b lima-guestagent daemon
   sudo cp /mnt/lima-cidata/meta-data /run/lima-boot-done

Created the user manually, for now.

$ limactl shell freebsd uname
To see the output from when your computer started, run dmesg(8).  If it has
been replaced with other messages, look at /var/run/dmesg.boot.
		-- Francisco Reyes <[email protected]>
FreeBSD

Probably want to disable those "tips"...

# $FreeBSD$
#
# .login - csh login script, read by login shell, after `.cshrc' at login.
#
# See also csh(1), environ(7).
#

# Query terminal size; useful for serial lines.
if ( -x /usr/bin/resizewin ) /usr/bin/resizewin -z

# Display a random cookie on each login.
if ( -x /usr/bin/fortune ) /usr/bin/fortune freebsd-tips

[XVilka]

Asked this some time ago: #1096

Also, since QEMU 8.0 there is a guest agent support for *BSD systems:

• https://www.qemu.org/docs/master/interop/qemu-ga.html
• https://wiki.qemu.org/ChangeLog/8.0#Guest_agent

[afbjorklund]

Thanks for the reminder. I used QEMU 4 ☺️

• *BSD support (VM) #1096

[afbjorklund]

The support is quite rough, but nerdctl and friends (containerd and runj) are available in the default pkg install.
It does require setting up a ZFS disk and pool, but this can be done using the limactl disk feature (/dev/vtbd1)

[anders@freebsd ~]$ sudo nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
WARN[0000] unable to determine runc version: exec: "runc": executable file not found in $PATH 
Client:
 Version:	v1.2.1
 OS/Arch:	freebsd/amd64
 Git commit:	<unknown>
 buildctl:
  Version:	

Server:
 containerd:
  Version:	1.7.0
  GitCommit:	unknown
 runc:
  Version:	
[anders@freebsd ~]$ runj --version
runj version v0.0.1-dev ()
go: go1.20.3
[anders@freebsd ~]$ sudo nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: 1.7.0
 Storage Driver: native
 Logging Driver: json-file
 Cgroup Driver: 
 Cgroup Version: 
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: zfs native
 Security Options:
 Kernel Version: 14.0-CURRENT
 Operating System: FreeBSD 14.0-CURRENT
 OSType: freebsd
 Architecture: amd64
 CPUs: 0
 Total Memory: 0B
 Name: freebsd
 ID: ebcf5f96-01a3-4f40-a063-e3f64163c948

You can run native (freebsd) containers, but so far I have not been able to run linux containers (issues mounting the fake /proc ) even though linux binaries (like limactl) do work on the VM (after enabling linux service and centos-7)

[afbjorklund]

https://www.freebsd.org/status/report-2023-01-2023-03/#_freebsd_as_a_tier_1_cloud_init_platform

One could probably do a simple user-data parser calling pw etc, before the python version is up

[afbjorklund]

Here is an example of a freebsd container image:

public.ecr.aws/samuelkarp/freebsd:13.1-RELEASE

[afbjorklund]

Currently the workaround for missing cloud-init is to open a GUI console.

video:
  display: "default"

From there, it should be possible to log in as root once the system is booted.

The template is using FreeBSD 14.0-CURRENT which is the upcoming release.

---

Telling lima that the ssh setup is complete:
#1508 (comment)

Instructions for adding additional users:
https://docs.freebsd.org/en/books/handbook/basics/#users-synopsis

Instructions for adding containerd packages:
https://docs.freebsd.org/en/books/handbook/ports/

Instructions for setting up a ZFS disk pool:
https://docs.freebsd.org/en/books/handbook/zfs/

[afbjorklund]

Similar work for Podman:

• Allow specifying a guest OS in podman machine init. containers/podman#19939

[fwilhe2]

Hi @afbjorklund, are there any updates on this? I think this would be a great way to tinker with FreeBSD.

[afbjorklund]

I don't have any further updates, but could do a rebase of the existing branch (#1509) if it helps anyone?

I guess 14.0 is out now, but I don't know if runj is more mature... Or if it is going to be in containerd 2.0

https://www.freebsd.org/releases/14.0R/announce/ (2023-11-20)

https://www.freebsd.org/status/report-2024-01-2024-03/cloud-init/

[glloydNAT]

I'm pretty excited about this idea but... what do you think about adding support for netbsd as well?

[afbjorklund]

Not sure there are any official cloud images, would need to use e.g. https://bsd-cloud-image.org/

What would a NetBSD image be used for? For FreeBSD, there was the concept of testing runj etc

If it is just to run a virtual machine with BSD, then you could use https://mac.getutm.app/gallery/

[glloydNAT]

I realize there may not be an OCI-compatible runtime native/exclusive to NetBSD and that, therefore, it may fall outside the purview of this project...

I am interested, however, in NetBSD's veriexec functionality as it could relate to running containerized apps. Similarly, a lock-down mechanism like securelevel.

I don't know.. just a thought.

[jay7x]

Just found some time to try FreeBSD in lima finally. My current goal is to be able to ssh into the VM in plain mode.

Tried BASIC-CLOUDINIT with UFS from here: https://download.freebsd.org/releases/VM-IMAGES/14.2-RELEASE/aarch64/Latest/.

It stuck on waiting for ssh. Console shows login prompt but no input from keyboard was accepted. Though, I was able to login via serial console. Network was good, ssh port was forwarded. But user was not created. Later I found there is no cloud-init at all in this image 🤔 Will check other images later.

[afbjorklund]

ater I found there is no cloud-init at all in this image

Apparently FreeBSD has started to use a cloud-init replacement called nuageinit: https://reviews.freebsd.org/D44141

It should be able to read the meta-data and user-data, that Lima creates on the cidata.iso (the "nocloud" citype)

https://github.com/freebsd/freebsd-src/tree/main/libexec/nuageinit (using lua)

831M https://download.freebsd.org/releases/VM-IMAGES/14.2-RELEASE/amd64/Latest/FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.zfs.qcow2.xz

770M https://download.freebsd.org/releases/VM-IMAGES/14.2-RELEASE/aarch64/Latest/FreeBSD-14.2-RELEASE-arm64-aarch64-BASIC-CLOUDINIT.zfs.qcow2.xz

Using ZFS instead of UFS, since it seems that it will be required for containers? For now, from freebsd podman testing

We are working on making Union File System work (will be similar to Linux overlayfs).

[jay7x]

Ah, that's interesting! I should try it on the weekend!

[afbjorklund]

@jay7x : I rebased the branch (guestagent-freebsd) - updated to lima master, with FreeBSD 15 using ZFS

It boots, but it looks like the custom lua parser is struggling with the cloud-config that Lima is generating...

The good news is that you can probably troubleshoot that (nuageinit) without needing the whole round-trip?

EDIT: Without the write_files, the user does get created. But the key is not added, since it doesn't support legacy*.

* lima still uses ssh-authorized-keys

it needs to use ssh_authorized_keys

---

The file sharing works too, but the kernel module is not loaded by default. https://reviews.freebsd.org/D41844

kldload /boot/kernel/virtio_p9fs.ko
mkdir /home/$USER /tmp/lima
mount -t p9fs mount0 /home/$USER
mount -t p9fs mount1 /tmp/lima

The packages for nerdctl and friends are available in the system, pkg install nerdctl containerd runj.

[anders@lima-freebsd ~]$ sudo nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
WARN[0000] unable to determine runc version: exec: "runc": executable file not found in $PATH 
Client:
 Version:	v2.0.3
 OS/Arch:	freebsd/amd64
 Git commit:	<unknown>
 buildctl:
  Version:	

Server:
 containerd:
  Version:	2.0.2
  GitCommit:	unknown
 runc:
  Version:	

[afbjorklund]

Here is a reproducer:

local yaml = require("yaml")

	local f, err = io.open("user-data")
	if err then
		nuage.err("error parsing nocloud user-data: " .. err)
	end
	local obj = yaml.eval(f:read("*a"))
	f:close()
	if not obj then
		nuage.err("error parsing nocloud user-data")
	end

For debugging yaml.lua:

https://github.com/freebsd/freebsd-src/blob/main/libexec/nuageinit/yaml.lua

lua5.3: ./yaml.lua:212: SyntaxError: invalid indentation, got 3.0 instead of 0.5, near "\n      #!/bin/sh\n      se"
stack traceback:
	[C]: in function 'error'
	./yaml.lua:212: in function 'yaml.tokenize'
	./yaml.lua:582: in function 'yaml.eval'
	test.lua:7: in main chunk
	[C]: in ?

---

Bug report, with patch: (parser still has other bugs, like skipping over #!/bin/sh)

• Text block followed by more items skips said items exosite/lua-yaml#15

The other bug is that the parser treats everything starting with # as "comment".

This means that it will mangle the contents of write_files, even when handling indent.

[afbjorklund]

Refactored the regular Makefile, to make adding FREEBSD next to LINUX be a smaller change...

• Refactor os and arch conditionals in Makefile #3206

Similar to adding the OS field to the regular template, even if there is only one OS on master.

There are some other hardcoded items, like $HOME.linux for home or GNU bash for the shell.

pkg/osutil/user.go: limaUser.HomeDir = "/home/{{.User}}.linux"

pkg/cidata/cidata.TEMPLATE.d/user-data: shell: /bin/bash

• Make it possible to configure user shell #3207

[afbjorklund]

Getting the boot scripts (i.e. non-plain) to work in FreeBSD will be a bit of a struggle...

Currently they are all using /bin/bash, but nuageinit does not implement "packages".
So it will be hard to install the required /usr/local/bin/bash during the initial boot.

The above configuration is only for the user login shell, not for the system scripts.

Porting all the boot scripts from GNU bash to plain /bin/sh might be too much work?
Thus it might have to wait until FreeBSD cloud-init has support for installing packages.