anita.1

.Dd Jul 7, 2022
.Dt ANITA 1
.Os
.Sh NAME
.Nm anita
.Nd Automated NetBSD Installation and Test Application
.Sh SYNOPSIS
.Nm
.Op Fl -workdir Ar work_directory
.Op Fl -vmm Ar qemu | xl | xm | gxemul | simh
.Op Fl -vmm-args Ar vmm_arguments
.Op Fl -disk-size Ar size
.Op Fl -run Ar command
.Op Fl -sets Ar sets
.Op Fl -test-timeout Ar timeout
.Op Fl -persist
.Op Fl -boot-from Ar cdrom | floppy
.Op Fl -structured-log
.Op Fl -structured-log-file Ar file
.Op Fl -tests Ar kyua | atf
.Op Fl -dtb Ar dtb
.Op Fl -image-format Ar format
.Op Fl -machine Ar machine
.Op Fl -partitioning-scheme Ar scheme
.Op Fl -xen-type Ar pv | pvshim | hvm | pvh
.Op Fl -no-entropy
.Ar mode
.Ar URL
.Sh DESCRIPTION
.Nm
is a tool for automated testing of the NetBSD installation procedure
and of NetBSD in general.  Using
.Nm ,
you can fully automate the process of downloading a NetBSD
distribution, installing it in a virtual machine, booting the
installed system, and running the NetBSD test suite.
.Pp
The NetBSD ports currently supported as targets (i.e., as the system
to install and run under emulation) are i386, amd64, sparc, sparc64,
evbarm-earmv7hf, evbarm-aarch64, pmax, hpcmips, vax, hppa, macppc,
alpha, and riscv-riscv64.  The host (the system running
.Nm )
can be any NetBSD port or even a different Unix-like system such
as Linux, FreeBSD, or macOS.
.Pp
i386 and amd64 targets can be run under either qemu
or Xen.  Other targets use either qemu, gxemul, or simh
depending on the target architecture.
.Sh EXAMPLES
To install NetBSD 10.0/i386 from the master NetBSD FTP site, enter
.Pp
.Dl anita install http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/
.Pp
Installing NetBSD/sparc, sparc64, vax, or hppa works a bit differently:
they use an ISO image instead of a directory containing boot floppies and sets:
.Pp
.Dl anita install http://ftp.netbsd.org/pub/NetBSD/iso/10.0/NetBSD-10.0-sparc.iso
.Pp
The evbarm and riscv ports do not currently support installation
using sysinst, so anita will instead install them by decompressing
the pre-built disk image from the release onto the beginning
of the virtual disk.  The target will then be rebooted to trigger
the automatic image resizing that happens on first boot,
and after that, these ports work the same as the other ports.
To download an evbarm or riscv image, use a command like the following:
.Pp
.Dl anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/
.Dl anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/riscv-riscv64/
.Pp
To boot the newly installed hard disk image and get a login prompt,
replace
.Ar install
with
.Ar interact :
.Pp
.Dl anita interact http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/
.Pp
When you are done interacting with the virtual machine, you can kill it by
typing control-a x (qemu) or control-c (gxemul).
.Pp
To run the NetBSD test suite on the installed system, use
.Pp
.Dl anita test http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/
.Pp
If you have a recent NetBSD host and a qemu version that supports
NVMM (the NetBSD Virtual Machine Monitor), you can use it to run the
tests several times faster than using qemu's built-in emulation:
.Pp
.Dl anita --vmm-args="-accel nvmm" test http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/
.Pp
To install a snapshot, use something like the following (adjusting
the version number in the sparc URL as needed):
.Pp
.Dl anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/i386/
.Dl anita install http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/latest/images/NetBSD-10.99.7-sparc.iso
.Pp
If you have built a NetBSD release locally using
.Ic "build.sh -R" ,
you can point
.Nm
directly at the RELEASEDIR or ISO using a "file:" URL:
.Pp
.Dl anita install file:///path/to/releasedir/i386/
.Pp
or simply use an absolute pathname as shorthand for the above:
.Pp
.Dl anita install /path/to/releasedir/i386/
.Sh HOW IT WORKS
.Nm
works by "screen scraping" the sysinst output over an emulated
serial console, recognizing sysinst prompts and sending canned
responses.
.Pp
.Nm
will create a work directory and cache the distribution sets, boot
floppies, and a hard disk image in it.  By default, the work directory
is a subdirectory of the current working directory, with a unique
name automatically generated from the distribution URL, for example,
.Pp
.Dl work-http---ftp.netbsd.org-pub-NetBSD-NetBSD-10.0-i386-+a4c39
.Pp
You can also specify the name of the work directory explicitly using
the
.Fl -workdir
option.
.Pp
When you rerun
.Nm
with the same URL, files that already exist in
the work directory will not be re-downloaded or rebuilt, so if you
run you
.Ic "anita install"
with the same URL twice in a row, the second
run will effectively be a no-op, and if you rerun
.Ic "anita interact" ,
the system will be booted from the existing disk image, skipping the
installation stage.  To force things to be redone, simply remove the
work directory.  If you remove just the hard disk image file
.Pa wd0.img ,
.Ic "anita install"
will recreate it from the cached distribution files.
.Pp
To ensure that the cached system state is always that of a
freshly installed system,
.Nm
enables the
.Ic qemu
snapshotting feature for the system disk by default (but not for other disks).
Therefore, if you log in and make changes to the system, they will not
be saved to the disk image file.  To override this behavior, specify the
.Fl -persist
option.
.Pp
.Sh MODES
The operation performed by
.Nm
is determined by the
.Ar mode
argument, which takes the following values:
.Bl -tag -width indent
.It Ar install
Install NetBSD if not already installed.
.It Ar boot
Install NetBSD if not already installed, then boot the
installed system to the login prompt.
.It Ar interact
Install NetBSD if not already installed, boot it,
and then connect the serial console to the terminal for
interactive use.  The
.Cm qemu
escape character control-a is in effect; for example, you can use
control-a x to exit, control-a c to enter the
.Cm qemu
monitor, or control-a b to send a break (useful for entering DDB).
Interacting with the boot blocks and other aspects of the boot process
is possible unless the
.Fl -run
option has been specified.  With
.Fl -run ,
the interactive session only starts once the
.Fl -run
command has been successfully run.
.Pp
.It Ar test
Install NetBSD if not already installed, then boot it and
run the test suite from
.Pa /usr/tests .
By default, the test suite is run using ATF.  If NetBSD was built with
.Sq MKKYUA=yes ,
you can use Kyua instead by passing the option
.Fl -tests
.Ar kyua .
The raw output of the test execution and various other test reports are
stored in a
.Pa tests/
subdirectory under the work directory.
.Pp
When Kyua is used to run the test suite, the
.Pa tests/
subdirectory contains the raw output of the tests execution, a copy of
the Kyua database in the
.Pa store.db
file with the results of the tests, and an itemized HTML report in the
.Pa html/
subdirectory.
.Pp
When the ATF tools are used to run the test suite, the
.Cm atf-report
output will be displayed on standard output and the following output
files are placed in the
.Pa tests/
subdirectory: the raw
.Cm atf-run
output in
.Pa test.tps ,
the output from
.Cm "atf-report -o xml"
in
.Pa test.xml ,
and the output from
.Cm "atf-report -o ticker"
in
.Pa test.txt .
To facilitate the further processing of the XML output into HTML,
the files
.Pa tests-results.xsl ,
.Pa tests-results.dtd ,
and
.Pa tests-results.css
are also included.
.It Ar print-workdir
Print the pathname of the work directory on standard output.
This is intended for use by scripts that need to access files
in the work directory, particularly when the
.Fl -workdir
option is not used but the name of the directory is automatically
generated.
.El
.Sh OPTIONS
The following command line options are supported:
.Bl -tag -width indent
.It Fl -workdir Ar directory
The work directory.  The default is an automatically generated
name under ".".
.It Fl -vmm Ar qemu | xl | xm | gxemul | simh
Specify the virtual machine monitor.
The default virtual machine monitor is qemu.
If the target system architecture is not supported by qemu,
.Nm
will automatically switch to
.Cm gxemul
or
.Cm simh
as needed.
If
.Nm
is running in a Xen dom0 and the target system architecture
is i386 or amd64, it is also possible to
specify
.Cm xl
or
.Cm xm ,
which will make
.Nm
install the target system in a Xen domU using the current
.Cm xl
or the historic
.Cm xm
interface, respectively.  This requires
running
.Nm
as root.  Any changes made to the system disk image will
be persistent whether or not the
.Fl -persist
option was given.  The Xen support should be considered experimental
and may not be fully tested.  For backwards compatibility,
.Ar xen
is accepted as a synonym for
.Ar xm .
.It Fl -vmm-args Ar string
Additional arguments to pass to the virtual machine monitor (e.g., qemu).
The arguments are given
as a single string, which may contain multiple arguments separated
by whitespace.  There is no way to pass an argument containing
whitespace.  This option was formerly called
.Fl -qemu-args ;
the old name is still accepted for backwards compatibility.
.It Fl -disk-size Ar size
The size of the virtual disk NetBSD gets installed on.  The default
is large enough to hold the OS installation itself when also using
default values for the
.Fl -sets
and
.Fl -memory-size
options, but if you need
additional space, you can specify a larger size.  The size is given in
bytes, or a suffix of k, M, G, or T can be used for kilo-, mega-,
giga-, or terabytes.
.It Fl -memory-size Ar size
The size of the virtual RAM.  The size is given in
bytes, or a suffix of k, M, or G can be used as with
the
.Fl -disk-size
option.  The default is architecture dependent.  Note that since
sysinst sizes the swap partition based on the amount of RAM, if you
run
.Cm anita install
with a large
.Fl -memory-size ,
you may also have to increase
.Fl -disk-size .
.It Fl -run Ar command
Log in to the virtual machine as root and execute the given shell
.Ar command
in it once it has booted.  This is only meaningful when used with the
.Ar boot
or
.Ar interact
command.  Since the command is sent to an interactive shell over the
console tty, it should be kept short and simple to avoid running into tty
limitations or quoting issues.  Complex commands may be executed by
preparing a disk image containing a file system containing a shell
script, and specifying something like
.Pp
.Dl --vmm-args '-hdb disk.img' --run 'mount /dev/wd1a /mnt && /mnt/script'
.Pp
The shell command is run using
.Cm /bin/sh
regardless of the login shell of the root user on the target system.
The exit status of the shell command is returned as the exit status
of
.Nm .
.It Fl -sets Ar sets
The distribution sets to install, as a comma-separated list.
For a minimal install, use something like
.Pp
.Dl --sets kern-GENERIC,modules,base,etc
.Pp
A kernel, base, and etc must always be included.
.It Fl -run-timeout Ar timeout
Set a timeout for the tests run using the
.Fl -run
option, in seconds.  The default is 3600 seconds (one hour).
.It Fl -test-timeout Ar timeout
Set a timeout for the tests run in the
.Cm test
mode, in seconds.  Starting with
.Nm
version 2.2, the timer is reset at the beginning of each ATF test
program, so the timeout only needs to be greater than the duration
of the longest test program rather than the full test run.
The default is 10800 seconds (3 hours).
.It Fl -persist
Store any changes to the contents of the system disk persistently,
such that they may affect future
.Nm
runs, instead of the default behavior where only the
.Ar install
mode can modify the disk contents and all other modes work with
an ephemeral snapshot copy of the freshly installed system.
.It Fl -boot-from Ar cdrom | floppy | kernel
For architectures that support booting from more than one type of
media (typically CD-ROM or floppies), specify which one to use.
Most architectures only support one type of boot media, and
specifying an unsupported typ will cause the boot to fail.  Due
to limitations of the emulators or their emulated firmware, some
architectures cannot be booted from install media at all, and
instead boot by passing a kernel directly to the emulator.
The default is
.Ar floppy
for i386,
.Ar kernel
for alpha, and
.Ar cdrom
for all other targets.
.It Fl -structured-log
Generate log output in a structured format similar to Python code.
Commands sent to the serial console by
.Nm
are logged as
.Cm send(t, 'command...') ,
where
.Va t
is a timestamp in seconds since the Unix epoch.
Data received are logged
as
.Cm recv(t, 'data...') .
Calls to pexpect's
.Fn expect
function are logged as
.Cm expect(t, 'regexp...') ,
and the actual strings matched by them as
.Cm match(t, '...') .
Unprintable characters in the data strings are escaped using Python
string syntax.
.Pp
The default is to do unstructured logging where the raw output from
the virtual machine console is sent to standard output as-is,
and the commands sent to the console are only logged if echoed
by the virtual machine.
.It Fl -structured-log-file
Like
.Fl -structured-log ,
but logs to a given file rather than to standard output,
and in addition to rather than instead of the default
unstructured logging.
.It Fl -tests Ar kyua | atf
The test framework to use for running tests. The default is
.Cm atf .
.It Fl -dtb
The location of the Device Tree Blob file, needed with the
.Ar evbarm-earmv7hf
port when using the default
.Ar vexpress-a15
machine type.  The default is the location of the file
.Pa vexpress-v2p-ca15-tc1.dtb
in the
.Pa dtb-arm-vexpress
package assuming both it and
.Nm
itself have been installed via pkgsrc.
.It Fl -xen-type Ar type
Select the type of virtualization to use with Xen.  This can be
.Ar pv ,
.Ar pvshim ,
.Ar hvm ,
or
.Ar pvh .
The default is
.Ar pv .
.It Fl -image-format Ar format
The disk image format to use for the virtual machine's system disk.
Supported values are
.Ar dense ,
a raw disk image that has been fully preallocated by writing zeros,
and
.Ar sparse ,
a raw disk image with holes (when supported by the underlying file
system).  The default is
.Ar dense .
.It Fl -machine Ar machine
The machine type to emulate.  This may be used with the
evbarm-earmv7hf port to select the
.Ar virt
qemu machine type instead of the default of
.Ar vexpress-a15 .
Not consistently supported for other ports and VMMs.
.It Fl -partitioning-scheme Ar scheme
Use the given partitioning scheme instead of the port's default one.
Currently only supported for the i386 and amd64 ports, where a scheme
of
.Ar MBR
can be selected instead of the default of GPT.
.It Fl -no-entropy
If sysinst prompts the user to enter entropy and offers an option not
to enter it, select that option.  This is intended for testing the
behavior of said option and the resulting installation, and only works
with certain versions of NetBSD that offer such an option.  The
default is to supply the guest being installed with entropy from the
host.
.El
.Sh DEBUGGING NETBSD USING ANITA
.Nm
can serve as a convenient platform of installing and booting NetBSD
for debugging purposes.  When doing this, it is useful to build NetBSD
with debug symbols and to install the source on the virtual machine to
enable source-level debugging.
.Pp
For more details, please see
.Dl http://wiki.netbsd.org/kernel_debugging_with_qemu/ .
.Sh SEE ALSO
.Xr atf-report 1 ,
.Xr atf-run 1 ,
.Xr qemu 1 ,
.Xr kyua 1 ,
.Xr tests 7
.Sh BUGS IN ANITA
.Nm
supports only a limited number of NetBSD ports.  There may
be other ports with working emulators, and support should
be added for those. Patches are welcome.
.Pp
.Nm
is likely to break whenever any significant change is made to
the sysinst user interface.
.Pp
Installing NetBSD releases older than 2.1 has not been tested.
.Pp
Exporting ATF test reports to the host system does not yet work
with the evbarm-earmv7hf or hpcmips ports for lack of support for
a second block device.
.Pp
.Sh BUGS IN NETBSD
.Pp
NetBSD/i386 releases older than 4.0 will install, but when booting
the installed image, they hang after the "root on ffs" message.
.Pp
NetBSD/i386 versions older than 2009-06-13 13:35:11 fail to find
any PCI buses when run under qemu; see PRs 38729 and 42681.
.Pp
NetBSD/vax is unable to run the ATF tests as of source date
2018.03.22.12.16.11.
.Pp
Timing is off by a factor of two when running on NetBSD hosts;
see PR 43997.
.Pp
For current reports of other NetBSD bugs found using anita, see
.Pp
.Dl http://releng.netbsd.org/test-results.html
.Pp
.Sh BUGS IN QEMU
.Pp
Versions of qemu known to work with
.Nm
on NetBSD hosts are
0.15, 1.2, 1.4.1, and 2.0.0nb4
or newer.
.Pp
Some floating point tests fail under qemu but not on real hardware,
presuambly due to bugs qemu's floating point emulation.
In particular, floating point exceptions are broken when using
the default TCG acceleration, as reported in
.Pp
.Dl https://gitlab.com/qemu-project/qemu/-/issues/215
.Pp
Running multithreaded programs (such as the NetBSD test suite) on an
emulated i386 or amd64 system used to require qemu patches that were
in pkgsrc beginning with qemu 0.12.3nb3. They were finally integrated
into qemu on 2011-12-11.  See PR 42158 and
.Pp
.Dl https://bugs.launchpad.net/bugs/569760
.Pp
for details.
.Pp
When attempting to install NetBSD-current in qemu 1.0, it panics
during the install kernel boot due to a regression in qemu's
emulation of the PCI configuration registers.  The work-around
is to use qemu 0.xx.  See PR 45671 and
https://bugs.launchpad.net/qemu/+bug/897771 for details.
This bug has since been fixed on the qemu mainline.
.Pp
In addition to the above, there have been several further
regression in the 1.x series of qemu that have impacted
.Nm :
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1089996
.Dl https://bugs.launchpad.net/qemu/+bug/1091241
.Dl https://bugs.launchpad.net/qemu/+bug/1127369
.Dl https://bugs.launchpad.net/qemu/+bug/1154328
.Pp
These are believed to be fixed in qemu 1.5.
.Pp
Installing NetBSD 5 or older on i386 or amd64 takes a long
time with recent versions of qemu because the bootloader countdown
runs at 1/20 the normal speed, and there is a long delay between
loading the kernel and the kernel printing its first console output,
which can easily be mistaken for a hang.  Please be patient. This
issue has been worked around in NetBSD 6 and newer; see PR 43156 for
details.
.Pp
Installing using qemu version 1.5.1, 1.6.0, 1.7.0, or 2.0.0
prior to 2.0.0nb4 on
NetBSD fails due to the serial console dropping characters; see
PR 48071 and the qemu bug reports
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1335444
.Dl https://bugs.launchpad.net/qemu/+bug/1399943
.Pp
The same qemu versions work when hosted on Linux, but
only by accident.  This bug is fixed for i386 and amd64
targets in qemu 2.0.0nb4 in pkgsrc and the qemu 2.1 release,
and finally fixed for sparc in qemu 2.8.
.Pp
Sending a break sequence to the serial console using
"control-a b" was broken, fixed, broken again, fixed
again, broken again as the fix was reverted
for causing another regression, and finally fixed again
in qemu 3.0:
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1654137
.Pp
Interacting with the boot blocks over the serial console
is also currently broken:
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1743191
.Pp
The qemu
.Fl icount
option looks useful for making the tests less dependent on host timing,
and its
.Ar sleep=on|off
argument might be used to speed up the tests by emulating the passage
of time instead of waiting for actual time to pass.  This is now
partly functional as
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1774677
.Pp
has been fixed, but causes spurious IDE disk errors in the guest.
Qemu also advertises record/replay capability, but it does not actually
work, as reported in
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1810590
.Pp
In qemu 5.1.0, booting NetBSD/sparc did not work:
.Pp
.Dl https://bugs.launchpad.net/qemu/+bug/1892540
.Pp
This is fixed since 5.2.0.
.Pp
.Sh BUGS IN KVM
.Pp
When
.Nm
is run on a Linux host using qemu-kvm 0.12.3, and is used to
boot a version of NetBSD-current newer than 2009-11-04
14:39:17, the emulated NetBSD system hangs during boot; see
PR 44069 for details.  This issue can be worked
around by passing Anita the command line option
.Fl -vmm-args
.Ar -no-kvm
to disable kvm.  The alternative
.Fl -vmm-args
.Ar -no-kvm-irqchip
performs better but doesn't quite work: the system installs and
boots, but the test suite occasionally fails to complete; see PR 44176.
.Pp
As of 2020, the above issue has been fixed.
.Pp
.Sh BUGS IN PYTHON
.Pp
Versions of
.Nm
prior to 1.40 may get the error
.Pp
.Dl [Errno ftp error] 200 Type set to I
.Pp
during the downloading of distribution sets; this is a regression in
Python 2.7.12:
.Pp
.Dl http://bugs.python.org/issue27973
.Pp
This problem is worked around in
.Nm
1.40.