Assemble templates using the new base setting

[jandubois]

This PR creates initial support for multi-file templates, mostly based on the suggestions I gave in #2520 (reply in thread).

Instance create vs. start stages

The basic idea behind "template embedding" is that when an instance is created, the lima.yaml that is stored in the instance directory will embed settings from additional
base template(s) and external scripts into a single composite document.

Once the instance is created, there are no longer external references and the instance can be started while offline, even if it was created from online template fragments.

Merging default settings from base templates

Merging of templates is controlled by the new base property, which takes the name of
a base template, e.g.

base: template://docker

It is expected that in most instances only a single base template is needed, but it is possible to provide a list:

base:
- variables.yaml
- http://example.com/template.yaml

The base template can in turn have their own base templates. For example my.yaml instance may be based on template://docker, which in turn could be based on template://ubuntu.

Merging is performed by copying every property from the base template that does not yet exist in the main template. For list properties the lists are appended. More details and exceptions are documented below.

Embedding of external provisioning scripts

Both provisioning scripts and probe scripts can now be located with the new file property specifying a template locator (details below). The file will be loaded and inlined under the script property. Therefore file and script are mutually exclusive.

provision:
- mode: user
  file: my-script.sh
probes:
- file: template://docker-probe.sh

Relative template locators

The values of the base and file properties are "template locators". They are the usual local filenames or file://, https://, or template://URLs.

In addition they can also be relative locators: names that don't have a URL scheme and don't start with a /. These names are resolved relative to the template in which they occur (similar to relative href properties in HTML).

base: base.yaml
provision:
- file: script.sh

If this template is loaded as templates/main.yaml, then the referenced template files will be fetched from templates/base.yaml and templates/script.sh.

If the template is loaded as template://main, then the files will be template://base.yaml and template://script.sh.

This means a template using relative locators will always fetch dependencies from the same (relative) location regardless of how it was loaded itself.

To make this work template://base.yaml needs to be equivalent to template://base. And it must be possible to use other file extensions, so template://script.sh loads the script and doesn't look for script.sh.yaml.

limactl template command

This PR adds new options to the limactl template copy subcommand.

• --verbatim copies a file exactly as is.

• --embed will merge in all external base templates and shell scripts.

• --embed-all will also merge in templates and files referenced by template:// URLs.

• --fill will apply override.yaml, default.yaml, and builtin defaults.

Without additional options the copy command will copy the file, but turn any relative locators into absolute locators. So the copy of the template will continue to reference the same base templates and scripts. Use the --verbatim option if you intend to copy those files locally as well.

Examples

Embedding a script

Merging in template://default

Detailed merging algorithm

Each template will recusively embed all dependencies in its base templates before merging them in.

Then each property, that does not exist in the main template will be copied over from the base template.

Properties that are lists are appended.

For provision and probes the order is reversed: the base entries are inserted before the main template entries, so that latter are executed last (see #2932 for discussion).

Special rules

The dns entries are not merged. The entries from a base template are only copied when the main template list is empty.

mountTypesSupported will have any duplicate entries removed.

The base property is removed if the base template has been embedded.

For provision and probes the file property is removed, and the actual content of the script is stored in the usual script property.

For minimumLimaVersion and vmOpts.qemy.minimumVersion the values are compared. and the base version is only copied when it is greater than the main version.

Combining list entries

Three list properties (additionalDisks, mounts, and networks) use combining logic based on a unique key (disk name, mount point, and interface name, respectively).

If a latter entry matches the key of an earlier entry, then any missing fields in the earlier entry are filled in from the latter entry, which is then deleted. This matches the logic already used for merging override.yaml and default.yaml.

There is a new wildcard feature now, where * will match any existing key, so can be used to provide defaults for all entries. This is mostly in preparation to use this mechanism for the builtin defaults.

For the combining list mechanism to work, all base templates must be merged (so all lists must contain the complete set of entries). Otherwise the key matching mechanism wouldn't work globally, but only for the parent template at each level.

[jandubois]

This PR still needs some integration tests, but I wanted to create it before 2024 was over. 😄

[jandubois]

@AkihiroSuda I believe I've addressed all your code review feedback so far, as well as all the implications from our discussion about digest support (the schema has been updated, but the digest computation/checking is left for a future PR).

I've rebased this PR since you originally reviewed it, but any further changes are all in separate commits. I've not yet squashed them, in case it makes it easier for you to continue the review; but let me know if you prefer that I squash them now!

[jandubois]

I will briefly to turn this into Draft mode again because I think there is another condition in useAbsLocators that is not fully implemented, and I need to figure out how to add a test for it.

[jandubois]

I had to rebase the PR to resolve the merge conflict with recent changes to pkg/limayaml/marshal.go. There are no other changes.

[AkihiroSuda]

Let's merge, after another rebase to add the licensing boilerplate

[jandubois]

Let's merge, after another rebase to add the licensing boilerplate

Rebased and copyright headers added...

[AkihiroSuda]

Thanks

[AkihiroSuda]

Created release/1.0 branch before merging this PR, so that we can release v1.0.x if we need