modules: enforce modules:default:roots:tcl to be unset and arch_folder:false (#276)

This is a minor bug fix, since it does not apply to any recipe currently
published in https://github.com/eth-cscs/alps-uenv, but it represents a
good sanity check that might help users.

Since the default configuration for spack would be to consider
`arch_folder: true`, if the recipes don't explicitly set it, the modules
get generated with an additional folder level, namely the arch_folder
(e.g. `/user-environment/modules/linux-sles15-neoverse_v2` instead of
simply `/user-environment/modules`).

The problem arises with usage, because `uenv` sets `MODULEPATH` to
`/user-environment/modules`, so the modules are not anymore reachable
simply by name, but they need to be prefixed with the specific
arch_folder (e.g. `linux-sles15-neoverse_v2/cmake` instead of `cmake`).
That does not translates just to a burden for the user that has to
prefix all module names with the arch_folder, but it actually does not
work because modules dependencies in modules themselves are not prefixed
with the `arch_folder`.

Spack implements this functionality by adding to `MODULEPATH` all
arch_folders available on the system.

I think we can live without functionality, but we should anyway enforce
it to catch earlier (at build time) potential problems (which would
otherwise show up just at usage time).

Author: albestro

docs/recipes.md

@@ -7,7 +7,7 @@ A recipe is comprised of the following yaml files in a directory:
 * `compilers.yaml`: the compilers provided by the stack.
 * `environments.yaml`: environments that contain all the software packages.
 * `modules.yaml`: _optional_ module generation rules
-    * follows the spec for [spack module configuration](https://spack.readthedocs.io/en/latest/module_file_support.html)
+    * follows the spec for [spack module configuration](https://spack.readthedocs.io/en/latest/module_file_support.html#the-modules-yaml-config-file-and-module-sets) with small exceptions (see [Modules](#modules) for more details).
 * `packages.yaml`: _optional_ define external packages
     * follows the spec for [spack package configuration](https://spack.readthedocs.io/en/latest/build_settings.html)
 * `repo`: _optional_ custom spack package definitions.
@@ -445,7 +445,14 @@ The presence of a `modules.yaml` file in the recipe is a necessary and sufficien
 !!! warning
     `config:modules` field has been deprecated. It can still be specified, but it has to be consistent with the presence of `modules.yaml` file.
 
-Modules are generated for the installed compilers and packages by spack. Rules for module generation in `modules.yaml` file should be provided as per the [spack documentation](https://spack.readthedocs.io/en/latest/module_file_support.html).
+Modules are generated for the installed compilers and packages by spack.
+
+!!! info "Rules for module generation in `modules.yaml`"
+    Stackinator relies on Spack's module generation capabilities, that's the reason why the `modules.yaml` file has to follow [Spack's specification](https://spack.readthedocs.io/en/latest/module_file_support.html#the-modules-yaml-config-file-and-module-sets).
+    But, **there are some differences**:
+
+    - `modules:default:arch_folder` defaults to `false`. If set to `true` an error is raised, as Stackinator does not support this feature;
+    - `modules:default:roots:tcl` is ignored, as Stackinator automatically configures the module root to be inside the uenv mount point.
 
 ## Custom Spack Packages
 

stackinator/recipe.py

@@ -69,8 +69,14 @@ def __init__(self, args):
         if modules_path.is_file():
             with modules_path.open() as fid:
                 self.modules = yaml.load(fid, Loader=yaml.Loader)
-                # Note: it should match MODULEPATH set by envvars and used by uenv view "modules"
-                self.modules["modules"]["default"]["roots"]["tcl"] = (pathlib.Path(self.mount) / "modules").as_posix()
+                schema.ModulesValidator.validate(self.modules)
+
+                # Note:
+                # modules root should match MODULEPATH set by envvars and used by uenv view "modules"
+                # so we enforce that the user does not override it in modules.yaml
+                self.modules["modules"].setdefault("default", {}).setdefault("roots", {}).setdefault(
+                    "tcl", (self.mount / "modules").as_posix()
+                )
 
         # DEPRECATED field `config:modules`
         if "modules" in self.config:

stackinator/schema.py

@@ -108,7 +108,16 @@ def check_config_version(instance):
             raise RuntimeError("incompatible uenv recipe version")
 
 
+def check_module_paths(instance):
+    try:
+        instance["modules"]["default"]["roots"]["tcl"]
+        root_logger.warning("'modules:default:roots:tcl' field is ignored and overwritten by stackinator.")
+    except KeyError:
+        pass
+
+
 ConfigValidator = SchemaValidator(prefix / "schema/config.json", check_config_version)
 CompilersValidator = SchemaValidator(prefix / "schema/compilers.json")
 EnvironmentsValidator = SchemaValidator(prefix / "schema/environments.json")
 CacheValidator = SchemaValidator(prefix / "schema/cache.json")
+ModulesValidator = SchemaValidator(prefix / "schema/modules.json", check_module_paths)

stackinator/schema/modules.json

@@ -0,0 +1,34 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Schema for Stackinator constraints on Spack Modules modules.yaml",
+  "type" : "object",
+  "required": ["modules"],
+  "properties" : {
+    "modules" : {
+      "type": "object",
+      "properties": {
+        "default": {
+          "type": "object",
+          "default": {"arch_folder": false},
+          "properties": {
+            "roots": {
+              "type": "object",
+              "additionalProperties": true,
+              "properties": {
+                "tcl": {
+                  "type": "string"
+                }
+              }
+            },
+            "arch_folder": {
+              "type": "boolean",
+              "const": false,
+              "default": false
+            }
+          }
+        }
+      }
+    }
+  }
+}
+

unittests/test_schema.py

@@ -199,3 +199,76 @@ def test_recipe_environments_yaml(recipe_paths):
         with open(p / "environments.yaml") as fid:
             raw = yaml.load(fid, Loader=yaml.Loader)
             schema.EnvironmentsValidator.validate(raw)
+
+
+@pytest.mark.parametrize(
+    "recipe",
+    [
+        dedent(
+            """
+            modules: {}
+            """
+        ),
+        dedent(
+            """
+            modules:
+              default:
+                arch_folder: false
+            """
+        ),
+        dedent(
+            """
+            modules:
+              # Paths tomodules: check when creating modules for all module sets
+              prefix_insmodules:pections:
+                bin:
+                  - PATH
+                lib:
+                  - LD_LIBRARY_PATH
+                lib64:
+                  - LD_LIBRARY_PATH
+
+              default:
+                arch_folder: false
+                # Where to install modules
+                tcl:
+                  all:
+                    autoload: none
+                  hash_length: 0
+                  exclude_implicits: true
+                  exclude: []
+                  projections:
+                    all: '{name}/{version}'
+            """
+        ),
+        dedent(
+            """
+            modules:
+              default:
+                roots:
+                  tcl: /path/which/is/going/to/be/ignored
+            """
+        ),
+    ],
+)
+def test_valid_modules_yaml(recipe):
+    instance = yaml.load(recipe, Loader=yaml.Loader)
+    schema.ModulesValidator.validate(instance)
+    assert not instance["modules"]["default"]["arch_folder"]
+
+
+@pytest.mark.parametrize(
+    "recipe",
+    [
+        dedent(
+            """
+            modules:
+              default:
+                arch_folder: true
+            """
+        ),
+    ],
+)
+def test_invalid_modules_yaml(recipe):
+    with pytest.raises(Exception):
+        schema.ModulesValidator.validate(yaml.load(recipe, Loader=yaml.Loader))