Fixes and suggestions for ReFrame documentation (#151)

Author: RMeli

Diff for: docs/pkg-reframe.md

@@ -1,12 +1,12 @@
 # ReFrame Testing Tutorial
 
-When ReFrame tests are enabled for a uenv, they are automatically run:
+When [ReFrame] tests are enabled for a uenv, they are automatically run:
 
 * in the CI/CD pipeline after the image has been built;
 * in daily/weekly testing of individual vClusters;
 * and when upgrading and updating vClusters.
 
-This page is a tutorial, that will guide you through the process of enabling testing in your uenv, and on writing portable tests that will run on any uenv-enabled system on Alps.
+This page is a tutorial that will guide you through the process of enabling testing for your uenv, and on writing portable tests that will run on any uenv-enabled system on [Alps].
 
 !!! info
 
@@ -18,7 +18,7 @@ This page is a tutorial, that will guide you through the process of enabling tes
 
 ## How uenv ReFrame testing works
 
-CSCS maintains a set of ReFrame tests in the CSCS ReFrame tests repository [eth-cscs/cscs-reframe-tests].
+CSCS maintains a set of [ReFrame] tests in the CSCS ReFrame tests repository [eth-cscs/cscs-reframe-tests].
 These tests cover a very wide range of features, including application tests, login node health and Slurm, and can be run on any vCluster on Alps.
 
 !!! info
@@ -27,15 +27,15 @@ These tests cover a very wide range of features, including application tests, lo
 
 Setting up tests for a uenv requires making changes to two repositories:
 
-* [eth-cscs/alps-uenv] **adding meta data to the uenv** to be used by ReFrame to:
+* [eth-cscs/alps-uenv] **adding metadata to the uenv** to be used by ReFrame to:
     * load the uenv and configure the environment so that it is ready to run tests;
     * and, choose which tests from the test suite are used to test the uenv.
 * [eth-cscs/cscs-reframe-tests] **updating and adding tests** in the that are relevant to the uenv.
     * might not be necessary if the tests already exist.
 
 ### uenv recipe meta data
 
-To enable ReFrame tests in your uenv, and yaml file `extra/reframe.yaml` should be added to the recipe.
+To enable ReFrame tests in your uenv, a yaml file `extra/reframe.yaml` should be added to the recipe.
 
 Below is an example `reframe.yaml` file:
 
@@ -57,21 +57,21 @@ This configuration defines a single _environment_ named `develop`, which corresp
 * `features`: a list of ReFrame features that are provided by the environment.
     * used to decide which tests will be run against the uenv.
     * in this case the uenv provides:
-        * `cuda`: expect tests that compile and test NVIDIA gpu aware problems.
+        * `cuda`: expect tests that compile and test NVIDIA GPU aware problems.
         * `mpi`: expect basic MPI tests that compile and validate MPI to be run. When combined with `cuda` above, tests for GPU-aware MPI will be run.
         * `arbor-dev`: a specific feature that specifies that _the environment provides everything required to build and run arbor_.
 * `cc`, `cxx`, `ftn`: define the compiler aliases
     * see the [ReFrame environment]s documentation.
 * `views`: **(optional)** a list of views to load.
     * in this case `develop` view is to be loaded.
 
-Uenv can provide multiple views, for different use cases.
+A uenv can provide multiple views, for different use cases.
 The most common example is a uenv that provides two views: one that provides an application, and another that provides the tools used to build the application. Another example is the `modules` and `spack` views, that expose a module interface or useful configuration for using Spack with the uenv.
 
 Similarly, it is possible to create multiple environments to test.
 The example below defines two environments that provide the same features, i.e. the same tests will be run on each.
 The first example is the one above, and the second sets up an equivalent environment using modules.
-This would be useful for a uenv that has some users who insist on using modules to set up their build enviroment.
+This would be useful for a uenv that has some users who insist on using modules to set up their build environment.
 
 ```yaml title="extra/reframe.yaml for multiple environments to test"
 develop:
@@ -105,25 +105,25 @@ modules:
 
 !!! question "What is an environment?"
 
-    What is the difference between using `module load`, activating a python venv, loading a spack environment, or a uenv view?
+    What is the difference between using `module load`, activating a python venv, loading a Spack environment, or a uenv view?
 
     Nothing!
 
     They all do the same thing - set environment variables.
 
     The main variables that change the behavior of the system are `PATH` and `LD_LIBRARY_PATH`, though there are many others like `PKG_CONFIG_PATH`, `CUDA_HOME`, `MODULEPATH` etc that will have more subtle effects on configuring and building software.
 
-    Configur an environment for running tests requires specifying the commands that will **modify and set environment variables** such that the tests can run. For example, a view or module might be loaded to make the executable of a scientific code be in `PATH`, or to add tools like `cmake`, `nvcc` and `gcc` to `PATH` so that we can run a test that builds an application.
+    Configuring an environment for running tests requires specifying the commands that will **modify and set environment variables** such that the tests can run. For example, a view or module might be loaded to make the executable of a scientific code be in `PATH`, or to add tools like `cmake`, `nvcc` and `gcc` to `PATH` so that we can run a test that builds an application.
 
 ## Creating uenv tests
 
 The final objective for adding tests to a uenv is to have:
 
-1. a uenv deployed with an `meta/extra/reframe.yaml` file;
+1. a uenv deployed with an `extra/reframe.yaml` file;
 2. and tests in the [eth-cscs/cscs-reframe-tests] repository
 
 In this second half of the tutorial, a workflow for doing this that minimises the amount of time spent
-waiting in job and ci/cd queues is provided.
+waiting in job and CI/CD queues is provided.
 Before starting, you will need the following:
 
 * a working uenv squashfs image with corresponding meta data path;
@@ -133,26 +133,26 @@ Before starting, you will need the following:
 
 ```bash
 # pull the image that you want to start developing tests for, e.g.:
-$ uenv image pull cp2k/24.7:v1
+$ uenv image pull cp2k/2024.2:v1
 
 # get the meta data path
 $ meta=$(uenv image inspect cp2k/2024.2:v1 --format={meta})
 
 # check the path - your location will be different
 $ echo ${meta}
 
-# create the reframe meta data eile
+# create the reframe meta data file
 $ mkdir -p ${meta}/extra
 $ vim ${meta}/extra/reframe.yaml
 ```
 
-The `meta` path is the meta data for the uenv for the image.
+The `meta` path is the metadata for the uenv for the image.
 
 ??? note "why create reframe.yaml in this location?"
 
     We inject the `reframe.yaml` file into the meta data in the uenv repo to create a "development environment", where it can be modified while developing the tests in an interactive shell.
 
-    The `reframe.yaml` file will be added to the recipe later, once it is time to start testing in a ci/cd pipeline.
+    The `reframe.yaml` file will be added to the recipe later, once it is time to start testing in a CI/CD pipeline.
 
 
 ### Step 2: set up the CSCS reframe tests
@@ -161,11 +161,11 @@ The next step is to check out and setup ReFrame and the CSCS ReFrame test suite.
 
 It might be a good idea to create a path for this work, and cloning the ReFrame and CSCS test suite repos as sub-directories.
 
-### set up ReFrame
+### Set up ReFrame
 
 The first step is to download and set up ReFrame:
 
-* clone from GitHub;
+* clone from [ReFrame GitHub repository](https://github.com/reframe-hpc/reframe)
 * run the bootstrap process that installs ReFrame's dependencies;
 * then add reframe to `PATH`.
 
@@ -174,14 +174,15 @@ The first step is to download and set up ReFrame:
 $ git clone [email protected]:reframe-hpc/reframe.git
 
 # run bootstrap process (only needs to be done once)
-$ (cd reframe; ./bootstrap.sh)
+$ cd reframe
+$ ./bootstrap.sh
 
 # add to PATH and verify that everything works
-$ export PATH=$PWD/reframe/bin:$PATH
+$ export PATH=$PWD/bin:$PATH
 $ reframe --version
 ```
 
-### set up the ReFrame tests
+### Set up the ReFrame tests
 
 The next step is to clone the CSCS test suite, and create a new branch where we will make any changes required to test our uenv.
 
@@ -199,7 +200,7 @@ git switch -c uenv-arbor
     Always create your working branch off of the `alps` branch.
     The `alps` branch is used for tests run on Alps vClusters. It will become the main branch, once Piz Daint is decommisioned.
 
-## Adding/Updating tests to a uenv
+## Adding/updating tests to a uenv
 
 Now everything is in place to implement the tests for your uenv, which will involve one or two of the following:
 
@@ -225,7 +226,7 @@ In this tutorial we will write tests that:
 3. run a benchmark with a single MPI rank on one GH200 GPU
 3. run a benchmark with 4 ranks and 4 GPUs on a GH200 node.
 
-[link to the tests](https://github.com/eth-cscs/cscs-reframe-tests/tree/alps/checks/apps/arbor).
+[Link to the tests](https://github.com/eth-cscs/cscs-reframe-tests/tree/alps/checks/apps/arbor).
 
 !!! note
 
@@ -243,18 +244,16 @@ import uenv
 ```
 
 It provides the `uenv.uarch()` function, that will be used to determine the uenv
-uarch (`gh200`, `a100`, `zen2`, etc) of the system where tests are running.
+uarch (`gh200`, `a100`, `zen2`, etc.) of the system where tests are running.
 We will see it in action below.
 
-### test: building the software
+### Test: building the software
 
-[Link](https://github.com/eth-cscs/cscs-reframe-tests/blob/alps/checks/apps/arbor/arbor-dev.py#L47-L93).
-
-Building is handled by a test, in this case called `arbor_build`, that derives from `rfm.CompileOnlyRegressionTest`.
+Building is handled by a test, in this case called [`arbor_build`](https://github.com/eth-cscs/cscs-reframe-tests/blob/alps/checks/apps/arbor/arbor-dev.py#L47-L93), that derives from `rfm.CompileOnlyRegressionTest`.
 
 !!! info
 
-    Points of interest are annotated in the code below with :heavy_plus_sign: symbols, click on them to expand.
+    Points of interest are annotated in the code below with :material-plus-circle: symbols, click on them to expand.
 
 ``` { .python .annotate }
 class arbor_build(rfm.CompileOnlyRegressionTest):
@@ -305,7 +304,7 @@ class arbor_build(rfm.CompileOnlyRegressionTest):
 1.  Restrict this test to only run in environments that provide the `arbor-dev` feature.
     This can be a list of environments, e.g. `['+arbor-dev+cuda', '+python']` would run the test in environments that provide both `arbor-dev` and `cuda`, or environments that provide `python`.
 2.  `arbor_download` is a [ReFrame fixture](https://reframe-hpc.readthedocs.io/en/v4.5.0/tutorial_fixtures.html), that handles downloading the source for Arbor.
-3.  The build stage is performed on a compute node using an sbatch job.
+3.  The build stage is performed on a compute node using a sbatch job.
     Required so that the environment is configured properly, by adding the
     correct flags to the script:
     ```
@@ -330,7 +329,7 @@ class arbor_build(rfm.CompileOnlyRegressionTest):
 
     The new approach of parameterising over uarch means that the test can be configured for _any_ vCluster with gh200 nodes.
 
-### test: run the unit tests
+### Test: run the unit tests
 
 The C++ Arbor library provides GoogleTest unit tests that are bundled in a single executable `unit`.
 The tests are not MPI enabled, and take less than 30 seconds to run 1000 individual tests.
@@ -360,11 +359,11 @@ class arbor_unit(rfm.RunOnlyRegressionTest):
 1.  This is the first time that we have added an annotation to a test.
     This is a "leaf" in our set of test dependencies, run after the download
     and build stages that are its dependencies have run.
-2.  The unit tests run quickly - so set a short time limit for higher priority queueing
+2.  The unit tests run quickly - so set a short time limit for higher priority queuing
 3.  The `arbor_build` stage has to be run before this test, to build the unit tests.
 4.  Just check that the tests passed - performance checks are implemented elsewhere
 
-### test: single GPU benchmark
+### Test: single GPU benchmark
 
 Use the `miniring` benchmark provided by Arbor to check both correctness and performance.
 
@@ -397,8 +396,8 @@ arbor_references = {
 }
 ```
 
-1.  Currently we only have Arbor performance targets for gh200, fields for `zen2` would be added for Eiger testing.
-2.  These are labelled reference targes. A link will be added when it is found in ReFrame's "documentation".
+1. Currently, we only have Arbor performance targets for `gh200`, fields for `zen2` would be added for Eiger testing.
+2. These are labelled reference targets. A link will be added when it is found in ReFrame's "documentation".
 
 The test itself:
 
@@ -449,7 +448,7 @@ class arbor_busyring(rfm.RunOnlyRegressionTest):
    In other words - the performance target is set dynamically based on the architecture of the node,
    instead of being hard coded using if-else statements in the test itself.
 
-     * `self.uarch` is one of the alps arch: `"gh200"`, `"zen2"`, `"a100"`, ... etc, or `None`
+     * `self.uarch` is one of the alps arch: `"gh200"`, `"zen2"`, `"a100"`, ... etc., or `None`
      * `self.current_partition.fullname` is the `vcluster:partition` string, for example `"daint:normal"` or `"todi:debug"`.
 
 !!! note
@@ -458,7 +457,7 @@ class arbor_busyring(rfm.RunOnlyRegressionTest):
     does not provide values for the current uarch.
     However, in such cases, no comparison is made and the test will pass.
 
-### test: MPI tests
+### Test: MPI tests
 
 ``` { .python .annotate }
 slurm_config = { #(1)
@@ -519,16 +518,23 @@ $ export UENV=arbor/v0.9:v1
     It will be a hard requirement that the meta data path will be in the same path
     as the `store.squashfs` file.
 
+!!! warning
+    If the `UENV` variable is not set proprely, the `reframe.yaml` file can't be found and you will see an error:
+    ```
+    ERROR: failed to load configuration: problem loading the metadata from 'extra/reframe.yaml' 
+    ```
+
 To run the tests use the following commands:
 
 ```bash
 # run the tests
 $ reframe -C cscs-reframe-tests/config/cscs.py \
-  -c cscs-reframe-tests/checks/apps/arbor/ -r --keep-stage-files
+  -c cscs-reframe-tests/checks/apps/arbor/ --keep-stage-files \
+  -r
 
 # perform a dry run of tests
 $ reframe -C cscs-reframe-tests/config/cscs.py \
-  -c cscs-reframe-tests/checks/apps/arbor/ -r --keep-stage-files \
+  -c cscs-reframe-tests/checks/apps/arbor/ --keep-stage-files \
   --dry-run
 ```
 
@@ -540,7 +546,7 @@ $ reframe -C cscs-reframe-tests/config/cscs.py \
 * `--keep-stage-files`: this will keep all of the intermediate scripts and configuration
     * stored in the `stage` sub-directory of the current path.
     * very useful for debugging problems with our tests.
-* `--dry-run`: generate all of the stage files and scripts without running the tests.
+* `--dry-run`: generate all the stage files and scripts without running the tests.
 
 !!! tip
 
@@ -552,6 +558,12 @@ $ reframe -C cscs-reframe-tests/config/cscs.py \
     Look in the `stage` path that is created in the path where you called reframe for all
     of the job scripts, build files, and results.
 
+!!! tip
+    If ReFrame tests fail because of `ReqNodesNotAvail` and you think it is a fluke, try setting
+    `RFM_IGNORE_REQNODENOTAVAIL=y`.
+
 [eth-cscs/alps-uenv]: https://github.com/eth-cscs/alps-uenv
 [eth-cscs/cscs-reframe-tests]: https://github.com/eth-cscs/cscs-reframe-tests
 [ReFrame environment]: https://reframe-hpc.readthedocs.io/en/stable/tutorial.html#environment-features-and-extras
+[ReFrame]: https://reframe-hpc.readthedocs.io/en/stable/
+[Alps]: https://www.cscs.ch/computers/alps