Update docs to reflect recent mpi syntax changes

Author: biddisco

docs/recipes.md

@@ -146,10 +146,10 @@ For example, in the recipe below, only `netcdf-fortran` will be built with the `
 
 Stackinator can configure cray-mpich (CUDA, ROCM, or non-GPU aware) or OpenMPI (with or without CUDA) (on a per-environment basis, by setting the `mpi` field in an environment.
 
-!!! note
+!!! note Note (on non default MPI veresions)
     Future versions of Stackinator will fully support OpenMPI, MPICH and MVAPICH when (and if) they develop robust support for HPE SlingShot 11 interconnect.
 
-    Current OpenMPI support has been tested lightly and is not guaranteed to be production ready - only [email protected] is supported (default is @5.0.6 at the time of writing) - CUDA is supported, ROCM has not yet been tested.
+    Current OpenMPI support has been tested lightly and is not guaranteed to be production ready - only [email protected] is supported (default is @5.0.6 at the time of writing - 2025.03.04) - CUDA is supported, ROCM has not yet been tested.
 
 If the `mpi` field is not set, or is set to `null`, MPI will not be configured in an environment:
 ```yaml title="environments.yaml: no MPI"
@@ -191,16 +191,57 @@ ompi-cuda-env:
   # ...
 ```
 #### Experimental libfabric 2.x support with cray-mpich
-HPE recently open-sourced the libfabric/cxi provider (and related drivers) and this can be built into cray-mpich by adding the `+cxi` variant to the spec
-```yaml title="environments.yaml: MPI using new libfabric/cxi stack"
+HPE has open-sourced the libfabric/cxi provider (and related drivers) and these can be built into cray-mpich by adding a dependency to a newer version of libfabric.
+The system default is [email protected] - which can be changed by adding a `depends` field to the yaml. A non default version (newer than 1.15.2) will trigger a build of libfabric using libcxi, cxi-driver, cassini-headers).
+
+This syntax appplies to both mpich and openmpi builds.
+```yaml title="environments1.yaml: cray-mpich using new libfabric/cxi stack"
 mpich-cxi-env:
   mpi:
-    spec: cray-mpich +cxi
+    spec: cray-mpich
     gpu: cuda
-  # ...
+    depends: [libfabric@main]
+    # on release of libfabric@2, we recommended using @2 in preference to @main ...
+```
+```yaml title="environments2.yaml: openmpi using new libfabric/cxi stack"
+openmpi-cxi-env:
+  mpi:
+    spec: openmpi
+    gpu: cuda
+    depends: [libfabric@main]
+    # on release of libfabric@2, we recommended using @2 in preference to @main ...
 ```
-OpenMPI does not provide a `cxi` option since it is mandatory to use it for builds on the alps cluster. Currently the performance of OpenMPI on Alps clusters might not be optimal and work is ongoing to fine tune it especially for intra-node performance.
+!!! note Note (on openmpi performance)
+    Currently the performance of OpenMPI on Alps clusters might not be optimal and work is ongoing to fine tune it especially for intra-node performance.
+
+#### Custom MPI builds
+If an experimental version of OpenMPI is required, the yaml syntax supports additional options to enable this. the `xspec` tag may be used to supply extra spack `spec` options.
+To illustrate usage, consider this example: a build of openmpi using a particular git branch named `mpi-continue-5.0.6` which in this case has a variant `+continuations` that is not available on the `main` or released branches.
+The `xspec` tag allows the user to supply arbitrary spack variants and options that replace the defaults used by stackinator in its absence.
+```yaml title="custom-openmpi.yaml: custom build of openmpi"
+openmpi-custom-env:
+  mpi:
+    spec: [email protected]=main
+    xspec: +continuations +internal-pmix fabrics=cma,ofi,xpmem schedulers=slurm +cray-xpmem
+    gpu: cuda
+    depends: ["libfabric@main"]
+```
+In this example, we must tell spack to fetch our custom git branch from a repo that is different from the default openmpi github version, by adding the following to our recipe `packages.yaml`
+```yaml title="custom-packages.yaml: custom repo for openmpi"
+  # mpi-continue-5.0.6 branch available from here
+  openmpi:
+    package_attributes:
+      git: https://github.com/your-username/ompi
+```
+
+!!! note Note (git tags)
+    To build using a specific git commit, use the syntax
+    ```
+    spec: [email protected]=main
+    ```
+It is therefore possible to build arbitrary versions of MPI using custom options/branches etc using these combinations of settings.
 
+### Version info
 !!! alps
 
     As new versions of cray-mpich are released with CPE, they are provided on Alps vClusters, via the Spack package repo in the [CSCS cluster configuration repo](https://github.com/eth-cscs/alps-cluster-config/tree/master/site/repo).