Multi-HPC Workflows (#218)

* Move jobs around so they can be part of a single parallel workflow (#203)

* settings-1-update.yml: Validate settings in parallel based on target

* deploy-2-start.yml: Output modules and spack location from workflow

* deploy-2-start.yml: Capitalize inputs.type values to keep in line with callers

* deploy-1-setup.yml: Update inputs/outputs to be suitable for a matrix job.
This matrix job must have the required I/O to do pre-deploy checks, as well as a deployment.

* Move check-spack-yaml and check-config jobs from ci.yml to deploy-1-setup.yml

* deploy-1-setup.yml: Use new inputs as inputs to deploy-2-start.yml

* deploy-1-start.yml: Upload 'outputs' of workflow as an artifact.
This is due to dynamic matrix job outputs not being collected appropriately.
See https://github.com/orgs/community/discussions/17245

* cd.yml: Run checks of target config in parallel

* ci.yml: Dynamically generate deployment comment, start matrix of deployment jobs, misc changes

* cd.yml: Update deployment matrix jq

* ci.yml: Add newline to deployment comment to fix formatting

* Update job names to include deploy target

* deploy-2-start.yml: Remove `release` and `build-db` jobs, output new general metadata artifact glob

* deploy-1-setup.yml: Output `general-metadata-artifact-glob` from `deploy-2-start.yml` invocation

* cd.yml: Add `release`, `build-db` jobs from `deploy-2-start.yml`

* deploy-2-start.yml: Prepend metadata artifacts with deployment environment

* cd.yml: Generate release notes based on deployment targets

* .gitignore venv, pytest metadata

* Create j2 templates for release notes and deployment comments

Co-authoured-by: Tom McAdam <[email protected]>

* Add scripts for jinja generation, requirements, created modules

* Update ci/cd to use templating script

* Add testing infrastructure

* Updated ci/cd calls to use script notation, removed --type arg

* ci.yml: Use output file as comment due to ACCESS-NRI/actions#20

* Add test validating non-collection of variables with the wrong `template_prefix`

* Remove outputs folder as it is unused

* Update requirements.txt filepath

* Added backslashes to multiline script invocation

* Update template

* Replace `env.SPACK_YAML_MODEL_YQ` to accomodate for multi-target-format `spack.yaml`

* Uppercase ROOT_PACKAGE

* Update target matrix generation to take into account model repos deployment targets

* Added comments

* Added new get-target-matrix action

* Update ci.yml, cd.yml to use get-target-matrix

* Simplify undeploy workflow

* cd.yml: Updated output variable

* Updated workflow reference to `undeploy-1-start.yml`

* get-target-matrix: Made vars more generic

* generate-build-metadata.bash: Reference explicit Gadi.* files

* tests/scripts/generate-build-metadata: Updated test data inputs/outputs

* cd.yml: Use Gadi-specific outputs/metadata for build-db upload

* Update README.md with information on this repository and it's workflows

* Add config/README.md to explain files within the folder

* Apply suggestions from code review

Co-authored-by: jo-basevi <[email protected]>
Signed-off-by: Tommy Gatti <[email protected]>

* Add GitHub search link

* Small scale fixes from ACCESS-TEST E2E Testing

* Split `deployment-environment` input into `deployment-target` and `deployment-type`

* Update all build-cd refs to latest `v4`, get latest default branch ref for non-action calls

* Use default `inputs.ref` for `checkout` action to get default branch

* Remove conditional logic for `deployment-type`s
This is because all environment are now of the form SUPERCOMPUTER TYPE

---------

Signed-off-by: Tommy Gatti <[email protected]>
Co-authored-by: jo-basevi <[email protected]>

Author: CodeGat

.github/actions/get-deploy-paths/README.md

@@ -30,7 +30,7 @@ jobs:
     steps:
       - name: Get Deployment Paths
         id: paths
-        uses: access-nri/build-cd/.github/actions/get-deploy-paths@main
+        uses: access-nri/build-cd/.github/actions/get-deploy-paths@vX
         with:
           spack-installs-root-path: ${{ vars.SPACK_INSTALLS_ROOT_PATH }}
           spack-version: "0.21"

.github/actions/get-target-matrix/README.md

@@ -0,0 +1,40 @@
+# Get Deployment Target Matrix Action
+
+Action that returns a matrix of deployment targets based on the caller's input and the valid targets in the build-cd repository.
+
+## Inputs
+
+| Name | Type | Description | Required | Default | Example |
+| ---- | ---- | ----------- | -------- | ------- | ------- |
+| `targets` | `string` (space-separated) | A space-separated list of deployment targets for the model deployment repository. | `true` | N/A | `"Gadi Setonix"` |
+
+## Outputs
+
+| Name | Type | Description | Example |
+| ---- | ---- | ----------- | ------- |
+| `valid-targets` | `string` | A JSON array of valid deployment targets, suitable for use as a matrix. | `"["Gadi", "Setonix"]"` |
+
+## Example
+
+```yaml
+# ...
+jobs:
+  generate-matrix:
+    runs-on: ubuntu-latest
+    outputs:
+      valid-targets: ${{ steps.generate.outputs.valid-targets }}
+    steps:
+      - id: generate
+        uses: access-nri/build-cd/.github/actions/get-target-matrix@vX
+        with:
+          targets: ${{ vars.MODEL_REPO_TARGETS }}
+
+  matrix:
+    runs-on: ubuntu-latest
+    needs: [generate-matrix]
+    strategy:
+      matrix:
+        target: ${{ fromJson(needs.generate-matrix.outputs.valid-targets) }}
+    steps:
+      - run: echo "Wow, this is running in ${{ matrix.target }}!"
+```

.github/actions/get-target-matrix/action.yml

@@ -0,0 +1,43 @@
+name: Get Deployment Target Matrix
+description: Action that returns a matrix of deployment targets based on the caller's input and the valid targets in the build-cd repository.
+author: Tommy Gatti
+inputs:
+  targets:
+    type: string
+    required: true
+    description: A space-separated list of deployment targets for the model deployment repository.
+outputs:
+  valid-targets:
+    description: A JSON array of valid deployment targets, suitable for use as a matrix.
+    value: ${{ steps.generate.outputs.targets }}
+runs:
+  using: composite
+  steps:
+    - name: Get deployment settings.json
+      uses: actions/checkout@v4
+      with:
+        repository: access-nri/build-cd
+
+    - name: Generate Deployment Target Matrix
+      shell: bash
+      id: generate
+      # These are used in the deploy job to determine the targets to deploy to - irrespective of the type of deployment.
+      # We use the list of valid targets (from build-cd) intersected with targets set in the caller repository to determine the final list.
+      run: |
+        valid_targets=$(jq --raw-output --compact-output \
+          '.deployment | keys' \
+          config/settings.json
+        )
+        repo_targets=$(jq --null-input --raw-output --compact-output \
+          --arg valids "${{ inputs.targets }}" \
+          '$valids | split(" ")'
+        )
+        # This is the intersection between the valid and caller-defined targets
+        targets=$(jq --null-input --raw-output --compact-output\
+          --argjson valid "$valid_targets" \
+          --argjson ours "$repo_targets" \
+          '$valid - ($valid - $ours)'
+        )
+
+        echo "From build-cd's valid targets: $valid_targets, and ${{ github.repository }}s chosen targets: $repo_targets, we will deploy to $targets"
+        echo "targets=$targets" >> $GITHUB_OUTPUT

.github/actions/validate-deployment-settings/README.md

@@ -22,7 +22,7 @@ This action validates various `ACCESS-NRI/build-cd` deployment settings.
 # ...
 - name: Validate settings
   id: settings
-  uses: access-nri/build-cd/.github/actions/validate-deployment-settings@main
+  uses: access-nri/build-cd/.github/actions/validate-deployment-settings@vX
   with:
     settings-path: ./some/settings.json
     target: Supercomputer

.github/actions/validate-repo-version/README.md

@@ -20,7 +20,7 @@ This action checks that the tags specified in a models `config/versions.json` is
 ```yaml
 # ...
 - id: validate
-  uses: access-nri/build-cd/.github/actions/validate-repo-version@main
+  uses: access-nri/build-cd/.github/actions/validate-repo-version@vX
   with:
     repo-to-check: spack-packages
     pr: 12

.github/workflows/cd.yml

@@ -27,14 +27,19 @@ on:
   #     - spack.yaml
 
 env:
-  SPACK_YAML_MODEL_YQ: .spack.specs[0]
+  # This attempts to get the spack.yaml model first via the multi-target
+  # .spack.definitions.root_package method, then the traditional '.spack.specs[0]'.
+  SPACK_YAML_MODEL_YQ: (.spack.definitions[] | select(."ROOT_PACKAGE") | .[][]) // .spack.specs[0]
+  METADATA_PATH: /opt/metadata
+  OUTPUTS_PATH: /opt/outputs
 jobs:
   defaults:
     name: Set Defaults
     # Unfortunately, you can't set a dynamic default value based on `inputs` yet
     runs-on: ubuntu-latest
     outputs:
       root-sbd: ${{ steps.root-sbd.outputs.default }}
+      targets: ${{ steps.target.outputs.valid-targets }}
     steps:
       - name: root-sbd
         id: root-sbd
@@ -45,20 +50,29 @@ jobs:
             echo "default=${{ inputs.root-sbd }}" >> $GITHUB_OUTPUT
           fi
 
+      - name: Generate Deployment Target Matrix
+        id: target
+        uses: access-nri/build-cd/.github/actions/get-target-matrix@v4
+        with:
+          targets: ${{ vars.RELEASE_DEPLOYMENT_TARGETS }}
+
   verify-settings:
     name: Verify Deployment Settings
     runs-on: ubuntu-latest
+    needs:
+      - defaults
+    strategy:
+      matrix:
+        target: ${{ fromJson(needs.defaults.outputs.targets) }}
     steps:
       - uses: actions/checkout@v4
         with:
-          repository: ACCESS-NRI/build-cd
-          ref: main
+          repository: access-nri/build-cd
 
-      - uses: access-nri/build-cd/.github/actions/validate-deployment-settings@main
+      - uses: access-nri/build-cd/.github/actions/validate-deployment-settings@v4
         with:
           settings-path: ./config/settings.json
-          # TODO: Turn this into a matrix job of targets
-          target: Gadi
+          target: ${{ matrix.target }}
           error-level: error
 
   push-tag:
@@ -103,11 +117,152 @@ jobs:
     needs:
       - defaults
       - push-tag
-    uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@main
+    strategy:
+      matrix:
+        target: ${{ fromJson(needs.defaults.outputs.targets) }}
+    uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@v4
     with:
-      ref: ${{ github.ref_name }}
-      version: ${{ needs.push-tag.outputs.name }}
-      root-sbd: ${{ needs.defaults.outputs.root-sbd }}
+      deployment-target: ${{ matrix.target }}
+      deployment-ref: ${{ github.ref_name }}
+      deployment-type: Release
+      deployment-version: ${{ needs.push-tag.outputs.name }}
+      spack-manifest-path: ./spack.yaml
+      spack-manifest-root-sbd: ${{ needs.defaults.outputs.root-sbd }}
     secrets: inherit
     permissions:
       contents: write
+      # We require pull-requests:write despite not using it in this path,
+      # because it is needed by the pull_request invocation of the workflow.
+      pull-requests: write
+
+  release:
+    name: Create Release
+    needs:
+      - defaults
+      - push-tag
+      - deploy-release
+    runs-on: ubuntu-latest
+    env:
+      TEMPLATED_RELEASE_BODY_PATH: /opt/release-body.md
+    outputs:
+      url: ${{ steps.release.outputs.url }}
+      created-at: ${{ steps.metadata.outputs.created-at }}
+    steps:
+      - name: Checkout build-cd
+        uses: actions/checkout@v4
+        with:
+          repository: access-nri/build-cd
+
+      - name: Download Metadata Artifacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: ${{ needs.deploy-release.outputs.general-metadata-artifact-glob }}
+          path: ${{ env.METADATA_PATH }}
+          merge-multiple: true
+
+      - name: Download Outputs Artifacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: ${{ needs.deploy-release.outputs.general-outputs-artifact-glob }}
+          path: ${{ env.OUTPUTS_PATH }}
+          merge-multiple: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: pip
+
+      - name: Install Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -q -q -r scripts/jinja_template/requirements.txt
+
+      - name: Generate Release Notes
+        id: release-body
+        env:
+          J2_MODEL: ${{ inputs.model }}
+          J2_VERSION: ${{ needs.push-tag.outputs.name }}
+          J2_ROOT_SBD: ${{ needs.defaults.outputs.root-sbd }}
+        run: |
+          python -m scripts.jinja_template.render_deployment_info \
+            --template scripts/jinja_template/templates/release-body.md.j2 \
+            --deployment-outputs ${{ env.OUTPUTS_PATH }} \
+            --output ${{ env.TEMPLATED_RELEASE_BODY_PATH }}
+
+      - name: Create Release
+        id: release
+        uses: softprops/action-gh-release@69320dbe05506a9a39fc8ae11030b214ec2d1f87  # v2.0.5
+        with:
+          tag_name: ${{ needs.push-tag.outputs.name }}
+          name: ${{ inputs.model}} ${{ needs.push-tag.outputs.name }}
+          body_path: ${{ env.TEMPLATED_RELEASE_BODY_PATH }}
+          generate_release_notes: true
+          fail_on_unmatched_files: true
+          files: |
+            ${{ env.METADATA_PATH }}/*.spack.yaml
+            ${{ env.METADATA_PATH }}/*.spack.lock
+            ${{ env.METADATA_PATH }}/*.spack.location
+            ${{ env.METADATA_PATH }}/*.spack.location.json
+
+      - name: Release Metadata
+        id: metadata
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: echo "created-at=$(gh release view --json createdAt --jq '.createdAt' --repo ${{ github.repository }})" >> $GITHUB_OUTPUT
+
+  # TODO: This will only upload data related to the Gadi deployment
+  build-db:
+    name: Build DB Metadata Upload
+    needs:
+      - defaults
+      - deploy-release
+      - release
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download Metadata Artifact
+        uses: actions/download-artifact@v4
+        with:
+          pattern: ${{ needs.deploy-release.outputs.general-metadata-artifact-glob }}
+          path: ${{ env.METADATA_PATH }}
+          merge-multiple: true
+
+      - name: Download Outputs Artifact
+        uses: actions/download-artifact@v4
+        with:
+          pattern: ${{ needs.deploy-release.outputs.general-outputs-artifact-glob }}
+          path: ${{ env.OUTPUTS_PATH }}
+          merge-multiple: true
+
+      - name: 'Get Gadi spack-* Repo Versions'
+        id: spack-versions
+        run: |
+          echo "packages-version=$(jq --raw-output --compact-output '.spack_packages_version' ${{ env.OUTPUTS_PATH }}/deploy-outputs.Gadi)" >> $GITHUB_OUTPUT
+          echo "config-version=$(jq --raw-output --compact-output '.spack_config_version' ${{ env.OUTPUTS_PATH }}/deploy-outputs.Gadi)" >> $GITHUB_OUTPUT
+
+      - name: Checkout Upload Script
+        uses: actions/checkout@v4
+        with:
+          repository: access-nri/build-cd
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ vars.PYTHON_VERSION }}
+          cache: pip
+
+      - name: Install Build Metadata Script Requirements
+        run: pip install -r tools/release_provenance/requirements.txt
+
+      - name: Upload Build Metadata
+        env:
+          BUILD_DB_CONNECTION_STR: ${{ secrets.BUILD_DB_CONNECTION_STR }}
+        run: |
+          ./scripts/generate-build-metadata.bash \
+            ${{ needs.release.outputs.url }} ${{ needs.release.outputs.created-at }} \
+            ${{ steps.spack-versions.outputs.packages-version }} ${{ steps.spack-versions.outputs.config-version }} \
+            ${{ env.METADATA_PATH }} ${{ env.OUTPUTS_PATH }} \
+            ${{ needs.defaults.outputs.root-sbd }} ${{ vars.BUILD_DB_PACKAGES }}
+
+          echo "Attempting upload of build_metadata.json"
+          python ./tools/release_provenance/save_release.py "${{ env.OUTPUTS_PATH }}/build_metadata.json"

.github/workflows/ci-closed.yml

@@ -9,10 +9,10 @@ run-name: ${{ inputs.model }} PR Closed Cleanup
 on:
   workflow_call:
     inputs:
-      model:
+      root-sbd:
         type: string
         required: true
-        description: The model that is being tested and deployed
+        description: The model package name that is going to be removed
   # Callers usually have the trigger:
   # pull_request:
   #   types:
@@ -29,19 +29,30 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       version-pattern: ${{ steps.version.outputs.pattern }}
+      targets: ${{ steps.target.outputs.valid-targets }}
     steps:
-      - name: Version Pattern
+      - name: Get Version Pattern
         id: version
         # For example, `access-om3-pr12-*`
-        run: |
-          repo_name_sanitized=$(echo ${{ github.event.repository.name }} | tr [:upper:] [:lower:] | tr '.' 'p' )
-          echo "pattern=${repo_name_sanitized}-pr${{ github.event.pull_request.number }}-*" >> $GITHUB_OUTPUT
+        run: echo "pattern=${{ inputs.root-sbd }}-pr${{ github.event.pull_request.number }}-*" >> $GITHUB_OUTPUT
+
+      - name: Generate Deployment Target Matrix
+        id: target
+        uses: access-nri/build-cd/.github/actions/get-target-matrix@v4
+        with:
+          targets: ${{ vars.PRERELEASE_DEPLOYMENT_TARGETS }}
 
   undeploy-prereleases:
     name: Undeploy Prereleases Matching ${{ needs.setup.outputs.version-pattern }}
     needs:
       - setup
-    uses: access-nri/build-cd/.github/workflows/undeploy-1-setup.yml@main
+    strategy:
+      matrix:
+        target: ${{ fromJson(needs.setup.outputs.targets) }}
+      fail-fast: false
+    uses: access-nri/build-cd/.github/workflows/undeploy-1-start.yml@v4
     with:
-      version-pattern: ${{ needs.setup.outputs.version-pattern }}
+      version-pattern: ${{ inputs.root-sbd }}-pr${{ github.event.pull_request.number }}-*
+      target: ${{ matrix.target }}
+      type: Prerelease
     secrets: inherit

.github/workflows/ci-comment.yml

@@ -13,7 +13,7 @@ on:
       root-sbd:
         type: string
         required: false
-        # The equivalent default is set in the defaults job below. 
+        # The equivalent default is set in the defaults job below.
         # default: ${{ inputs.model }}
         description: |
           The name of the root Spack Bundle Definition, if it is different from the model name.
@@ -53,7 +53,9 @@ jobs:
       contents: write
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      SPACK_YAML_MODEL_YQ: .spack.specs[0]
+      # This attempts to get the spack.yaml model first via the multi-target
+      # .spack.definitions.root_package method, then the traditional '.spack.specs[0]'.
+      SPACK_YAML_MODEL_YQ: (.spack.definitions[] | select(."ROOT_PACKAGE") | .[][]) // .spack.specs[0]
       SPACK_YAML_MODEL_PROJECTION_YQ: .spack.modules.default.tcl.projections.${{ needs.defaults.outputs.root-sbd }}
     steps:
       - uses: actions/checkout@v4