MAINT - CI improvements (security and maintenance) (#2077)

I thought I would do some winter/summer cleaning (depending on your
location) ⛄🌞.

This PR adds several improvements/updates to our CI with a focus on
improving the contributor experience and security
Details below:

## 🔒 Security focused

- Use SHA for third-party actions and our internal action for
development setup
- Replace potentially [dangerous trigger
`workflow_run`](https://woodruffw.github.io/zizmor/audits/#dangerous-triggers)
for `workflow_call` and use as a reusable workflow
- Create and use a dedicated environment for releases: 
- [x] Created a `pst-release` environment in the repo (restricted to
`main` only)
  - [x] Use `pst-release` for our `release-PST` step in `publish.yml`
  - [ ] Add `pst-release` as the default env in PyPI
- Add a `zizmor.yml` workflow to run static analysis on our GH workflows
- Add explicit `persist-credentials: false` to relevant actions (where
we do not need further git operations)

## 👩🏽‍🎤 Contributor experience

- Prevent the `pre-release.yml` workflow from running in repos not under
the `pydata` org (forks)
- Our CI workflow has grown significantly with the various tests and
checks. This PR splits it into:
 -  `CI.yml`: pytest, a11y-tests, profiling, coverage
- `docs.yml`: docs-related checks like building across OSes and Python
and Sphinx versions, check for broken links (new, note that I had to fix
some broken links to get this in 🙈 and there seem to be still some
others to fix)
- Add `tox run -e docs-linkcheck` to check for broken links in our docs
- Add workflow to add a link to the RTD docs preview in PRs.

## 🧰 Maintenance

- Adds Python `3.13` to our testing matrices (`3.12` is left as the
`default` until we are confident all is ok with `3.13`)
- Add an explicit `ubuntu-22.04` target as `ubuntu-latest` will soon be
`24.04` (being rolled out right now) -> I think I might actually have
explicit versions on both and only change to latest (or not) when the
rollout is completed

----

## Questions / notes

- @drammock, we have `"sphinx-theme-builder @
https://github.com/pradyunsg/sphinx-theme-builder/archive/87214d0671c943992c05e3db01dca997e156e8d6.zip",`
in our `project. tool` and `tox.ini`. I do not believe this pin is
needed anymore, so I would like to remove it, too. WDYT?
- @drammock did you create the token for Anaconda.org? I would like to
make this an environment secret (vs a repository secret as it is right
now)
- Also, while adding a new environment, I noticed a `github-pages`
environment that I do not think we are using, so I'd like to delete it.
- Finally, I deleted a leftover `PYPI_TOKEN,` which should have been
removed when we changed to trusted publishers.
 
Closes #2095

---------

Co-authored-by: Daniel McCloy <[email protected]>

Author: trallard

.github/actions/set-dev-env/action.yml

@@ -1,3 +1,8 @@
+# Reusable action to set our PST development environment
+# DO NOT use for release jobs since we cache dependencies
+# IMPORTANT: if you make changes to this action, you will need to open a follow-up
+# PR after merge to update the action SHA in the a11y, CI, docs, and prerelease
+# workflows
 name: Setup PST CI environment
 description: Create a PST dev environment
 

.github/dependabot.yml

@@ -4,15 +4,15 @@ updates:
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:
-      interval: "weekly"
+      interval: "monthly"
     labels:
       - "tag: dependencies"
       - "tag: CI"
   # npm
   - package-ecosystem: "npm"
     directory: "/"
     schedule:
-      interval: "weekly"
+      interval: "monthly"
     labels:
       - "tag: dependencies"
       - "tag: javascript"

.github/workflows/CI.yml

@@ -1,3 +1,8 @@
+# This workflow runs our usual CI jobs: testing with pytest, profiling,
+# and coverage checks for PST
+# IMPORTANT: if you make changes to this workflow, you will need to open a follow-up
+# PR after merge to update the action SHA in the publish workflow
+
 name: continuous-integration
 
 # Concurrency group that uses the workflow name and PR number if available
@@ -21,6 +26,9 @@ on:
     branches:
       - main
   pull_request:
+    branches:
+      - "*"
+  # allows this to be used as a composite action in other workflows
   workflow_call:
   # allow manual triggering of the workflow, while debugging
   workflow_dispatch:
@@ -31,10 +39,12 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
+        # https://github.com/actions/runner-images
         # macos-14==latest
-        # ubuntu-20.04==latest
-        os: ["ubuntu-latest", "ubuntu-24.04", "macos-14", "windows-latest"]
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        # ubuntu-24.04==latest
+        # windows-2022==latest
+        os: ["ubuntu-latest", "ubuntu-22.04", "macos-14", "windows-latest"]
+        python-version: ["3.9", "3.13"]
         sphinx-version: [""]
         include:
           # oldest Python version with the oldest Sphinx version
@@ -58,12 +68,17 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+
       - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
+        # Important: make sure to update the SHA after making any changes to the set-dev-env action
+        uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
         with:
           python-version: ${{ matrix.python-version }}
           pandoc: true
+
       - name: "Run tests ✅"
         shell: bash
         run: |
@@ -77,125 +92,40 @@ jobs:
           else
             python -Im tox run -e compile-assets,i18n-compile,py$(echo ${{ matrix.python-version }} | tr -d .)-tests
           fi
+
       - name: "Upload coverage data to GH artifacts 📤"
         if: matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest' && matrix.sphinx-version == 'dev'
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
         with:
           name: coverage-data-${{ matrix.python-version }}
           path: .coverage
           if-no-files-found: ignore
           include-hidden-files: true
 
-  # Only run accessibility tests on the latest Python version (3.12) and Ubuntu
-  a11y-tests:
-    name: "a11y-tests (ubuntu-latest, 3.12)"
-    runs-on: ubuntu-latest
-    steps:
-      - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
-      - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
-        with:
-          python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
-          pandoc: true
-          graphviz: true
-      - name: "Run accessibility tests with playwright 🎭"
-        # build PST, build docs, then run a11y-tests
-        run: python -Im tox run -m a11y
-      - name: "Upload Playwright traces, if any 🐾"
-        uses: actions/upload-artifact@v4
-        if: ${{ failure() }}
-        with:
-          name: playwright-traces
-          path: test-results/
-
-  # Build our docs (PST) on major OSes and check for warnings
-  build-site:
-    name: "build PST docs"
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: ["3.12"]
-        include:
-          # oldest Python version with the oldest Sphinx version
-          - os: ubuntu-latest
-            python-version: "3.9"
-            sphinx-version: "6.1"
-    runs-on: ${{ matrix.os }}
-    steps:
-      - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
-      - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
-        with:
-          python-version: ${{ matrix.python-version }}
-          pandoc: true
-          graphviz: true
-      - name: "Build docs and check for warnings 📖"
-        shell: bash
-        run: |
-          # check if there is a specific Sphinx version to build with
-          # example substitution: tox run -e py39-sphinx61-docs
-          if [ -n "${{matrix.sphinx-version}}" ]; then
-            python -Im tox run -e py$(echo ${{ matrix.python-version }} | tr -d .)-sphinx$(echo ${{ matrix.sphinx-version }} | tr -d .)-docs
-          # build with the default Sphinx version
-          # example substitution: tox run -e py312-docs
-          else
-            python -Im tox run -e py$(echo ${{ matrix.python-version }} | tr -d .)-docs
-          fi
-
-  # Run Lighthouse audits on the built site (kitchen-sink only)
-  lighthouse-audit:
-    needs: build-site
-    runs-on: ubuntu-latest
-    env:
-      DOCS_DIR: "audit"
-    steps:
-      - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
-      - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
-        with:
-          python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
-      - name: "Copy kitchen sink to a tiny site"
-        run: |
-          mkdir -p ${{ env.DOCS_DIR }}/site
-          cp -r docs/examples/kitchen-sink ${{ env.DOCS_DIR }}/site/kitchen-sink
-          printf "Test\n====\n\n.. toctree::\n\n   kitchen-sink/index\n" > ${{ env.DOCS_DIR }}/site/index.rst
-          echo 'html_theme = "pydata_sphinx_theme"' > ${{ env.DOCS_DIR }}/site/conf.py
-          echo '.. toctree::\n   :glob:\n\n   *' >> ${{ env.DOCS_DIR }}/site/index.rst
-
-          # build docs without checking for warnings
-          python -Im tox run -e docs-no-checks
-
-      - name: "Audit with Lighthouse 🔦"
-        uses: treosh/lighthouse-ci-action@v12
-        with:
-          configPath: ".github/workflows/lighthouserc.json"
-          temporaryPublicStorage: true
-          uploadArtifacts: true
-          runs: 3 # Multiple runs to reduce variance
-
   coverage:
     name: "Check coverage"
     needs: run-pytest
     runs-on: ubuntu-latest
-    # avoid running this on schedule, releases, or workflow_call
-    if: github.event_name != 'schedule' && github.event_name != 'release' && github.event_name != 'workflow_call'
+    # avoid running this on schedule, releases, workflow_call, or workflow_dispatch
+    if: github.event_name != 'schedule' && github.event_name != 'release' && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch'
     permissions:
       contents: write
       pull-requests: write
     steps:
       - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+
       - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
+        uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
         with:
           python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
+
       - run: python -Im pip install --upgrade coverage[toml]
+
       - name: "Download coverage data 📥"
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16
         with:
           pattern: coverage-data-*
           merge-multiple: true
@@ -213,45 +143,60 @@ jobs:
           python -Im coverage report --fail-under=80
 
       - name: "Upload HTML report if check failed 📤"
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
         with:
           name: html-report
           path: htmlcov
         if: ${{ failure() }}
 
       # seems we need to call this from the main CI workflow first
       - name: "Coverage comment 💬"
-        uses: py-cov-action/python-coverage-comment-action@v3
+        uses: py-cov-action/python-coverage-comment-action@b2eb38dd175bf053189b35f738f9207278b00925
         id: coverage_comment
         with:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: "Store Pull Request comment to be posted 📤"
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
         if: steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'
         with:
           # leave default names
           name: python-coverage-comment-action
           path: python-coverage-comment-action.txt
 
   profiling:
-    needs: [build-site, run-pytest]
+    needs: [run-pytest]
     runs-on: ubuntu-latest
     steps:
       - name: "Checkout repository 🛎"
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+
       - name: "Setup CI environment 🛠"
-        uses: ./.github/actions/set-dev-env
+        uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
         with:
           # 3.12 is not supported by py-spy yet
           python-version: "3.11"
+
       - name: "Run profiling with py-spy 🕵️‍♂️"
         # profiling needs to be run as sudo
         run: python -m tox run -e py311-profile-docs -- -o docbuild_profile.svg
         continue-on-error: true
+
       - name: "Upload profiling data to GH artifacts 📤"
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
         with:
           name: profile-results
           path: docbuild_profile.svg
           if-no-files-found: ignore
+
+  # Calling the coverage-comment action from the main CI workflow
+  # we might want to pin the SHA once merged
+  coverage-comment:
+    uses: ./.github/workflows/coverage.yml
+    needs: [coverage]
+    permissions:
+      contents: write
+      pull-requests: write
+      actions: read

.github/workflows/a11y.yml

@@ -0,0 +1,64 @@
+# Accessibility tests for PST, using playwright and axe-core
+
+name: a11y-tests
+
+# Concurrency group that uses the workflow name and PR number if available
+# or commit SHA as a fallback. If a new build is triggered under that
+# concurrency group while a previous build is running it will be canceled.
+# Repeated pushes to a PR will cancel all previous builds, while multiple
+# merges to main will not cancel.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
+env:
+  FORCE_COLOR: "1" # Make tools pretty
+  DEFAULT_PYTHON_VERSION: "3.12" # keep in sync with tox.ini
+  PIP_DISABLE_PIP_VERSION_CHECK: "1" # Don't check for pip updates
+
+permissions: {}
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - "*"
+  # allows this to be used as a composite action in other workflows
+  workflow_call:
+  # allow manual triggering of the workflow, while debugging
+  workflow_dispatch:
+
+jobs:
+  a11y-tests:
+    strategy:
+      fail-fast: true
+      matrix:
+        # ubuntu-latest = 24.04
+        os: ["ubuntu-latest", "ubuntu-22.04", "macos-14"]
+        browser: ["firefox", "chromium"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - name: "Checkout repository 🛎"
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+
+      - name: "Setup CI environment 🛠"
+        uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
+        with:
+          python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
+          pandoc: true
+          graphviz: true
+
+      - name: "Run accessibility tests with playwright 🎭"
+        # build PST, build docs, then run a11y-tests
+        run: python -Im tox run -e compile-assets,i18n-compile,py312-docs,a11y-tests-${{ matrix.browser }}
+
+      - name: "Upload Playwright traces, if any 🐾"
+        uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6
+        if: ${{ failure() }}
+        with:
+          name: playwright-traces
+          path: test-results/