Skip to content

Commit

Permalink
CI and development environment overhaul
Browse files Browse the repository at this point in the history
This removes the docker image logic from the CI setup and instead
replaces it with a setup based on pixi. As part of that it moves the
dependency specification from localbuild/meta.yaml to pixi.toml and
pixi.lock. This turns the MSS repository into a single source of truth
for both the application code as well as the development environment
(whereas the latter was previously only specified in the docker images,
and not reproducible in any way).

Setting up a development environment is as simple as installing pixi and
running `pixi shell` (or `pixi run <cmd>`, or `pixi install` to just
create the environment, etc.). This environment will, by construction,
be the same that is used in the CI as well (modulo platform
differences).

There is a new workflow that periodically (once a week on Monday)
recreates the pixi lockfile and opens a PR for that update. The checks
in that PR essentially serve as a replacement for the previous scheduled
runs to ensure that no dependency update breaks MSS. Merging that PR is
a manual step that can be done just as with any other PR and would then
update the environment on the given target branch. This is essentially
what was previously the triggering of a docker image creation.

Including new dependencies can be done with `pixi add`, which will also
automatically add the dependency to the (pre-existing) lockfile. This
means dependency additions can be part of the PR that necessitate them
and they won't affect the entire environment (as they previously did,
where they would trigger a full image rebuild) but instead just add that
new package to the existing specification.
  • Loading branch information
matrss committed Feb 11, 2025
1 parent 7d597f2 commit 1212e20
Show file tree
Hide file tree
Showing 30 changed files with 18,190 additions and 602 deletions.
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
* text=auto
# GitHub syntax highlighting
pixi.lock linguist-language=YAML
14 changes: 7 additions & 7 deletions .github/workflows/build_docs_gallery.yml
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
name: Build Gallery

on:
pull_request:
pull_request:

jobs:
Test-MSS-docs:
runs-on: ubuntu-latest

container:
image: openmss/testing-develop

steps:
- uses: actions/checkout@v4

- uses: prefix-dev/[email protected]
with:
pixi-version: latest
cache: true
environments: dev
- name: Create gallery
timeout-minutes: 5
run: |
cd docs
mamba run --no-capture-output -n mssenv python conf.py
pixi run -e dev python conf.py
24 changes: 10 additions & 14 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,30 +18,26 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Set up Python 3
uses: actions/setup-python@v5
- uses: prefix-dev/[email protected]
with:
python-version: "3.x"
pixi-version: latest
cache: true
environments: dev
- name: Lint with codespell
run: |
python -m pip install --upgrade pip
pip install codespell
git ls-files -z | xargs -0 codespell --check-filenames
run: git ls-files -z | xargs -0 pixi run -e dev codespell --check-filenames

flake8:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
- uses: prefix-dev/[email protected]
with:
python-version: "3.10"
pixi-version: latest
cache: true
environments: dev
- name: Lint with flake8
run: |
python -m pip install --upgrade pip
pip install flake8 flake8-builtins
git ls-files -z '**.py' | xargs -0 flake8
run: git ls-files -z '**.py' | xargs -0 pixi run -e dev flake8

no-crlf-in-git:
runs-on: ubuntu-latest
Expand Down
26 changes: 5 additions & 21 deletions .github/workflows/testing-all-oses.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,27 +21,11 @@ jobs:
os: ["macos-13", "macos-14", "ubuntu-latest"]
steps:
- uses: actions/checkout@v4
- name: Build requirements.txt file
run: |
sed -n '/^requirements:/,/^test:/p' localbuild/meta.yaml |
sed -e "s/.*- //" |
sed -e "s/menuinst.*//" |
sed -e "s/.*://" > requirements.tmp.txt
cat requirements.d/development.txt >> requirements.tmp.txt
echo "pytest-randomly" >> requirements.tmp.txt
sed -e '/^$/d' -e '/^#.*$/d' -e 's/\s*# \[not win\]$//' requirements.tmp.txt > requirements.txt
rm requirements.tmp.txt
cat requirements.txt
- name: Get current year and calendar week
id: year-and-week
run: echo "year-and-week=$(date +%Y-%V)" >> "$GITHUB_OUTPUT"
- uses: mamba-org/setup-micromamba@v2
- uses: prefix-dev/[email protected]
with:
environment-file: requirements.txt
environment-name: ci
cache-environment: true
# Set the cache key in a way that the cache is invalidated every week on monday
cache-environment-key: environment-${{ steps.year-and-week.outputs.year-and-week }}
pixi-version: latest
cache: true
environments: dev
- name: Run tests
timeout-minutes: 40
run: micromamba run -n ci env QT_QPA_PLATFORM=offscreen pytest -v -n logical --durations=20 --cov=mslib tests
run: pixi run -e dev env QT_QPA_PLATFORM=offscreen pytest -v -n logical --durations=20 --cov=mslib tests
19 changes: 0 additions & 19 deletions .github/workflows/testing-develop.yml

This file was deleted.

16 changes: 0 additions & 16 deletions .github/workflows/testing-gsoc.yml

This file was deleted.

26 changes: 0 additions & 26 deletions .github/workflows/testing-scheduled.yml

This file was deleted.

19 changes: 0 additions & 19 deletions .github/workflows/testing-stable.yml

This file was deleted.

79 changes: 0 additions & 79 deletions .github/workflows/testing.yml

This file was deleted.

39 changes: 39 additions & 0 deletions .github/workflows/update-pixi-lockfile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Update pixi lockfile

on:
schedule:
# At 04:00 on Monday
- cron: "0 4 * * 1"
# ...or manually
workflow_dispatch:

jobs:
pixi-update:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
base_branch: ["develop", "stable"]
steps:
- uses: actions/checkout@v4
with:
ref: ${{ matrix.base_branch }}
- name: Generate new lockfile
uses: prefix-dev/[email protected]
with:
pixi-version: latest
run-install: false
- name: Update pixi lockfile
run: |
set -o pipefail
pixi update --json | pixi exec pixi-diff-to-markdown >> diff.md
- name: Create or update pull request
uses: peter-evans/create-pull-request@v7
with:
token: ${{ secrets.PAT }}
branch: automation/update-pixi-lockfile
delete-branch: true
add-paths: pixi.lock
commit-message: Update pixi lockfile
title: Update pixi lockfile
body-path: diff.md
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ docs/gallery/code
docs/gallery/plots.html
docs/videos/
build/
mss.egg-info/
mslib.egg-info/
tutorials/recordings
tutorials/cursor_image.png
# pixi environments
.pixi
Loading

0 comments on commit 1212e20

Please sign in to comment.