Rewrite integration test in pytest

Move it from tasks/ into test/ as it involves more than the tasks
container, and could e.g. also grow integration tests for metrics.

Also set up ruff and mypy static code checks, and add a pyproject.toml
configuration for these.

Author: allisonkarlitskaya

.github/workflows/tests.yml

@@ -10,9 +10,11 @@ jobs:
       - name: Install dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y make python3-pyflakes python3-pycodestyle
+          sudo apt-get install -y make python3-pyflakes python3-pycodestyle python3-pip python3-pytest
+          # `pip install .[test]` does not work properly on Ubuntu 22.04
+          sudo pip install ruff mypy types-PyYAML
 
-      - name: Run unit tests
+      - name: Run lint tests
         run: make check
 
   tasks:
@@ -35,6 +37,13 @@ jobs:
           git config user.email [email protected]
           git rebase origin/main
 
+      - name: Install test dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y make python3-pip python3-pytest
+          # `pip install .[test]` does not work properly on Ubuntu 22.04
+          sudo pip install pytest_container
+
       # HACK: Ubuntu 22.04 has podman 3.4, which isn't compatible with podman-remote 4 in our tasks container
       # This PPA is a backport of podman 4.3 from Debian 12; drop this when moving `runs-on:` to ubuntu-24.04
       - name: Update to newer podman
@@ -53,10 +62,16 @@ jobs:
 
       - name: Build tasks container if it changed
         if: steps.containers_changed.outputs.tasks
-        run: make tasks-container
+        run: |
+          make tasks-container
+          # ensure the test uses the locally built one
+          # pytest_container doesn't have a "pull on demand" mode, so pull all
+          # of them and disable pulling in the test
+          podman pull docker.io/rabbitmq quay.io/minio/minio quay.io/minio/mc
+          echo "PULL_ALWAYS=0" >> "$GITHUB_ENV"
 
       - name: Test local deployment
         run: |
-          echo '${{ secrets.GITHUB_TOKEN }}' > ~/.config/github-token
+          echo '${{ secrets.GITHUB_TOKEN }}' > github-token
           PRN=$(echo "$GITHUB_REF" | cut -f3 -d '/')
-          tasks/run-local.sh -p $PRN -t ~/.config/github-token
+          python3 -m pytest -vv --pr $PRN --pr-repository '${{ github.repository }}' --github-token=github-token

.gitignore

@@ -6,3 +6,4 @@ local-data/
 tasks/credentials/**/*.key
 tasks/credentials/**/*.pem
 __pycache__
+port_check.lock

Makefile

@@ -6,6 +6,8 @@ all:
 check:
 	python3 -m pyflakes tasks tasks/container/webhook
 	python3 -m pycodestyle --max-line-length=120 --ignore=E722 tasks tasks/container/webhook
+	if command -v mypy >/dev/null && pip show types-PyYAML >/dev/null; then mypy test; else echo "SKIP: mypy or types-PyYAML not installed"; fi
+	if command -v ruff >/dev/null; then ruff check test; else echo "SKIP: ruff not installed"; fi
 
 TAG := $(shell date --iso-8601)
 TASK_SECRETS := /var/lib/cockpit-secrets/tasks

ansible/roles/webhook/tasks/main.yml

@@ -7,7 +7,7 @@
     dest: /run/cockpit-tasks-webhook.yaml
     mode: preserve
 
-# keep this in sync with tasks/run-local.sh
+# keep this in sync with test/test_deployment.py
 - name: Generate flat files from RabbitMQ config map
   shell: |
     rm -rf /etc/rabbitmq

pyproject.toml

@@ -0,0 +1,69 @@
+[project]
+name = "cockpituous"
+description = "Cockpit CI infrastructure"
+classifiers = [
+    "License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)"
+]
+
+dependencies = [
+    "pyyaml",
+]
+
+version = "1"
+
+[project.optional-dependencies]
+test = [
+    "ruff",
+    "mypy",
+    "pytest",
+    "pytest_container",
+    "types-PyYAML",
+]
+
+[tool.setuptools]
+packages = []
+
+[tool.pytest.ini_options]
+addopts = "-m 'not shell'"
+markers = [
+    "shell: interactive shell for development, skipped on normal runs",
+]
+
+[tool.ruff]
+exclude = [
+    ".git/",
+]
+line-length = 118
+src = []
+
+[tool.ruff.lint]
+select = [
+    "A",       # flake8-builtins
+    "B",       # flake8-bugbear
+    "C4",      # flake8-comprehensions
+    "D300",    # pydocstyle: Forbid ''' in docstrings
+    "DTZ",     # flake8-datetimez
+    "E",       # pycodestyle
+    "EXE",     # flake8-executable
+    "F",       # pyflakes
+    "FBT",     # flake8-boolean-trap
+    "G",       # flake8-logging-format
+    "I",       # isort
+    "ICN",     # flake8-import-conventions
+    "ISC",     # flake8-implicit-str-concat
+    "PIE",     # flake8-pie
+    "PLE",     # pylint errors
+    "PGH",     # pygrep-hooks
+    "PT",      # flake8-pytest-style
+    "RSE",     # flake8-raise
+    "RUF",     # ruff rules
+    "T10",     # flake8-debugger
+    "TCH",     # flake8-type-checking
+    "UP032",   # f-string
+    "W",       # warnings (mostly whitespace)
+    "YTT",     # flake8-2020
+]
+
+[[tool.mypy.overrides]]
+module = ["pytest_container.*", "testinfra.*"]
+ignore_missing_imports = true

tasks/README.md

@@ -60,33 +60,37 @@ Some helpful commands:
 
 # Deploying locally for development, integration tests
 
-For hacking on the webhook, task container, bots infrastructure,, or validating
+For hacking on the webhook, task container, bots infrastructure, or validating
 new container images, you can also run a [podman pod](http://docs.podman.io/en/latest/pod.html)
 locally with  RabbitMQ, webhook, minio S3, and tasks containers.
 Without arguments this will run some purely local integration tests:
 
-    tasks/run-local.sh
+    pytest
 
-This will also generate the secrets in a temporary directory, unless they
-already exist in `tasks/credentials/`. By default this will use the
+This will also generate the secrets in a temporary directory.
+By default this will use the
 [`ghcr.io/cockpit-project/tasks:latest`](https://ghcr.io/cockpit-project/tasks)
-container, but you can run a different tag by setting `$TASKS_TAG`.
+container, but you can run a different image by setting `$TASKS_IMAGE`.
 
 You can also test the whole GitHub → webhook → tasks → GitHub status workflow
-on some cockpituous PR with specifying the PR number and a GitHub token:
+on some cockpituous PR with specifying the PR number, your GitHub token, and
+optionally a non-default repository for testing against a fork:
 
-    tasks/run-local.sh -p 123 -t ~/.config/cockpit-dev/github-token
+    pytest -vvsk test_real_pr --pr 123 --pr-repository yourfork/cockpituous --github-token=/home/user/.config/cockpit-dev/github-token
 
 This will run tests-scan/tests-trigger on the given PR and trigger an
 [unit-tests](../.cockpit-ci/run) test which simply does `make check`.
 
 You can get an interactive shell with
 
-    tasks/run-local.sh -i
+    pytest -sm shell
 
 to run things manually. For example, use `publish-queue` to inject a job into
 AMQP, or run `job-runner` or some bots command.
 
+If you run the test multiple times, you can speed it up with `PULL_ALWAYS=0`
+which avoids re-pulling the container images.
+
 # Running with toolbx
 
 This container can also be used for local development with

tasks/mock-github

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # Mock GitHub API server for testing an opened PR or an issue for an image-refresh
-# You can run this manually in `tasks/run-local.sh -i` with `podman cp` and running
+# You can run this manually in `pytest -sm shell` with `podman cp` and running
 # cd bots
 # PYTHONPATH=. ./mock-github cockpit-project/bots $(git rev-parse HEAD) &
 # export GITHUB_API=http://127.0.0.7:8443