Skip to content

Commit 096f1ba

Browse files
isudakisudak
authored
Set up test configuration (#792)
- Put packages for tests into requirements-tests.txt - Sort packages in requirements.txt - Make pyproject.toml like in other `albs-*` repos - Ignore pytest warnings for the Pydantic package - Create test_db and web_server_tests services in docker-compose.yml - Added tests/test-alembic.ini and tests/test-vars.env for tests Resolves: AlmaLinux/build-system/issues/225
1 parent d61af3c commit 096f1ba

12 files changed

+198
-124
lines changed

‎.github/workflows/migrations-check.yml

+7-25
Original file line numberDiff line numberDiff line change
@@ -30,41 +30,23 @@ jobs:
3030
git merge --no-ff --no-log origin/master
3131
- name: Create dirs and files
3232
run: |
33-
mkdir -p /home/runner/work/albs-web-server/alts
34-
mkdir -p /home/runner/work/albs-web-server/albs-frontend
35-
mkdir -p /home/runner/work/albs-web-server/albs-node
36-
mkdir -p /home/runner/work/albs-web-server/albs-sign-node
37-
mkdir -p /home/runner/work/albs-web-server/alma-tests-cacher
38-
touch /home/runner/work/albs-web-server/alma-tests-cacher/vars.env
39-
mkdir -p /home/runner/work/albs-web-server/albs-sign-file
40-
touch /home/runner/work/albs-web-server/albs-sign-file/.env
41-
- name: Add env file
42-
run: |
43-
touch ./vars.env
44-
echo ALTS_TOKEN="secret" >> ./vars.env
45-
echo ALMALINUX_CLIENT="secret" >> ./vars.env
46-
echo ALMALINUX_CLIENT_SECRET="secret" >> ./vars.env
47-
echo GITHUB_CLIENT="secret" >> ./vars.env
48-
echo GITHUB_CLIENT_SECRET="secret" >> ./vars.env
49-
echo POSTGRES_PASSWORD="password" >> ./vars.env
50-
echo JWT_SECRET="secret" >> ./vars.env
51-
echo POSTGRES_DB="almalinux-bs" >> ./vars.env
52-
echo PACKAGE_BEHOLDER_ENABLED="False" >> ./vars.env
53-
echo DATABASE_URL="postgresql+asyncpg://postgres:password@db/almalinux-bs" >> ./vars.env
54-
echo SYNC_DATABASE_URL="postgresql+psycopg2://postgres:password@db/almalinux-bs" >> ./vars.env
55-
echo PULP_DATABASE_URL="postgresql+psycopg2://postgres:password@db/almalinux-bs" >> ./vars.env
33+
mkdir -p ../{alts,albs-frontend,albs-node,albs-sign-file,albs-sign-node,alma-tests-cacher}
34+
touch ../albs-sign-file/.env
35+
ln -sf tests/test-vars.env vars.env
5636
- name: Set up Docker Buildx
5737
uses: docker/setup-buildx-action@v3
5838
- name: Build Docker image
5939
uses: docker/build-push-action@v5
6040
with:
6141
context: .
42+
target: web-server-tests
6243
load: true
6344
cache-from: type=gha
6445
cache-to: type=gha,mode=max
6546
- name: Start services
66-
run: docker compose up -d --no-deps web_server db
47+
run: docker compose up -d test_db
6748
- name: Check migrations
68-
run: docker compose run --rm --no-deps web_server alembic --config alws/alembic.ini upgrade head
49+
run: docker compose run --rm web_server_tests alembic --config tests/test-alembic.ini upgrade head
6950
- name: Stop services
51+
if: always()
7052
run: docker compose down --volumes

‎.github/workflows/pytest.yml

+19-42
Original file line numberDiff line numberDiff line change
@@ -11,63 +11,39 @@ jobs:
1111
runs-on: ubuntu-latest
1212
timeout-minutes: 10
1313
steps:
14-
- uses: actions/checkout@v4
15-
name: Check out repository
16-
with:
17-
ref: ${{ github.event.pull_request.head.sha }}
14+
- name: Check out repository
15+
uses: actions/checkout@v4
16+
1817
- name: Create dirs and files
1918
run: |
20-
mkdir -p /home/runner/work/albs-web-server/alts
21-
mkdir -p /home/runner/work/albs-web-server/albs-frontend
22-
mkdir -p /home/runner/work/albs-web-server/albs-node
23-
mkdir -p /home/runner/work/albs-web-server/albs-sign-file
24-
mkdir -p /home/runner/work/albs-web-server/albs-sign-node
25-
mkdir -p /home/runner/work/albs-web-server/alma-tests-cacher
26-
touch /home/runner/work/albs-web-server/alma-tests-cacher/vars.env
27-
touch /home/runner/work/albs-web-server/albs-sign-file/.env
28-
- name: Add env file
29-
run: |
30-
touch ./vars.env
31-
echo ALTS_TOKEN="secret" >> ./vars.env
32-
echo ALMALINUX_CLIENT="secret" >> ./vars.env
33-
echo ALMALINUX_CLIENT_SECRET="secret" >> ./vars.env
34-
echo GITHUB_CLIENT="secret" >> ./vars.env
35-
echo GITHUB_CLIENT_SECRET="secret" >> ./vars.env
36-
echo POSTGRES_PASSWORD="password" >> ./vars.env
37-
echo JWT_SECRET="secret" >> ./vars.env
38-
echo POSTGRES_DB="test-almalinux-bs" >> ./vars.env
39-
echo PACKAGE_BEHOLDER_ENABLED="False" >> ./vars.env
40-
echo DATABASE_URL="postgresql+asyncpg://postgres:password@db/test-almalinux-bs" >> ./vars.env
41-
echo SYNC_DATABASE_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs" >> ./vars.env
42-
echo PULP_DATABASE_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs" >> ./vars.env
43-
echo FASTAPI_SQLA__PULP__SQLALCHEMY_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs" >> ./vars.env
44-
echo FASTAPI_SQLA__PULP__SQLALCHEMY_POOL_PRE_PING=True >> ./vars.env
45-
echo FASTAPI_SQLA__PULP__SQLALCHEMY_POOL_RECYCLE=3600 >> ./vars.env
46-
echo FASTAPI_SQLA__ASYNC__SQLALCHEMY_URL="postgresql+asyncpg://postgres:password@db/test-almalinux-bs" >> ./vars.env
47-
echo FASTAPI_SQLA__ASYNC__SQLALCHEMY_ECHO_POOL=True >> ./vars.env
48-
echo FASTAPI_SQLA__TEST__SQLALCHEMY_URL="postgresql+asyncpg://postgres:password@db/test-almalinux-bs" >> ./vars.env
49-
echo SQLALCHEMY_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs" >> ./vars.env
50-
echo SQLALCHEMY_POOL_PRE_PING=True >> ./vars.env
51-
echo SQLALCHEMY_POOL_RECYCLE=3600 >> ./vars.env
52-
cat alws/alembic.ini | sed 's/almalinux-bs/test-almalinux-bs/' | tee alws/alembic.ini
19+
mkdir -p ../{alts,albs-frontend,albs-node,albs-sign-file,albs-sign-node,alma-tests-cacher}
20+
touch ../albs-sign-file/.env
21+
ln -sf tests/test-vars.env vars.env
22+
5323
- name: Set up Docker Buildx
5424
uses: docker/setup-buildx-action@v3
25+
5526
- name: Build Docker image
5627
uses: docker/build-push-action@v5
5728
with:
5829
context: .
30+
target: web-server-tests
5931
load: true
6032
cache-from: type=gha
6133
cache-to: type=gha,mode=max
34+
6235
- name: Start services
63-
run: docker compose up -d --no-deps web_server db
36+
run: docker compose up -d test_db
37+
6438
- name: Run unit tests (pytest)
65-
run: docker compose run -v /tmp:/tmp --no-deps web_server bash -o pipefail -c '
66-
pytest -vv --ignore alws/ --cov-report term-missing:skip-covered
67-
--cov-report xml:/tmp/coverage.xml --junitxml=/tmp/pytest.xml --cov=alws
68-
tests/ | tee /tmp/pytest-coverage.txt'
39+
run: docker compose run --rm -v /tmp:/tmp web_server_tests bash -c "
40+
pytest -v --cov
41+
--cov-report xml:/tmp/coverage.xml --junitxml=/tmp/pytest.xml
42+
--cov-report term-missing:skip-covered | tee /tmp/pytest-coverage.txt"
43+
6944
- name: Stop services
7045
run: docker compose down --volumes
46+
7147
- name: Pytest coverage comment
7248
uses: MishaKav/pytest-coverage-comment@main
7349
id: coverageComment
@@ -82,6 +58,7 @@ jobs:
8258
hide-comment: false
8359
remove-link-from-badge: false
8460
junitxml-path: /tmp/pytest.xml
61+
8562
- name: Create the Badge
8663
if: ${{ github.ref == 'refs/heads/master' && steps.coverageComment.outputs.coverage }}
8764
uses: schneegans/[email protected]

‎.github/workflows/syntax.yml

+8-12
Original file line numberDiff line numberDiff line change
@@ -18,21 +18,17 @@ jobs:
1818
with:
1919
files: |
2020
**/*.py
21-
- name: Prepare python env
21+
- name: Install python packages
2222
if: ${{ steps.changed-files.outputs.all_changed_files }}
23-
run: |
24-
python -m venv env
25-
source env/bin/activate
26-
pip install -U pip
27-
pip install -r requirements.txt
23+
run: pip install -r requirements-tests.txt
2824
- name: Run pylint
2925
id: pylint
3026
if: ${{ steps.changed-files.outputs.all_changed_files }}
3127
run: |
32-
delimiter='$(openssl rand -hex 8)'
33-
echo 'report<<$(delimiter)' >> $GITHUB_OUTPUT
34-
env/bin/pylint ${{ steps.changed-files.outputs.all_changed_files }} --py-version 3.9 --exit-zero >> $GITHUB_OUTPUT
35-
echo '$(delimiter)' >> $GITHUB_OUTPUT
28+
delimiter=$(openssl rand -hex 8)
29+
echo "report<<$delimiter" >> $GITHUB_OUTPUT
30+
pylint ${{ steps.changed-files.outputs.all_changed_files }} --py-version 3.9 --exit-zero >> $GITHUB_OUTPUT
31+
echo $delimiter >> $GITHUB_OUTPUT
3632
- name: Post pylint output
3733
uses: mshick/add-pr-comment@v2
3834
if: ${{ steps.changed-files.outputs.all_changed_files }}
@@ -48,7 +44,7 @@ jobs:
4844
message-id: pylint-report
4945
- name: Run black
5046
if: ${{ steps.changed-files.outputs.all_changed_files }}
51-
run: env/bin/black ${{ steps.changed-files.outputs.all_changed_files }} --exclude alws/alembic --check --diff --color --target-version py39
47+
run: black ${{ steps.changed-files.outputs.all_changed_files }} --exclude alws/alembic --check --diff --color --target-version py39
5248
- name: Run isort
5349
if: ${{ steps.changed-files.outputs.all_changed_files }}
54-
run: env/bin/isort ${{ steps.changed-files.outputs.all_changed_files }} --diff --color --check-only --py 39
50+
run: isort ${{ steps.changed-files.outputs.all_changed_files }} --diff --color --check-only --py 39

‎Dockerfile

+11-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM almalinux/9-base:latest
1+
FROM almalinux/9-base:latest as web-server
22

33
RUN <<EOT
44
set -ex
@@ -18,3 +18,13 @@ RUN <<EOT
1818
EOT
1919

2020
ADD --chmod=755 https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh /
21+
22+
23+
FROM web-server as web-server-tests
24+
25+
COPY requirements-tests.txt .
26+
RUN <<EOT
27+
set -ex
28+
pip3 install -r requirements-tests.txt
29+
rm requirements-tests.txt
30+
EOT

‎docker-compose.yml

+21-3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,24 @@ version: "3.9"
22

33
services:
44

5+
test_db:
6+
image: postgres:13
7+
env_file:
8+
- vars.env
9+
ports:
10+
- "5432:5432"
11+
12+
web_server_tests:
13+
image: albs-web-server-tests:latest
14+
env_file:
15+
- vars.env
16+
build:
17+
context: .
18+
dockerfile: Dockerfile
19+
target: web-server-tests
20+
volumes:
21+
- ".:/code"
22+
523
db:
624
image: postgres:13
725
env_file:
@@ -99,13 +117,13 @@ services:
99117
ports:
100118
- "8088:8000"
101119
build:
102-
dockerfile: Dockerfile
103120
context: .
121+
dockerfile: Dockerfile
122+
target: web-server
104123
volumes:
105124
- "./alws:/code/alws"
106125
- "./scripts:/code/scripts"
107126
- "./reference_data:/code/reference_data"
108-
- "./tests:/code/tests"
109127
- "../volumes/pulp/exports:/srv/exports"
110128
command: |
111129
bash -c "
@@ -380,7 +398,7 @@ services:
380398
max-file: "3"
381399

382400
git_cacher:
383-
image: quay.io/almalinuxorg/git-cacher:latest
401+
image: git-cacher:latest
384402
build:
385403
dockerfile: Dockerfile.git-cacher
386404
context: .

‎pyproject.toml

+27-2
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,36 @@ profile = "black"
33
split_on_trailing_comma = true
44

55
[tool.black]
6-
line-length = 79
6+
line-length = 120
77
skip-string-normalization = true
88
# see https://black.readthedocs.io/en/stable/the_black_code_style/future_style.html#preview-style
99
preview = true
1010
enable-unstable-feature = ["hug_parens_with_braces_and_square_brackets"]
1111

1212
[tool.pylint]
13-
max-line-length = 80
13+
max-line-length = 120
14+
15+
# Minimum line length for functions/classes that require docstrings
16+
docstring-min-length = 50
17+
18+
# https://pylint.readthedocs.io/en/stable/user_guide/checkers/features.html
19+
disable = [
20+
"C0114", # missing-module-docstring
21+
"R0902", # too-many-instance-attributes
22+
"R0913", # too-many-arguments
23+
"W1514", # unspecified-encoding
24+
]
25+
26+
[tool.coverage.run]
27+
source = ['.']
28+
29+
[tool.coverage.report]
30+
skip_empty = true
31+
include = ['alws/*']
32+
33+
[tool.pytest.ini_options]
34+
pythonpath = '.'
35+
testpaths = ['tests']
36+
filterwarnings = [
37+
'ignore:Pydantic serializer warnings:UserWarning'
38+
]

‎requirements-tests.txt

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
pytest==8.1.1
2+
pytest-cov==5.0.0
3+
pyfakefs==5.4.0
4+
5+
# Linters
6+
isort[colors]==5.13.2
7+
black==24.3.0
8+
pylint==3.1.0
9+
bandit[toml]==1.7.8

‎requirements.txt

+23-28
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,35 @@
1-
fastapi==0.110.1
2-
fastapi-users[all]==13.0.0
3-
# Can be updated only after migration to SQLAlchemy > 2.0
4-
fastapi-users-db-sqlalchemy==6.0.1
5-
uvicorn==0.29.0
6-
asyncpg==0.29.0
7-
psycopg2-binary==2.9.9
8-
pydantic==2.6.4
9-
pydantic-settings==2.2.1
1+
PyYAML==6.0.1
102
SQLAlchemy==2.0.29
11-
aiohttp==3.9.3
12-
aiohttp-retry==2.8.3
133
aiofiles==23.2.1
4+
aiohttp-retry==2.8.3
5+
aiohttp==3.9.3
146
aioredis==2.0.1
15-
alembic==1.13.1
167
alembic-postgresql-enum==1.1.2
17-
pytest==8.1.1
18-
pytest-cov==5.0.0
19-
isort[colors]==5.13.2
20-
black==24.3.0
21-
pylint==3.1.0
8+
alembic==1.13.1
9+
asyncpg==0.29.0
10+
dramatiq[rabbitmq, watch]==1.16.0
11+
email-validator>=2.0
12+
# Can be updated only after migration to SQLAlchemy > 2.0
13+
fastapi-users-db-sqlalchemy==6.0.1
14+
fastapi-users[all]==13.0.0
15+
fastapi==0.110.1
16+
feedgen==1.0.0
17+
httpx-oauth==0.13.3
18+
jinja2==3.1.3
2219
jmespath==1.0.1
23-
PyYAML==6.0.1
2420
lxml==5.2.1
25-
syncer==2.0.3
26-
dramatiq[rabbitmq, watch]==1.16.0
27-
python-multipart>=0.0.7
21+
markdown==3.6
22+
pgpy==0.6.0
2823
plumbum==1.8.2
24+
psycopg2-binary==2.9.9
25+
pydantic-settings==2.2.1
26+
pydantic==2.6.4
27+
python-multipart>=0.0.7
28+
sentry-sdk[fastapi]==1.45.0 # requires FastAPI>=0.79.0
29+
syncer==2.0.3
2930
tap.py==3.1
30-
jinja2==3.1.3
31-
pgpy==0.6.0
32-
markdown==3.6
33-
httpx-oauth==0.13.3
34-
feedgen==1.0.0
31+
uvicorn==0.29.0
3532
websockets==12.0
36-
email-validator>=2.0
37-
sentry-sdk[fastapi]==1.45.0 # requires FastAPI>=0.79.0
3833
git+https://github.com/AlmaLinux/[email protected]#egg=immudb_wrapper
3934
git+https://github.com/AlmaLinux/[email protected]#egg=errata2osv
4035
git+https://github.com/AlmaLinux/[email protected]#egg=albs_github

‎tests/README.md

+12-10
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,23 @@
66

77
`mock_classes.py` - a module which contain base class with `httpx` request method, setup logic for each test suite and HTTP status codes
88
## How to run tests locally
9-
1. Create `test-almalinux-bs` database
10-
2. Adjust variables in `vars.env`
9+
1. Adjust variables in `vars.env`
1110
```
1211
POSTGRES_DB="test-almalinux-bs"
1312
POSTGRES_PASSWORD="password"
14-
DATABASE_URL="postgresql+asyncpg://postgres:password@db/test-almalinux-bs"
15-
SYNC_DATABASE_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs"
16-
PULP_DATABASE_URL="postgresql+psycopg2://postgres:password@db/test-almalinux-bs"
13+
DATABASE_URL="postgresql+asyncpg://postgres:password@test_db/test-almalinux-bs"
14+
SYNC_DATABASE_URL="postgresql+psycopg2://postgres:password@test_db/test-almalinux-bs"
15+
PULP_DATABASE_URL="postgresql+psycopg2://postgres:password@test_db/test-almalinux-bs"
1716
```
18-
3. Up docker-compose services
17+
or use `test-vars.env` in the `tests` folder
18+
```bash
19+
ln -sf tests/test-vars.env vars.env
20+
```
21+
3. Start the `test_db` service
1922
```bash
20-
docker-compose up -d --no-deps web_server db
23+
docker compose up -d test_db
2124
```
22-
4. Run `pytest` within `web_server` container
25+
3. Run `pytest` within `web_server_tests` container
2326
```bash
24-
docker-compose run --no-deps --rm web_server bash -c 'source env/bin/activate && pytest -v --ignore alws/'
27+
docker compose run --rm web_server_tests pytest -v
2528
```
26-
- we ignore `alws/` directory because it's contains files which names starts with `test*.py`

0 commit comments

Comments
 (0)