feat(deposition): add ENA integration tests (#3700)

* start adding integration tests for ena deposition: mock request to the backend and slack and submit to ENA dev * add integration tests in github action * fixes: In order for us to receive slack notifications if given BioProject and BioSample do not exist the project and sample table must have an entry in state "HAS_ERRORS" -> I have changed this and added an appropriate test, currently we would not receive a notification but the submission would be stopped. * fixes: If BioSample is given the results table must be updated with both "ena_sample_accession" (ena private accession) and "biosample_accession" (INSDC accession), both can be used for creating assemblies but "biosample_accession" is used when uploading external metadata to Loculus. * define webin-cli version in .webinrc file * improves typing
loculus-project · Feb 24, 2025 · 57936bc · 57936bc
1 parent 705f365
commit 57936bc
Show file tree

Hide file tree

Showing 13 changed files with 816 additions and 48 deletions.
diff --git a/.github/workflows/ena-submission-tests.yaml → .../workflows/ena-submission-unit-tests.yaml b/.github/workflows/ena-submission-tests.yaml → .../workflows/ena-submission-unit-tests.yaml
@@ -1,16 +1,16 @@
-name: ena-submission-tests
+name: ena-submission-unit-tests
 on:
   # test
   pull_request:
     paths:
       - "ena-submission/**"
-      - ".github/workflows/ena-submission-tests.yaml"
+      - ".github/workflows/ena-submission-unit-tests.yaml"
   push:
     branches:
       - main
   workflow_dispatch:
 concurrency:
-  group: ci-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}-ena-submission-tests
+  group: ci-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}-ena-submission-unit-tests
   cancel-in-progress: true
 jobs:
   unitTests:
@@ -31,7 +31,7 @@ jobs:
             post-cleanup: 'all'
       - name: Run tests
         run: |
-            pip install -e .
+            pip install .
             python3 scripts/test_ena_submission.py
         shell: micromamba-shell {0}
         working-directory: ena-submission
diff --git a/.github/workflows/ena-submission-workflow-tests.yml b/.github/workflows/ena-submission-workflow-tests.yml
@@ -0,0 +1,63 @@
+name: ena-submission-workflow-tests
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/ena-submission-workflow-tests.yml"
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+concurrency:
+  group: ci-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}-ena-submission-workflow-tests
+  cancel-in-progress: true
+jobs:
+  Tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      ENA_USERNAME: ${{ secrets.ENA_SUBMISSION_USER }}
+      ENA_PASSWORD: ${{ secrets.ENA_SUBMISSION_PASSWORD }}
+    services:
+      postgres:
+        image: postgres:15
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: unsecure
+          POSTGRES_DB: loculus
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Flyway
+        run: |
+          curl -L -o flyway.tar.gz "https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.0.0/flyway-commandline-10.0.0-linux-x64.tar.gz"
+          tar -xzf flyway.tar.gz
+          sudo mv flyway-10.0.0 /opt/flyway
+          echo "/opt/flyway" >> $GITHUB_PATH
+      - name: Run Flyway
+        run: flyway -url=jdbc:postgresql://localhost:5432/loculus -schemas=ena_deposition_schema -user=postgres -password=unsecure -locations=filesystem:./ena-submission/flyway/sql migrate
+      - name: Install WEBIN CLI
+        run: |
+          WEBIN_CLI_VERSION=$(cat ena-submission/.webinrc)
+          wget -q "https://github.com/enasequence/webin-cli/releases/download/${WEBIN_CLI_VERSION}/webin-cli-${WEBIN_CLI_VERSION}.jar" -O ena-submission/webin-cli.jar
+      - name: Set up micromamba
+        uses: mamba-org/setup-micromamba@v2
+        with: 
+            environment-file: ena-submission/environment.yml
+            micromamba-version: 'latest'
+            init-shell: >-
+                bash
+                powershell
+            cache-environment: true
+            post-cleanup: 'all'
+      - name: Run tests
+        run: |
+            pip install .
+            python3 scripts/test_ena_submission_integration.py
+        shell: micromamba-shell {0}
+        working-directory: ena-submission
diff --git a/ena-submission/.webinrc b/ena-submission/.webinrc
@@ -0,0 +1 @@
+8.1.1
diff --git a/ena-submission/Dockerfile b/ena-submission/Dockerfile
@@ -19,9 +19,11 @@ RUN micromamba config set extract_threads 1 \
 # Set the environment variable to activate the conda environment
 ARG MAMBA_DOCKERFILE_ACTIVATE=1
 
-ENV WEBIN_CLI_VERSION 8.1.1
+COPY .webinrc /tmp/.webinrc
+
 USER root
-RUN wget -q "https://github.com/enasequence/webin-cli/releases/download/${WEBIN_CLI_VERSION}/webin-cli-${WEBIN_CLI_VERSION}.jar" -O /package/webin-cli.jar
+RUN export WEBIN_CLI_VERSION=$(cat /tmp/.webinrc) && \
+    wget -q "https://github.com/enasequence/webin-cli/releases/download/${WEBIN_CLI_VERSION}/webin-cli-${WEBIN_CLI_VERSION}.jar" -O /package/webin-cli.jar
 USER $MAMBA_USER
 
 COPY --chown=$MAMBA_USER:$MAMBA_USER . /package

diff --git a/ena-submission/README.md b/ena-submission/README.md
@@ -94,11 +94,11 @@ In a loop:
 
 - Get sequences in `submission_table` in state SUBMITTED_SAMPLE
   - if (there exists an entry in the `assembly_table` for the corresponding (accession, version)):
-    - if (entry is in status SUBMITTED): update `assembly_table` to SUBMITTED_ASSEMBLY.
+    - if (entry is in status SUBMITTED): update `assembly_table` to SUBMITTED_ALL.
     - else: update `assembly_table` to SUBMITTING_ASSEMBLY.
   - else: create assembly entry in `assembly_table` for (accession, version).
 - Get sequences in `submission_table` in state SUBMITTING_SAMPLE
-  - if (corresponding `assembly_table` entry is in state SUBMITTED): update entries to state SUBMITTED_ASSEMBLY.
+  - if (corresponding `assembly_table` entry is in state SUBMITTED): update entries to state SUBMITTED_ALL.
 - Get sequences in `assembly_table` in state READY, prepare files: we need chromosome_list, fasta files and a manifest file, set status to WAITING
   - if (submission succeeds): set status to WAITING and fill in results: ena-internal `erz_accession`
   - else: set status to HAS_ERRORS and fill in errors
@@ -197,6 +197,17 @@ You can also use the `deposition_dry_run.py` script to produce the same output f
 python scripts/deposition_dry_run.py --log-level=DEBUG --data-to-submit=results/approved_ena_submission_list.json --mode=assembly --center-name="Yale" --config-file=config/config.yaml
 ```
 
+You can also run the integration tests locally, these will submit sequences to ENA dev. Be careful when modifying these tests to always set `test=true`. 
+
+```sh
+docker run --name test-postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=unsecure -e POSTGRES_DB=loculus -p 5432:5432 -d postgres
+flyway -url=jdbc:postgresql://localhost:5432/loculus -schemas=ena_deposition_schema -user=postgres -password=unsecure -locations=filesystem:./flyway/sql migrate
+python3 scripts/test_ena_submission_integration.py
+```
+
+The tests perform the same tests as described in the section on testing submission locally below. 
+
+
 ### Testing submission locally
 
 1. Run loculus locally (need prepro, backend and ena-submission pod), e.g.