Add Postgres support to Autosubmit.

Add Postgres support to Autosubmit

Implemented with SQLAlchemy, abstract/OOP/protocols, a new configuration key DATABASE_BACKEND.

Tested with Docker and TestContainers. More tests added (unit and integration).

Author: LuiggiTenorioK

.coveragerc

@@ -18,6 +18,7 @@ omit =
 parallel = True
 source = ./autosubmit
 timid = False
+sigterm = True
 
 [report]
 exclude_lines =

.github/workflows/ci.yaml

@@ -33,10 +33,10 @@ jobs:
   #       run: |
   #         ruff check .
 
-  test:
+  test-unit:
     # needs: lint
     runs-on: ubuntu-latest
-    timeout-minutes: 10
+    timeout-minutes: 15
 
     strategy:
       matrix:
@@ -50,9 +50,11 @@ jobs:
         uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
+          # See: https://github.com/actions/setup-python#caching-packages-dependencies
+          cache: 'pip' # caching pip dependencies
 
       - name: Install system dependencies
-        run: sudo apt-get install -y graphviz rsync curl
+        run: sudo apt-get install -y curl graphviz rsync
 
       - name: Install dependencies
         run: |
@@ -64,7 +66,8 @@ jobs:
           pytest \
             --cov=autosubmit --cov-config=.coveragerc \
             --cov-report=xml:test/coverage.xml --cov-append \
-            test/unit
+            --dist load \
+            test/unit -m "not postgres"
 
       - name: Coverage report
         run: |
@@ -85,6 +88,16 @@ jobs:
     strategy:
       matrix:
         python-version: ["3.9", "3.10", "3.11", "3.12"]
+        pytest-markers: ["not postgres"]
+        # Here we include an extra build with the lowest version of Python and Postgres.
+        # We do it this way to avoid wasting CICD resources since we already tested the
+        # code with SQLite. So we run just one build with Postgres (normally newer versions
+        # of Python may succeed while older fail -- e.g. using newer features in a code that
+        # must support older versions).
+        # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#example-adding-configurations
+        include:
+          - python-version: '3.9'
+            pytest-markers: 'postgres'
 
     steps:
       - name: Checkout code
@@ -94,6 +107,8 @@ jobs:
         uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
+          # See: https://github.com/actions/setup-python#caching-packages-dependencies
+          cache: 'pip' # caching pip dependencies
 
       - name: Install system dependencies
         run: sudo apt-get install -y curl git graphviz rsync
@@ -119,8 +134,9 @@ jobs:
           pytest \
             --cov=autosubmit --cov-config=.coveragerc \
             --cov-report=xml:test/coverage.xml --cov-append \
+            --dist load \
             test/integration \
-            -m 'not slurm'
+            -m "${{ matrix.pytest-markers }}"
 
       - name: Coverage report
         run: |
@@ -130,7 +146,7 @@ jobs:
       - name: Upload coverage artifact
         uses: actions/upload-artifact@v4
         with:
-          name: coverage_integration_py-${{ matrix.python-version }}
+          name: coverage_integration_py-${{ matrix.python-version }}-${{ matrix.pytest-markers }}
           path: coverage.xml
           retention-days: 7
 
@@ -307,7 +323,7 @@ jobs:
           cffconvert --validate -i CITATION.cff
 
   coverage:
-    needs: [test, test-integration, test-regression, test-slurm, test-docs]
+    needs: [test-unit, test-integration, test-regression, test-slurm, test-docs]
     runs-on: ubuntu-latest
     timeout-minutes: 2
     steps:

CHANGELOG.md

@@ -1,5 +1,12 @@
 ### 4.1.16: Unreleased
 
+This release adds support to Postgres using SQLAlchemy, without removing the
+SQLite support. By default, Autosubmit will use SQLite. Postgres support is
+experimental and not recommended for production yet.
+
+Autosubmit Config Parser code has been merged back into this code base. LOCAL
+platform does not support wrappers anymore (it was used for testing).
+
 **Bug fixes:**
 
 - Fixed issue with the verification of dirty Git local repositories in operational experiments #2446
@@ -20,6 +27,7 @@
   automatically answer yes to prompts #2569
 - Improvement of error message when `LOCAL` project location is a file, not a directory #1972 #1254
 - Removed the code for wrappers with local platform that were create only for tests #2522
+- Added SQLAlchemy as the main database entrypoint, enabling backends of Sqlite (default) and Postgres (new) #2187
 
 ### 4.1.15: Bug fixes, enhancements, and new features
 
@@ -48,7 +56,7 @@ the filter the jobs. Not using any value for `-fp` still returns all jobs.
 - EDITO Autosubmit-Demo container updated to install API in different environment #2398
 - Update portalocker requirement from <=3.1.1 to <=3.2.0 #2423
 - Additional files are now generated upon using the `autosubmit inspect` command #2323
-- Operational runs now require no pending commits, ensuring a cleaner workflow. [#2220](https://github.com/BSC-ES/autosubmit/issues/2220), [PR](https://github.com/BSC-ES/autosubmit/pull/2293)
+- Operational runs now require no pending commits, ensuring a cleaner workflow. [#2220](https://github.com/BSC-ES/autosubmit/issues/2220), [PR](https://github.com/BSC-ES/autosubmit/pull/2293) 
 
 ### 4.1.14: Bug fixes, enhancements, and new features