Contains all the job_list changes

Fixing issues

fixing post rebase issues

Remove an innecesary call

Contains all the job_list changes

small fix for update from file

fix bug in check_running

Fix for a large amount of inserts

Fix for split

Fixed postgres init

Finished the integration tests

wip ( only one case missing)

WIP moved to db,

Integration test for differences

wip

docs fix

regression test fix

docs not working

Fix pipeline

docs try

update version

Fixing remaining issues

Main loop changes, db changes, test changes, check_alljobs changes

wrapper checker reworked (wip)

wrappers not quite working yet

rebase

fixing wrappers not all working yet

trying un succesfully to fix tests

Fixing and adding tests for horizontal wrapper

trying to add tests

fix tests

rebase + fix for docker ( not sure why I needed to do it)

Contains all as-postgres-joblist implementation

Fixing tests, deleting unused code, main wrapped db is no longer deleted

Fixed stats, set_status, recovery # maybe something is not working properly

Recover last_log/ready_data moved to a db consult instead of files

post rebase fix

Sqlalchemy was not fully supporting sqlite

Changes recovery #2577,  #2563

fix comment

backups/restore, fix monitor

unused code

fixing docker issue ( revert 5fcfc8f)

fix tests

fix rebase bug

Fix platform issue for local platform and status for create

Adding more tests, monitor is not working well in wrappers, fixed issue with job.parents not being filled

Renamed db fields and other fixes

adding tests

remove unused code

Contains all as-postgres-joblist implementation

fix wrong ident

postgres was failing due not getting the table from the schema

more efficient

rework clean_unused_nodes to cover all cases

Adding some tests for wrappers section..

disabling tests for code that is not on this branch

fixing ident after GUI conflict resolve

Fixing tests

some tests

delete debug stuff

Wrappers run, ( could be optimized tho)

fixed some wrapper issues, but not all

Revised the todo's

Fixed issues with multi-wrappers ( run untested)
added more fields
Attempt to fix concurrent runs

fixed -H, added expid protection, fixed create_required_folders

marking xfail, testing option

some XFAIL marked tests

fixing tests ( some integration fails rng)

Rebase + trying to fix tests after rebase.

Integration fixed, regression failling the config one I think is due a change of TYPE: pkl -> type: sqlite

27 unit tests not working yet

Super commit - Changed joblist, change Status filters, tests passing

All current tests passing

non-skipped Unit tests working

fixing wrapper tests almost there

fixing wrapper tests ( last kind of tests)

unit job_list test remade

more tests and trying to renable run_members ( -rom)

another test fixing commit

Modifying one test

REmove tests that doesn't make sense now, added one more todo

lint

added TODO of missing scenarios for the checkpoints edges..

Added more tests

added tests

regression and integration working

fixing tests...

fixing regression tests

Added calculate_auto_splits in the brother autosubmitconfigparser (sqlite_joblist branch)

fix basic detect

WIP trying to remove chunks > EXPERIMENT.numchunks

Adding code to detect dynamic changes

more fix

fixes

fixing stuff... all kinda works kinda not

wrapper fix

wrapper working and not at the same time (know why)

wrappers fixes

fix call

monitor working

wrapper methods

very wip stuff

fix display issues

tests fixing bugs

intregation tests

updated test

some cleanup

Trying to save preview wrappers...

added some wrapper methods

fixing adding tests

Revising integration tests..

Revising tests.. ( 36 pending)

Revising tests.. ( 121 pending)

Regression tests working

unload working ( with issues in the last log)

log and job_data working again

Fix total jobs output, added count statments

Full Run working

Almost getting the load of the full list

Run working! ( kinda, dynamic load missing)

Fixing the different issues with special_conditions.. as I needed to adapt it anyway to the new format..

[wip]

[wip]

add if to setstate for log process

Load working, but run fails ( logical it has old code still)

Load almost working

Re-arranged.

Now we will only have self.graph.

Added property self.graph_dict , self.job_list that are dynamic views of self.graph

Renamed _job_list to job_list

Added upsert, save loads etc

save working

wip

added check_folders to make compatible older expids

renamed pkl -> db

Renamed tables

Author: dbeltrankyl

.github/workflows/docker.yaml

@@ -11,10 +11,10 @@ permissions:
 jobs:
   build-docker-images:
     name: Build Docker images
-    env:
-      DOCKERHUB_SECRET: ${{ secrets.DOCKERHUB_TOKEN }}
-    # NOTE: Secrets are not available for external contributors, so we skip the job.
-    if: ${{ secrets.DOCKERHUB_TOKEN }}
+    # NOTE: The Docker loging command apparently stopped working at some point
+    #       probably due to some security changes in either/both GitHub and
+    #       Dependabot. To solve this, we do not build docker images here.
+    if: ${{ github.actor != 'dependabot[bot]' }}
     runs-on: ubuntu-latest
     timeout-minutes: 15
 

VERSION

@@ -1 +1 @@
-4.1.15
+4.2

autosubmit/autosubmit.py

@@ -14,7 +14,6 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Autosubmit.  If not, see <http://www.gnu.org/licenses/>.
-
 try:
     # noinspection PyCompatibility
     from configparser import SafeConfigParser
@@ -223,3 +222,27 @@ def read():
 
         BasicConfig._update_config()
         return
+
+    @staticmethod
+    def generate_dirs():
+        """
+        Generates the directory structure needed for Autosubmit operation
+        """
+        Path(BasicConfig.DB_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.LOCAL_ROOT_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.STRUCTURES_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.GLOBAL_LOG_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.DEFAULT_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.JOBDATA_DIR).mkdir(parents=True, exist_ok=True)
+        Path(BasicConfig.HISTORICAL_LOG_DIR).mkdir(parents=True, exist_ok=True)
+
+        os.chmod(BasicConfig.DB_DIR, 0o770)
+        os.chmod(BasicConfig.LOCAL_ROOT_DIR, 0o770)
+        os.chmod(BasicConfig.STRUCTURES_DIR, 0o770)
+        os.chmod(BasicConfig.GLOBAL_LOG_DIR, 0o770)
+        os.chmod(BasicConfig.DEFAULT_OUTPUT_DIR, 0o770)
+        os.chmod(BasicConfig.JOBDATA_DIR, 0o770)
+        os.chmod(BasicConfig.HISTORICAL_LOG_DIR, 0o770)
+
+        # Local experiment directories are created when needed
+        return

autosubmit/config/basicconfig.py

@@ -12,5 +12,5 @@ MAIL:
   NOTIFICATIONS: False
   TO:
 STORAGE:
-  TYPE: "pkl"
+  TYPE: "db"
   COPY_REMOTE_LOGS: True

autosubmit/config/configcommon.py

@@ -12,5 +12,5 @@ MAIL:
   NOTIFICATIONS: False
   TO:
 STORAGE:
-  TYPE: "pkl"
+  TYPE: "db"
   COPY_REMOTE_LOGS: True

autosubmit/config/files/autosubmit-dummy.yml

@@ -24,21 +24,19 @@
 from pathlib import Path
 from typing import List, Optional, cast
 
-from autosubmit.config.basicconfig import BasicConfig
 from sqlalchemy import delete, select, Connection, insert, text, update, func
 from sqlalchemy.schema import CreateTable
 
+from autosubmit.config.basicconfig import BasicConfig
 from autosubmit.database import tables, session
 from autosubmit.log.log import Log, AutosubmitCritical
 
 Log.get_logger("Autosubmit")
 
-
 CURRENT_DATABASE_VERSION = 1
 TIMEOUT = 15
 
 
-
 def create_db(qry):
     """
     Creates a new database for autosubmit
@@ -96,22 +94,16 @@ def open_conn(check_version=True):
             version = row[0]
         except sqlite3.OperationalError:
             # If this exception is thrown it's because db_version does not exist.
-            # Database is from 2.x or 3.0 beta releases
-            try:
-                cursor.execute('SELECT type FROM experiment;')
-                # If type field exists, it's from 2.x
-                version = -1
-            except sqlite3.Error:
-                # If raises and error , it's from 3.0 beta releases
-                version = 0
+            version = 0
+            Log.warning("Database version table does not exist. Assuming version 0.")
 
         # If database version is not the expected, update database....
-        if version < CURRENT_DATABASE_VERSION:
+        if int(version) < int(CURRENT_DATABASE_VERSION):
             if not _update_database(version, cursor):
                 raise AutosubmitCritical('Database version does not match', 7001)
 
         # ... or ask for autosubmit upgrade
-        elif version > CURRENT_DATABASE_VERSION:
+        elif int(version) > int(CURRENT_DATABASE_VERSION):
             raise AutosubmitCritical('Database version is not compatible with this autosubmit version. '
                                      'Please execute pip install autosubmit --upgrade', 7002)
     return conn, cursor
@@ -159,9 +151,9 @@ def save_experiment(name: str, description: Optional[str], version: Optional[str
 
     try:
         result = queue.get(True, TIMEOUT)
-    except Exception:
-        raise AutosubmitCritical(f"The database process exceeded the timeout limit {TIMEOUT}s. "
-                                 f"Your experiment {name} couldn't be stored in the database.")
+    except BaseException:
+        raise AutosubmitCritical(
+            "The database process exceeded the timeout limit {0}s. Your experiment {1} couldn't be stored in the database.".format(TIMEOUT, name))
     finally:
         proc.terminate()
     return result
@@ -235,7 +227,7 @@ def get_autosubmit_version(expid):
     :type expid: str
     :return: If experiment exists returns the autosubmit version for it, if not returns None
     :rtype: str
-    """    
+    """
     fn = _get_autosubmit_version
     if BasicConfig.DATABASE_BACKEND == 'postgres':
         fn = _get_autosubmit_version_sqlalchemy
@@ -246,9 +238,9 @@ def get_autosubmit_version(expid):
 
     try:
         result = queue.get(True, TIMEOUT)
-    except Exception:
+    except Exception as e:
         raise AutosubmitCritical(f"The database process exceeded the timeout limit {TIMEOUT}s. "
-                                 f"Get experiment {expid} version failed to complete.")
+                                 f"Get experiment {expid} version failed to complete. {str(e)}")
     finally:
         proc.terminate()
     return result
@@ -297,7 +289,6 @@ def delete_experiment(experiment_id):
     fn = _delete_experiment
     if BasicConfig.DATABASE_BACKEND == 'postgres':
         fn = _delete_experiment_sqlalchemy
-
     queue = multiprocessing.Queue(1)
     proc = multiprocessing.Process(target=fn_wrapper, args=(fn, queue, experiment_id))
     proc.start()
@@ -408,10 +399,8 @@ def _check_experiment_exists(name, error_on_inexistence=True):
 def get_experiment_description(expid: str):
     if BasicConfig.DATABASE_BACKEND == 'postgres':
         return _get_experiment_description_sqlalchemy(expid)
-
-    check_db()
-
     try:
+        check_db()
         (conn, cursor) = open_conn()
     except DbException as e:
         raise AutosubmitCritical(
@@ -488,7 +477,7 @@ def _get_autosubmit_version(expid):
     # behaviour with the next sentence
     conn.text_factory = str
     cursor.execute('SELECT autosubmit_version FROM experiment WHERE name=:expid', {
-                   'expid': expid})
+        'expid': expid})
     row = cursor.fetchone()
     close_conn(conn, cursor)
     if row is None:
@@ -508,7 +497,7 @@ def _last_name_used(test=False, operational=False, evaluation=False):
     :type test: bool
     :return: last experiment identifier used, 'empty' if there is none
     :rtype: str
-    """    
+    """
     check_db()
 
     try:
@@ -734,7 +723,7 @@ def _get_experiment_description_sqlalchemy(expid) -> List[List[str]]:
 
 
 def _update_experiment_description_version_sqlalchemy(
-    name: str, description: Optional[str] = None, version: Optional[str] = None
+        name: str, description: Optional[str] = None, version: Optional[str] = None
 ) -> bool:
     # Conditional update statement
     if description is None and version is None:
@@ -808,7 +797,7 @@ def _last_name_used_sqlalchemy(test=False, operational=False, evaluation=False)
 
 def _delete_experiment_sqlalchemy(experiment_id: str) -> bool:
     if not _check_experiment_exists_sqlalchemy(
-        experiment_id, False
+            experiment_id, False
     ):  # Reference the no anti-lock version.
         return True
 
@@ -862,3 +851,13 @@ def get_connection_url(db_path: Optional['Path'] = None) -> str:
         raise ValueError('For SQLite databases you MUST provide a database file.')
 
     return f'sqlite:///{str(Path(db_path).resolve())}'
+
+
+def check_db_path(db_path: Optional[Path], must_exists: bool = True) -> bool:
+    """Check if the database path exists."""
+    if db_path and not db_path.exists() and must_exists:
+        raise ValueError(f'Database path not found {str(db_path)}!')
+    elif db_path and not db_path.exists() and not must_exists:
+        return False
+    else:
+        return True