HPC2020 issue with Autosubmit

[eferre1]

Hi @dbeltrankyl @kinow, I'm having an issue with Autosubmit, we've checked a bit with @dbeltrankyl offline but it doesn't seem to be any of the usual mistakes.

Description

When running an experiment in HPC2020 (ECMWF), using ecaccess as the connection mechanism, it stops without any clear error. It usually transfers some of the files (template and additional) and then stops. Probably during the ecaccess commands, and it might be receiving some output from it that it is not treated properly making the autosubmit process stop without dying (the process is still active in the machine, as well as the ecaccess). But checking manually the ecaccess commands I have not been able to reproduce the issue (it might also be due to too many simultaneous connections, which could be solved by using mput instead of put if that is the issue, but I have no idea).

• Short log:

9 of 9 jobs remaining (09:06)
No jobs to check for platform ECMWF-HPC2020
File /esarchive/autosubmit/t0lr/proj/auto-ecearth4/templates/synchronize.tmpl.sh exists
Calculating possible ready jobs for ECMWF-HPC2020
Section:SYNCHRONIZE can submit 20 jobs at this time
[WARNING] Script t0lr_SYNCHRONIZE has some empty variables. An empty value has substituted these variables

• ASLOG last lines:

9 of 9 jobs remaining (09:06)
2025-10-21 09:06:55,746 Sleep: 10
2025-10-21 09:06:55,746 Number of retrials: 0
2025-10-21 09:06:55,747 Checking jobs for platform=ECMWF-HPC2020
2025-10-21 09:06:55,748 No jobs to check for platform ECMWF-HPC2020
2025-10-21 09:06:55,750 Updating FAILED jobs
2025-10-21 09:06:55,818 Updating WAITING jobs
2025-10-21 09:06:55,819 Update finished
2025-10-21 09:06:55,826 Saving JobList: /esarchive/autosubmit/t0lr/pkl/job_list_t0lr.pkl.tmp
2025-10-21 09:06:55,854 JobList saved in /esarchive/autosubmit/t0lr/pkl/job_list_t0lr.pkl
2025-10-21 09:06:55,857 File /esarchive/autosubmit/t0lr/proj/auto-ecearth4/templates/synchronize.tmpl.sh exists
2025-10-21 09:06:55,859 Number of jobs available: 20
2025-10-21 09:06:55,860 Number of jobs ready: 1
2025-10-21 09:06:55,861 Jobs ready for ECMWF-HPC2020: 1
2025-10-21 09:06:55,862 Calculating possible ready jobs for ECMWF-HPC2020
2025-10-21 09:06:55,878 Section:SYNCHRONIZE can submit 20 jobs at this time
2025-10-21 09:06:55,883 Saving JobList: /esarchive/autosubmit/t0lr/pkl/job_list_t0lr.pkl.tmp
2025-10-21 09:06:55,904 JobList saved in /esarchive/autosubmit/t0lr/pkl/job_list_t0lr.pkl
2025-10-21 09:06:55,906 
Jobs ready for ECMWF-HPC2020: 1
[WARNING] 2025-10-21 09:06:55,935 Script t0lr_SYNCHRONIZE has some empty variables. An empty value has substituted these variables
2025-10-21 09:06:55,991 Creating Scripts
2025-10-21 09:06:56,172 Sending Files

• ps output:

[eferre1@bscesautosubmit03 ~]$ nohup autosubmit run t0lr >> ~/mn5_logs.txt &
[3] 330680
[eferre1@bscesautosubmit03 ~]$ ps
 330680 pts/0    00:00:02 autosubmit
 330688 pts/0    00:00:00 python3.12
 330689 pts/0    00:00:02 autosubmit log 
 330697 pts/0    00:00:00 ecaccess-file-p
 330797 pts/0    00:00:00 ps

• Autosubmit version: 4.1.15-conda
• Machine/VM/environment name (if applicable): bscesautosubmithub03 and 04
• Experiment ID (if applicable): t0lr
• Experiment tasks & log path if applicable):
  • /esarchive/autosubmit/t0lr/tmp/ASLOGS/20251021_114029_run.log (and others in this path)
  • pre-SYNCHRONIZE (files - template and additional - are partially moved to the hpc, but no submission of the job)

Reproducible Example

In theory, running a copy of the experiment t0lr (with dummyfied templates with sleeps) should already show the same behavior of Autosubmit, as it is happening even before starting the first job.

Expected Behaviour

The files should be transferred properly and then the job submission happen to the hpc slurm system.

[eferre1]

After some time left alone, it seems like it did something and then stopped again. The process is still running, and the job has been submitted in the HPC and run, but the log hasn't been recovered yet and the log doesn't show any extra information:

9 of 9 jobs remaining (12:45)
No jobs to check for platform ECMWF-HPC2020
File /esarchive/autosubmit/t0lr/proj/auto-ecearth4/templates/synchronize.tmpl.sh exists
Calculating possible ready jobs for ECMWF-HPC2020
Section:SYNCHRONIZE can submit 20 jobs at this time
[WARNING] Script t0lr_SYNCHRONIZE has some empty variables. An empty value has substituted these variables
ecmwf-hpc2020(log_recovery): Exiting.
500 Can't connect to boaccess.ecmwf.int:443
Invalid ECaccess Token
END failed--call queue aborted.
Job: t0lr_SYNCHRONIZE submitted with job_id: 23647846 and workflow commit: a0664db27b8020583376a296cc5b40bbcbec5e0c
t0lr_SYNCHRONIZE submitted
Process ECMWF-HPC2020_log_recovery started with pid 346387


9 of 9 jobs remaining (14:09)
Checking 1 jobs for platform ECMWF-HPC2020
ecmwf-hpc2020(log_recovery): Starting...
Job t0lr_SYNCHRONIZE is QUEUING
ecmwf-hpc2020(log_recovery): successfully connected.


9 of 9 jobs remaining (14:10)
Checking 1 jobs for platform ECMWF-HPC2020
Job t0lr_SYNCHRONIZE is QUEUING


9 of 9 jobs remaining (14:10)
Checking 1 jobs for platform ECMWF-HPC2020
ecmwf-hpc2020(log_recovery): Exiting.

[dbeltrankyl]

Hello @eferre1

ecmwf-hpc2020(log_recovery): Exiting.

That's because we have one process per platform aside of the main process.

The blocked one is the main one, so after 5 mins without news from the main one, these platform process end. But not sure what is causing that the submission ends and then it blocks again

[dbeltrankyl]

Maybe you can try with the DISABLE_RECOVERY_THREADS setting?

ex:

PLATFORMS:
  ECMWF:
    ...
    DISABLE_RECOVERY_THREADS: TRUE

[eferre1]

Maybe you can try with the DISABLE_RECOVERY_THREADS setting?

ex:

PLATFORMS:
ECMWF:
...
DISABLE_RECOVERY_THREADS: TRUE

trying this it shows a different error then:

Autosubmit is running with v4.1.15
Checking the connection to all platforms in use
[ECMWF-HPC2020] Correct user privileges for host hpc-login
[ECMWF-HPC2020] Connection successful to host hpc-login
[ERROR] Arguments provided: Namespace(version=False, logfile='DEBUG', logconsole='WARNING', command='run', expid='t0lr', notransitive=False, update_version=False, start_time=None, start_after=None, run_only_members=None, profile=False)
[ERROR] This is the experiment: <t0lr> which had an issue with the command: <run> and it is currently using the Autosubmit Version: 4.1.15.
 [CRITICAL] Unexpected error: 'NoneType' object has no attribute 'set'.
 Please report it to Autosubmit Developers through Git
More info at https://autosubmit.readthedocs.io/en/master/troubleshooting/error-codes.html

[dbeltrankyl]

Oh looks like that setting is broken ( cc: @VindeeR as you may need it for the PBS platform, if there is 2FA)

[kinow]

I've added X11 + 2FA to our SSH test container. I've already written some X11 tests and they are working, and my next batch of tests is for 2FA, so maybe some of those tests will help. FWIW Miyabi & Param Rudra also appear to have 2FA.

[eferre1]

Hi, just removing the DISABLE_RECOVERY_THREADS line and relaunching today, and it seems to be working fine for the moment. Maybe it was some ECMWF issue with ecaccess? I'll check this days if it keeps working or not, but if it does, then you might be able to close this issue as it could be that is unrelated to Autosubmit. I'll let you know, thanks for the help.

[kinow]

Not a problem. Yesterday I added some 2FA tests, and everything seemed to work fine, although I'm using google-authenticator, and I think ECMWF uses something else (another utility I forgot the name now). Keep us posted, please, @eferre1 . Thanks!

[dbeltrankyl]

Perfect @eferre1

But in any case, I would leave this open in case we can mitigate the issue somehow as if it was working manually it should work with Autosubmit as well. I'll move it to another milestone tho

Just a question, yesterday the job was submitted? (without autosubmit noticing it) or nothing was submitted? (Trying to see if the issue is that autosubmit doesn't get an ID, or that the command hangs)