Job not submitted if file in FILE does not exist

[franra9]

Description

Hi beloved Autosubmit developers! I have what it seems to be a bug for you 🐛

What happens is what follows:

I have a job DN that has 2 files under JOB.FILE. I noticed that if some of the files in there do not exist, the job is never submitted, getting stuck in READY step.

I see that in the logs there is a message: File /appl/AS/AUTOSUBMIT_DATA/a2lr/proj/git_project/templates/dn.sh exists that concerns the first file, but no message from the second one. (test.sh).

Saving JobList: /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl.tmp
JobList saved in /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl
File /appl/AS/AUTOSUBMIT_DATA/a2lr/proj/git_project/templates/dn.sh exists
Number of jobs available: 20
Number of jobs ready: 0
Calculating possible ready jobs for LUMI
Saving JobList: /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl.tmp
JobList saved in /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl

Jobs ready for LUMI: 0
Number of jobs available: 20
Number of jobs ready: 0
Calculating possible ready jobs for local
Saving JobList: /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl.tmp
JobList saved in /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl

Jobs ready for local: 0
Number of jobs available: 1
Number of jobs ready: 1
Jobs ready for LUMI-LOGIN: 1
Calculating possible ready jobs for LUMI-LOGIN
Section:DN can submit 1 jobs at this time
Saving JobList: /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl.tmp
JobList saved in /appl/AS/AUTOSUBMIT_DATA/a2lr/pkl/job_list_a2lr.pkl

• Autosubmit version: 4.1.14
• Machine/VM/environment name (if applicable): lumi_vm
• Experiment ID (if applicable): a2lr
• Experiment tasks & log path if applicable):
  • e.g. a2lr_*_fc0_*DN*
  • e.g. /appl/AS/AUTOSUBMIT_DATA/a2lr/tmp/ASLOGS
  • debug output file in /appl/AS/AUTOSUBMIT_DATA/a2lr/nohup_debug.out

@dbeltrankyl is already aware of this, see internal comment here.

Reproducible Example

I copied the experiment into a fresh one a2lx (as a2lr I am still using it for my workflow devs).

I think that copying this file, creating and running you should find the same error.

Expected Behaviour

The DN job to be submitted and or getting a message concerning the rest of files under the FILE key, if this is what I think the bug comes from.

Thanks in advance!

Francesc

[dbeltrankyl]

Hello @franra9

Thanks for opening the issue:

Some questions:

1. Is it blocking? If the file exists, it works, no?
2. The issue is that it doesn't warn you. Or do you want a different behaviour?
3. When all files exists, it works, no?

The expected behaviour is to check, before submitting, all files that don't exist and raise a Critical error. For both the additional files and the main script

Do you want to tune this behavior for some reason? Maybe, in the yaml config, we could add the possibility of marking a file as optional or mandatory.

[dbeltrankyl]

The expected behaviour is to check, before submitting, all files that don't exist and raise a Critical error. For both the additional files and the main script

Regarding this, the issue is determining when to do it. Because some files could not exist if the other job were not run. ( Example: Auto-ecearth, the sim.sh won't exist until the Local_setup is run). So I don't know how to do this check before starting the run, it must be checked in real-time or with the autosubmit inspect

[franra9]

hi @dbeltrankyl

Is it blocking? If the file exists, it works, no?

It is not blocking. If the files exist, it works, yes

The issue is that it doesn't warn you. Or do you want a different behaviour?

The issue is that if one of the files do not exist, AS did not warn me about this and never stopped. The experiment was still alive, with a "READY" job (DN) that was never submitted.

When all files exists, it works, no?

Yes

Regarding this, the issue is determining when to do it. Because some files could not exist if the other job were not run. ( Example: Auto-ecearth, the sim.sh won't exist until the Local_setup is run). So I don't know how to do this check before starting the run, it must be checked in real-time or with the autosubmit inspect

I understand this behaviour, this is not the case of the current issue though.

I hope I clarified everything you needed. :)

[dbeltrankyl]

Keeping it for 4.1.16 as it can easily waste time

[ainagaya]

This happened today to Marvin (can't find him) with an additional file. He was using another version of the a submodule that contains a file that we use as additional file, and therefore, the additional file didn't exist.
Just FYI

[dbeltrankyl]

Hello!

I'll do this because I had to touch a related function in the v4.2.0 branch, due to a bug with the monitor -cw , so I'm also adding the additional files check and will cherry-pick it in a PR for 4.1.16 ( cc: @kinow )