Wrapper engine integration

[pablogoitia]

Hello, Autosubmit team 👋🏻

I am opening this issue in order to address possible questions about the wrapper engine integration process. The tools that we have chosen are Flux and Parsl, being Flux the first one to be integrated.

By now, what I am doing is to debug Autosubmit's create and run commands to find and understand the components involved in the wrapper creation and its later submission. The idea is, once I have taken my notes, to review how Flux's submission batch scripts work and start making some small adaptations to cover elemental cases (for example, running a vertical wrapper).

If possible, and if it does not consume much time, I would like to temporally implement Flux as another wrapping method, as AS_Thread or Srun, to facilitate debugging, integration, and testing for myself.

[pablogoitia]

Hi! While debugging I have found a function that seems to be called twice. It is the function to generate the job scripts plus the common wrapper script: _create_scripts(), at the job_packages.py file.

In line 188, the function submit_unthreaded() is called, which contains a call to _create_scripts() at the last line. Then, after returning, it is called again in line 190. In my case, as I am building a package for a wrapper, the _create_scripts() that is being called belongs to the JobPackageThread (in both calls).

Do you know if this behaviour is expected in any other scenario? Could the line be removed to avoid the overhead produced by writing files twice?

[dbeltrankyl]

Hello @pablogoitia

It is fixed on one of my recent PR's.. but it not reviewed yet.

https://github.com/BSC-ES/autosubmit/pull/2693/files

Maybe you can test and review it :)

[pablogoitia]

Here are some class diagrams I auto-generated from the source code to help me better understand the relationships between classes (inheritance, etc.). Note there are some spread ones.

Job module

Platform module

[pablogoitia]

18/11

In recent days, the construction of vertical and horizontal wrappers using the Flux method has been completed. Today, vertical-horizontal wrappers have also been implemented.

The experiment consists of a pair of job sections (SIM and OTHER), each with vertical dependencies. The wrapper builder object receives a list with two lists of jobs, one per section. However, I don't understand why the jobs in these lists arrive out of order and without respecting the dependencies. In the case of vertical wrappers, the list of jobs arrived correctly ordered. In this screenshot, you can see the list of lists expanded:

This leads to jobs being executed in the wrong order, as can be seen in the following screenshot of the experiment database table, sorted by start time (ignore the failed task because it succeeded):

The definition of workflow is as follows:

JOBS:
  SIM:
    FILE: SIM.sh
    DEPENDENCIES: SIM-1
    RUNNING: chunk
    PROCESSORS: 10
    WALLCLOCK: 00:01
  OTHER:
    FILE: SIM.sh
    DEPENDENCIES: OTHER-1
    RUNNING: chunk
    PROCESSORS: 12
    WALLCLOCK: 00:01

WRAPPERS:
  WRAPPER_SIM_OTHER:
    POLICY: "flexible"
    TYPE: "vertical-horizontal"
    JOBS_IN_WRAPPER: "SIM OTHER"
    METHOD: FLUX

Next steps:

• Determine the cause why AS is sending wrong job lists to the wrapper builder. Is it AS' fault or is it mine? Have I declared correctly the dependencies in the configuration file? Is the multi-section case natively supported for vertical-horizontal wrappers?
• To implement horizontal-vertical wrappers.
• To test error handling (retrials) behaviour of implemented wrapper builders.

[dbeltrankyl]

Determine the cause why AS is sending wrong job lists to the wrapper builder. Is it AS' fault or is it mine? Have I declared correctly the dependencies in the configuration file? Is the multi-section case natively supported for vertical-horizontal wrappers?

It is an AS fault.

To implement horizontal-vertical wrappers.

I think you will face similar issues

In #2619 it should be fixed ( not 100% sure)

Maybe you can just focus on implementing it, and once it's in master or as a test in a different branch, I can rebase for 4.2 and launch your tests there

[pablogoitia]

It is an AS fault.

Thanks for checking it, @dbeltrankyl!

I think you will face similar issues

No problem because I think for now I can just check whether the jobs are running in the same order that appears in the lists.

Maybe you can just focus on implementing it, and once it's in master or as a test in a different branch, I can rebase for 4.2 and launch your tests there

Sounds good to me, thanks Dani

[pablogoitia]

It doesn't need to be solved right now, but flux start is being called multiple times:

(flux) [bsc032177@glogin1 LOG_a004]$ cat a004_ASThread_SIM_17637201669780_10_6.err
load MINICONDA/24.1.2 (PATH, LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH,
CPLUS_INCLUDE_PATH, PKG_CONFIG_PATH, MANPATH) 
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh
flux-start: /home/bsc/bsc032177/.conda/envs/flux/libexec/flux/cmd/flux-broker /usr/bin/bash flux_runner.sh

With the following configuration:

#SBATCH -n 10
#SBATCH --ntasks-per-node=1