lotss hr

Group members: Leah, Neal

Monday

Goal for the day: get split-directions working with toil.

Progress:

• Neal has gotten toil-cwl-runner to work on galahad. Doing this can be quite cluster-specific, and dependent on python setup/environment (toil-cwl-runner is pip-installed). Current kludges needed:

Use python3.12, will need urllib3 downgrade by pip3 to 1.26.2; conda install nodejs

Needs the _ddsel directories manually deleting between runs

Needs the vlbi-cwl.sif and node_alpine.sif copying back into the main singularity directory from the ../pull directory

Needs everything in real directories for the software directories, soft links do not work

• Leah has been using Jurjen's alternative workflow split-directions-toil.cwl after resolving a couple issues:

  apptainer appears to have a bug where it will accept APPTAINERENV_PREPEND_PATH but it is not used correctly; the env variable SINGULARITYENV_PREPEND_PATH does work with apptainer and should be used instead.

  --preserve-entire-environment doesn't work -- use --preserve-environment VAR1 VAR2 VAR3 etc for all of the APPTAINERENV variables

  see issue here: https://git.astron.nl/RD/VLBI-cwl/-/issues/27 for minor changes that need to happen until VLBI-cwl is updated

• There is now a python script which can be incorporated into the monitor script which splits the image_catalogue.csv file into chunks of 10 and submits one split-directions-toil job per chunk; this is done as an array job and is being tested with a limit of 2 array jobs at a time.