HPC* variables randomly missing

[kinow]

Description

From https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/1125 (link protected, but will summarize here to non-BSC'ers).

@ainagaya gathered three cases where variables were not replaced by AS. In all of these cases, eventually the variables were replaced, without anything obvious done by users that could have caused this.

• Autosubmit version: 4.1.14 (probably 4.1.15)
• Machine/VM/environment name (if applicable): destine
• Experiment ID (if applicable): t062, a2j1, t05s

Reproducible Example

This is the tricky part.

Expected Behaviour

The behavior is consistent (either works, or doesn't)

[dbeltrankyl]

Hello @kinow thanks for opening the issue, I forgot to do so

Maybe relevant comment:

https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/1118#note_376242

[kinow]

Thanks Dani!

[dbeltrankyl]

I was debugging a2fx

How I debugged it:

• new expid
• mkdir -p $expid/proj/git_project
• scp paste the ../git_project/conf and ../git_project/templates/ folder to the new expid
• Change minimal.yml expid key to the new_expid, and I am also using marenostrum5 instead of Lumi
• Do autosubmit create $expid
• ( ensure that you have a ~/platforms.yml file )
• Check the templates used
• for all the templates:
  • Remove all except the "header" section of the
  • add ( except in the ones running locally)

if [ -z "$HPCROOTDIR" ]; then
  echo "Error: HPCROOTDIR variable is not set or is empty."
  exit 1
fi

• autosubmit run $expid

What I found is that the sim_icon.sh didn't have the HPCROOTDIR=%HPCROOTDIR% defined ( which was the actual issue Leo had ( confirm it if not ) , but I did not find the random missing

Cc: @Lerriola

[kinow]

@dmcclainbsc reported having the same problem in the testing suite 2 weeks ago, on August 26. Internal links:

• https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/merge_requests/789#note_388622
• https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/1220#note_388592

Experiment t063, with 4.0.14. Thanks for letting us know, Damien!

[kinow]

And I think this issue may have bitten @mandresm 4 days ago, in DestinE experiment t05t. Internal link: https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/1160#note_391984

[mandresm]

And I think this issue may have bitten @mandresm 4 days ago, in DestinE experiment t05t. Internal link: https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/1160#note_391984

I have just cleaned the experiment with the testing_suite --clean command (sorry), so there is not much left to check. I am running it again, I'll report if the issue persists

[dmcclainbsc]

Hello! I believe this is another example:

/appl/AS/AUTOSUBMIT_DATA/t096/tmp/LOG_t096/t096_19900101_fc0_2_SIM.20250918121211.err

HPCROOTDIR was not substituted which led to failure.

[kinow]

Hello! I believe this is another example:

/appl/AS/AUTOSUBMIT_DATA/t096/tmp/LOG_t096/t096_19900101_fc0_2_SIM.20250918121211.err

HPCROOTDIR was not substituted which led to failure.

@dmcclainbsc I had a look at your experiment, and cloned it into a new experiment in my local laptop. A quick create + inspect, showed me that the HPCROOTDIRappears to be created in that template file as

set -xuve

# HEADER

HPCROOTDIR=${1:-/gpfs/scratch/ehpc01/bsc984535/a002}
....

Different than yours,

HPCROOTDIR=${1:-}
+ HPCROOTDIR=

The SIM template appears to be correct,

HPCROOTDIR=${1:-%HPCROOTDIR%}

Will try to debug it some other time... my plan is to create a branch in Autosubmit where the config parser code (now part of Autosubmit) prints a bit more. Not doing quite what's expected for #2505 , but at least being more transparent on what config files are being read. And then read the code to understand where/how that could happen.

Thanks

[kinow]

This happened again for @mbatllem in (internal link). Expid a2nk, AS version 4.1.14, DestinE.

[kinow]

This happened again for @igonzal-bsc, twice in internal link, both times with 4.1.14.

[Lerriola]

This happened again for me @Lerriola, in internal link, using AS version 4.1.14, DestinE.