CCE compile fails on xcf with "no space left on device"

[MichaelWhitall]

Hi folks,

I'm getting some odd UM compile failures on the xce/f:

[FAIL] ar: creating /var/spool/jtmp/8270384.xcf00.1RXFu3/peLWjDiUUD/libum-atmos.a
[FAIL] ar: /var/spool/jtmp/8270384.xcf00.1RXFu3/peLWjDiUUD/libum-atmos.a: No space left on device
[FAIL] link 1.1 ! um-atmos.exe <- um/src/control/top_level/um_main.F90
[FAIL] ! um-atmos.exe : update task failed

Does this mean that a disk required for the compilation is full? Where / what files should I delete to get my compiles to work? Note its odd as this was a UM vn12.1 rose stem run and most of the compiles succeeded; the only ones that failed in this way are CCE with OMP (safe, high and fast optimisation levels). The noOMP equivalents succeeded (as did ifort, gnu, etc).

Note I've put this question on the Yammer HPC group already, but just thought I'd ask here too in case anyone not looking at that might know?

Cheers!
Mike

[mcdalvi]

Hi,
This is most likely due to the tempsize (local disk space on node) exceeding the one reserved by the build task.

I 'think' the {{ set tmpdir_size("XXGB") in this specification is the one to change for XC40 builds:
rose-stem/site/meto/family-xc40.rc

Try after increasing tmpdir_size to 6 or 8 GB from the default 3GB

--
Mohit

[MichaelWhitall]

Awesome thanks Mohit!

I tried changing that line in rose-stem/site/meto/family-xc40.rc to:
{{ set_tmpdir_size("6GB") }}
I then saved the file and reran rose stem. But the compile failed in the same way and if I open up the job file for the failing compile it still contains:
#PBS -l tmpsize=3GB
Is there something else I need to do to make rose stem pick up the new settings from family-xc40.rc?

Cheers!
Mike

[MichaelWhitall]

Update: it turns out that the setting of tmpsize for XC40 builds in:
rose-stem/site/meto/family-xc40.rc
gets overwritten by the templating for UM_BUILD in:
rose-stem/site/meto/tasks-xc40.rc
which was setting tmpdir_size back to 3GB.

After amending the latter file to change tmpdir_size, the compile job actually gets a higher tmpsize as desired, and the compile succeeds :)

[mcdalvi]

Ah ok, that is good to know !
Thanks Mike !