Get ILAMB to work on NIRD

[rosiealice]

Noting that this is in our to-do list.

Much of the discussion of this is currently in
NorESMhub/noresm3_dev_simulations#48

[rosiealice]

I had got ILAMB working on Caspeer using Keith's instructions ( I will post these below)

Moving Keith's output onto NIRD and trying to analyse that did not work.
Moving my FATES-SP_45 case output onto Casper did not work
Trying to analyse a CLM5 simulation I had done on casper did not work.

Next things to try:
Use a simulation from Adrianna or Keith?

[rosiealice]

KEITH CASPER ILAMB NOTES

—————————————————————————————————————————————————
ONE-TIME

Add the following to your default shell, e.g., .tcshrc, and then source your, e.g., .tcshrc:
setenv ILAMB_ROOT /glade/campaign/cesm/community/lmwg/diag/ILAMB

cd /glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb

NOTE that if you need to update your ilamb_casper_lmwg environment, then do
conda env remove -n ilamb_casper_lmwg

module load conda
mamba conda env create -f ilamb.yml (for some this works without adding mamba in front)
conda activate ilamb_casper_lmwg
pip install ./
python -c "import ILAMB; print(ILAMB.version)" (should result in "2.7")
cd bin
./ilamb-setup

mkdir /glade/work/$USER/diagnostics_ilamb
mkdir /glade/work/$USER/diagnostics_ilamb/runs

---

Workaround (one-time) for a permission problem with generated_version.py

Once a user has installed ILAMB as above there is one file that changes ownership to the person
installing ILAMB, e.g., if oleson is the user, then:
-rw-rw----+ 1 oleson cesm 239 Feb 15 11:10 generated_version.py
To avoid problems for the next user not having permissions to write to this file, do this as a
workaround:
chmod a+rwx /glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb/build/lib/ILAMB/generated_version.py

---

EACH TIME YOU SETUP A CONFRONTATION

mkdir /glade/work/$USER/diagnostics_ilamb/runs/$CASE
cd /glade/work/$USER/diagnostics_ilamb/runs/$CASE
cp /glade/campaign/cesm/community/lmwg/diag/ILAMB/SCRIPTS/ilamb_diagnostics_casper .
cp /glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb/src/ILAMB/data/ilamb_nohoff_final_CLM.cfg . (if a
BGC confrontation)
cp /glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb/src/ILAMB/data/ilamb_nohoff_final_CLM_SP.cfg . (if a
SP confrontation)
cp /glade/campaign/cesm/community/lmwg/diag/ILAMB/SCRIPTS/model_setup.txt .

EDIT ilamb_diagnostics_casper and model_setup.txt accordingly for your own run
See https://www.ilamb.org/doc/ilamb_run.html for instructions regarding model_setup.txt

qsub ./ilamb_diagnostics_casper

[rosiealice]

The following are instructions for the 'vanilla' setup of ILAMB on Casper.

Log in to casper.hpc.ucar.edu
in BASH shell.

ONE TIME
Add the following to your default shell, e.g., .tcshrc, and then source your, e.g., .tcshrc:

setenv ILAMB_ROOT /glade/campaign/cesm/community/lmwg/diag/ILAMB

then

cd /glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb

module load conda
conda env create -f ilamb.yml #(for some this works without adding mamba in front)
conda activate ilamb_casper_lmwg
pip install ./
python -c "import ILAMB; print(ILAMB.__version__)"                        (should result in "2.7")
cd bin
./ilamb-setup

EACH TIME

cd /glade/work/rfisher/diagnostics_ilamb/runs/ctsm51_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist2

#PBS -N ilamb_diagnostics_casper
#PBS -q casper
#PBS -l select=1:ncpus=16:mpiprocs=16:mem=100G
#PBS -l walltime=12:00:00
#PBS -A P93300041
#PBS -j oe
#PBS -k eod
#PBS -m abe
#PBS -M YOUR_EMAIL_ADDRESS
export ilamb_code=/glade/campaign/cesm/community/lmwg/diag/ILAMB/CODE/ilamb
export diagdir=/glade/work/$USER/diagnostics_ilamb/runs
export caseid=ctsm51_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist2
export casedir=$diagdir/$caseid
export MPI_UNBUFFERED_STDIO=true
export TMPDIR=$TMPDIR
mkdir -p $TMPDIR

module load conda
conda activate ilamb_casper_lmwg

export MPLBACKEND=Agg
export ILAMB_ROOT=/glade/campaign/cesm/community/lmwg/diag/ILAMB
export builddir=_build_newscore_CLM50CTSM51BGC

mpiexec $ilamb_code/bin/ilamb-run  --config ./ilamb_nohoff_final_CLM.cfg --build_dir $casedir/$builddir  --regions global --model_setup $casedir/model_setup.txt --filter .clm2.h0.

#Sometimes this seems to not work with mpiexec and you need to run without

n.b. that the standard cfg file has an issue with the albedo calculation, so switched to ilamb_config_simple_CLM.cfg

[rosiealice]

test1. Tried moving keith's output to the scratch dir.

/glade/derecho/scratch/rfisher/clm50_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist/lnd/hist

and running:

$ilamb_code/bin/ilamb-run  --config ./ilamb_simple_CLM.cfg --build_dir $casedir/$builddir  --regions global --model_setup $casedir/model_setup_test1.txt --filter .clm2.h0.

Where model_setup_test1.txt is

# Model Name    , Location of Files                                                                                     ,  Shift From,  Shift To
CLM50_move      , /glade/derecho/scratch/rfisher/clm50_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist/lnd/hist/
CLM50           , /glade/campaign/cgd/tss/common/Land_Only_Simulations/CTSM51_DEV/CLM50_CTSM51_LAND_ONLY_RELEASE/clm50_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist/lnd/hist/
CTSM51          , /glade/campaign/cgd/tss/common/Land_Only_Simulations/CTSM51_DEV/CLM50_CTSM51_LAND_ONLY_RELEASE/ctsm51_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_hist/lnd/hist/

This (simply moving the output directories around) results in a crash where the moved output does not run and the standard output does.

[rosiealice]

Debugging with @maritsandstad illustrates a few strange things. Notably that the isTemporalproperty can change from true to false for different time slice files in the history output.

Next pass is to write the instructions for this to run on NIRD to test if we can recreate this error there.