Fix ocean restarts when cold starting with DOIAU=YES

[jswhit2]

Description

Fixes logic in forecast_predet.sh so that ocean restarts are written correctly when DOIAU="YES" for a cold-started experiment.

Note: there are currently no CI tests that cover this.

Resolves #3277

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

run a 3dvar coupled experiment.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[jswhit2]

@KateFriedman-NOAA I think this complements PR #3215, which handled the staging part of the problem

[KateFriedman-NOAA]

Awesome, thanks @jswhit2 ! Could you share the experiment setup and/or output/logs that you ran? It would be helpful for us to see the cold-start case you ran since we don't currently have any CI tests that cover this scenario. Thanks!

[jswhit2]

@KateFriedman-NOAA - I ran the test on gaea, but I can copy the yamls over to hera if you don't have access

[KateFriedman-NOAA]

@jswhit2 I have Gaea access and can view them there, please share the path, thanks!

[jswhit2]

on gaeac6 I ran

pslot="C192coupled3dvar_test" HPC_ACCOUNT="ira-da" RUNTESTS="/gpfs/f6/ira-da/scratch/Jeffrey.S.Whitaker/GWTESTS" ICSDIR_ROOT="/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/replayics/C192mx025" ./workflow/create_experiment.py --yaml /ncrc/home2/Jeffrey.S.Whitaker/global-workflow/yamls/C192mx025_3DVarAOWCDA.yaml

after copying /ncrc/home2/Jeffrey.S.Whitaker/global-workflow/gaeac6.yaml to workflow/hosts. After setting up the experiment, I edited config.base in the EXPDIR and set MAKE_PREPBUFR=NO

[KateFriedman-NOAA]

@jswhit2 Thanks for that info! Would you mind opening your IC set permissions? I am unable to access them during the staging job. Thanks!

Kate.Friedman@gaea61:/gpfs/f6/drsa-precip3/scratch/Kate.Friedman/RUNTESTS/EXPDIR/C192coupled3dvar_test> ll /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/replayics/C192mx025/
ls: cannot access '/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/replayics/C192mx025/': Permission denied

[jswhit2]

@KateFriedman-NOAA I just copied the replayics directory to /gpfs/f6/ira-da/world-shared

[KateFriedman-NOAA]

Great, thanks @jswhit2 !

[KateFriedman-NOAA]

Change looks good. I was able to replicate the error with the provided test case on Gaea-C6 and then the fcst job worked after implementing the fix in my testing clone. Thanks for this fix @jswhit2 !

[jswhit2]

@aerorahul does this look OK to you?

[aerorahul]

@aerorahul does this look OK to you?

Not sure.
When DOIAU == YES, warm_start has to be .true.. However, for a cold-start experiment (the variable is EXP_WARM_START = NO|false, DOIAU can still be YES.
I agree the logic here is confusing and likely compounded because we can have cold/warm start for atmosphere, but IAU increments are only useful during a warm start.

I'll look a little deeper and get back.

[aerorahul]

Since there are no CI tests that cover this, running CI on this PR is going to be pointless. I suggest we merge and open any issues if reports of failures follow.