Update to soca that significantly improves memory/runtime

[shlyaeva]

Description

This update goes with JCSDA-internal/soca#1130 that refactors data storage in soca, reduces memory footprint and more carefully handles halos and intermediate variables in soca states and increments.

The update in the gdas-utils here changes how we create an increment with a subset of variables, since assigning increments with different variables is now prohibited.

I ran both low- and high- res hybrid experiments and confirmed that all the tasks (bmat, envar, letkf, recenter, chkpt) behave as expected, and all the differences are explained (and are a sign of better code in the branch). The differences in the logs are: removal of some intermediate variables that were not used in some states; fixes of scrambled data in some intermediate variables. The netcdf output produced by all of those tasks is the same as with develop with the exception of ice.ssh_recentering_error files that are not used (it’s the ice files) but on develop have garbage in the ice variables.

Differences in runtime and memory (hera/intel, 0.25 degree case):
EnVar:

dev:
        Runtime: 413.73 sec, Memory: total: 685.13 Gb
soca branch:
Runtime: 372.81 sec, Memory: total: 470.56 Gb

LETKF:

dev:
        Runtime: 2305.72 sec, Memory: total: 4333.59 Gb
soca branch:
Runtime: 2264.41 sec, Memory: total: 1910.04 Gb

The soca tag will need to change once soca PR is merged.

Companion PRs

JCSDA-internal/soca#1130

Automated CI tests to run in Global Workflow

• atm_jjob
• C96C48_ufs_hybatmDA
• C96C48_hybatmaerosnowDA
• C48mx500_3DVarAOWCDA
• C48mx500_hybAOWCDA
• C96C48_hybatmDA

[shlyaeva]

@guillaumevernieres, @RussTreadon-NOAA: it would be great to run automated tests here for my sanity check. This change will be required at least for the nightly branch once JCSDA-internal/soca#1130 is merged (likely very soon, it's only waiting on my approval). I don't know what the best way to handle this is: set this for merge into the nightly branch?

[RussTreadon-NOAA]

@shlyaeva , once JCSDA-internal/soca#1130 is merged, the stable-nightly script will pull the updated soca hash into a working copy of feature/stable-nightly. The stable-nightly script runs GDASApp ctests (including g-w tests) on Hera.

Do we need bring the updated utils/soca/gdas_ens_handler.h from this PR into feature/stable-nightly for all ctests to pass?

[shlyaeva]

Do we need bring the updated utils/soca/gdas_ens_handler.h from this PR into feature/stable-nightly for all ctests to pass?

Yes, once JCSDA-internal/soca#1130 is merged.

[RussTreadon-NOAA]

OK. Let's get JCSDA-internal/soca#1130 merged. We can work on the remaining steps after the updated soca hash is in feature/stable-nightly.

[shlyaeva]

@RussTreadon-NOAA soca PR just got merged.

[RussTreadon-NOAA]

The change to utils/soca/gdas_ens_handler.h from this PR has been merged into feature/stable-nightly. This was done in GDASApp PR #1507

The soca hash updated in this PR was not retained. soca PR #1130 has been merged into soca develop. The soca hash we want to bring into GDASApp is 7675ed14 . This hash will be brought into feature/stable-nightly via the stable-nightly update.