Add code changes for the implementation of the low-resolution B #1441

guillaumevernieres · 2025-01-13T21:19:03Z

As the title says.

Dependencies

Configuration for the low res marine B jcb-gdas#66
Add capability to run diff resolutions for marine anl and background global-workflow#3238
Update of the static files in glopara: Update to the marine JEDI fix files global-workflow#3239

Work done in this PR

Interp to low-resolution:

variance partitioning
preparation of the perturbations for the envar
hybrid weights
...

Testing

Visual check showing the variance explained by the steric height (30 members)

left is native resolution, right is on the 1/2 deg grid.

"Science" note: The explained variance above is almost the opposite to what I would have expected, and what we saw in the previous offline ensemble, or even what we see in tendencies. Check if there's a bug in the explained variance calculation.

Automated CI tests to run in Global Workflow

emcbot · 2025-01-13T22:55:25Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Mon Jan 13 21:38:37 UTC 2025 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jan 13 22:32:01 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Mon Jan 13 22:55:24 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 53

shlyaeva

Did not test, but I reviewed all the code changes, they look good to me.

emcbot · 2025-01-15T22:47:44Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Wed Jan 15 21:37:00 UTC 2025 on hfe09
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Jan 15 22:26:38 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Wed Jan 15 22:47:43 UTC 2025
Tests: 92% tests passed, 4 tests failed out of 53
	2042 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2043 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2054 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2055 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1441/global-workflow/sorc/gdas.cd/build/log.ctest

emcbot · 2025-02-04T01:39:02Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Mon Feb  3 23:46:06 UTC 2025 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb  4 00:34:26 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb  4 01:39:01 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 54

emcbot · 2025-02-04T01:55:29Z

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Mon Feb  3 12:19:57 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Feb  3 13:00:49 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Mon Feb  3 19:55:29 CST 2025
Tests: 98% tests passed, 1 tests failed out of 54
	2039 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800 (Timeout)
Tests: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1441/global-workflow/sorc/gdas.cd/build/log.ctest

guillaumevernieres · 2025-02-04T13:54:54Z

Weird, it looks like it used the wrong commit of the g-w branch. I merged develop before resubmitting the ci but that;s not what was cloned here: /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1441/global-workflow

emcbot · 2025-02-04T15:39:23Z

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Tue Feb  4 08:04:59 AM CST 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb  4 09:11:58 AM CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb  4 09:39:22 AM CST 2025
Tests: 100% tests passed, 0 tests failed out of 54

emcbot · 2025-02-04T16:09:05Z

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Tue Feb  4 08:05:32 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb  4 08:46:00 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb  4 10:09:04 CST 2025
Tests: 100% tests passed, 0 tests failed out of 54

RussTreadon-NOAA

GDASApp ctests including C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA pass on Hera, Hercules, and Orion.

Approve.

AndrewEichmann-NOAA

looks ok

shlyaeva · 2025-02-04T21:41:19Z

Here are the changes that I needed to do for the recenter task to succeed with a zero exit status:

diff --git a/parm/soca/berror/soca_ensrecenter.yaml b/parm/soca/berror/soca_ensrecenter.yaml
index b700601..b7d8e20 100644
--- a/parm/soca/berror/soca_ensrecenter.yaml
+++ b/parm/soca/berror/soca_ensrecenter.yaml
@@ -60,5 +60,5 @@ output increment:
   date: '{{ MARINE_WINDOW_END_ISO }}'
   exp: trash
   type: incr
-  output file: 'ocn.recenter.incr.%mem%.nc'
+  output file: 'recenter.incr.%mem%.nc'
   pattern: '%mem%'

and

diff --git a/utils/soca/gdas_ens_handler.h b/utils/soca/gdas_ens_handler.h
index 741055c..dd5fe9c 100644
--- a/utils/soca/gdas_ens_handler.h
+++ b/utils/soca/gdas_ens_handler.h
@@ -169,7 +171,30 @@ namespace gdasapp {
           postProcIncr.setToZero(incr);

           // Save the increments used to initialize the ensemble forecast
-          result = postProcIncr.save(mom6_incr, i+1);
+          result = postProcIncr.save(mom6_incr, i+1, {"ocn"});

It fixes two issues: missing ice file (that we don't use in this usecase anyway) and wrong ocean file name.
I didn't test beyond this task.

…3238) This PR mostly includes configuration changes to allow the use a B-matrix at a defined resolution that may or may not be the same as the background. - Addition of an interpolation step in the B-matrix job - version update for the fix files - **the `gdas.cd` update should be the correct one now.** Resolves #3230 Resolves NOAA-EMC/GDASApp/issues/1375 # Dependencies - NOAA-EMC/GDASApp/pull/1441 - NOAA-EMC/jcb-gdas/pull/66 --------- Co-authored-by: Anna Shlyaeva <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]> Co-authored-by: Rahul Mahajan <[email protected]>

* develop: added original date (#1570) Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569) Fixing window_begin and end in yaml for redating ADT (#1565) Save LETKF mean analysis and mean increment for verification (#1566) Update JEDI hashes (20250324) (#1567) Removes obsolete json files from marine bufr2ioda (#1564) Add hybrid 4denvar as a phase3 validation configuration (#1552) Bugfix/ocn ecen (#1550) Bugfix SSS obs for LETKF (#1560) Bugfix ocean ensemble recentering (#1558) apply ci path patch only when running g-w ctests (#1554) Update JEDI hashes (20250317) (#1531) add option to create_experiment.py in create_exp.sh (#1546) Update jcb hashes and support initial IODA stats capability (#1547) Fix GW-CI CTest dependencies (#1534) Converts prepoceanobs to use JCB-style obs list and config files (#1527) JEDI-based ensemble recentering and analysis calculation (#1488) tools for phase3 validation (#1526) Update JEDI hashes (20250306) (#1520) New jcb-gdas # and insitu obs prep bug fix (#1523) Marine in situ obs update (SSS, moorings, drifters) (#1511) Save marine background and analysis spread (#1517) Minor build and install updates and ctest cleanup (#1512) correct logic for upper loop limit in chem_diagb (#1516) Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504) Update JEDI hashes (20250225) (#1475) Add a depth dependent e-fold scale (#1494) Update jcb and da-utils hashes to support land DA. (#1509) Add options to inflate the ensemble while recentering (#1501) Add components to use jcb yamls for marine letkf job (#1486) Add GNSSRO bufr2ioda and json files (#1448) Removal of the marine verification tools (#1499) update logic to identify MSU machine (#1498) IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472) Add sea ice recentering (#1478) marine vrfy bugfix (#1483) add LD_LIBRARY_PATH patch for wcoss2 build (#1484) Bugfix for ensemble recenter + add a test (#1477) Add code changes for the implementation of the low-resolution B (#1441) update spack-stack path in Hera modulefiles (#1471) fixed glider output filename (#1469) add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470) update jcb-gdas hash 20250128 (#1466) Time series of omb stats and html doc (#1463) Update marine vrfy HTML codes (#1464) Automate the verify tools to submit from start pdy to end pdy (#1458) Update an offline marine vrfy tool (#1447) Update JEDI hashes (20250127) (#1460)

…(#3238) This PR mostly includes configuration changes to allow the use a B-matrix at a defined resolution that may or may not be the same as the background. - Addition of an interpolation step in the B-matrix job - version update for the fix files - **the `gdas.cd` update should be the correct one now.** Resolves #3230 Resolves /issues/1375 # Dependencies - /pull/1441 - NOAA-EMC/jcb-gdas/pull/66 --------- Co-authored-by: Anna Shlyaeva <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]> Co-authored-by: Rahul Mahajan <[email protected]>

guillaumevernieres added 3 commits January 11, 2025 12:00

optional diagb at lowres

004aef2

works, needs more testing so still wip

68d8150

Merge branch 'develop' into feature/lowres-b

595cfaf

guillaumevernieres requested a review from shlyaeva January 13, 2025 21:20

guillaumevernieres self-assigned this Jan 13, 2025

guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Jan 13, 2025

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jan 13, 2025

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jan 13, 2025

shlyaeva previously approved these changes Jan 14, 2025

View reviewed changes

change res for hyb. weights

5ece15f

guillaumevernieres dismissed shlyaeva’s stale review via 5ece15f January 14, 2025 21:47

chg res for ice pert

7c2a3c8

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Passed Automated testing with global-workflow successful on Hera labels Jan 15, 2025

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jan 15, 2025

guillaumevernieres added 2 commits January 15, 2025 16:35

configurable domains to be saved

cfb239a

norms

e303dec

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jan 15, 2025

guillaumevernieres added 4 commits January 15, 2025 20:12

optional local weights

0252442

...

2f12db4

Merge branch 'develop' into feature/lowres-b

d7f2db7

point to companion jcb-gdas

b85abad

guillaumevernieres added hercules-GW-RT Queue for automated testing with global-workflow on Hercules and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Jan 16, 2025

emcbot added the hera-GW-RT-Running Automated testing with global-workflow running on Hera label Feb 3, 2025

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 4, 2025

emcbot added hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 4, 2025

RussTreadon-NOAA added hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion and removed hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Feb 4, 2025

emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 4, 2025

emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 4, 2025

RussTreadon-NOAA self-requested a review February 4, 2025 16:10

RussTreadon-NOAA approved these changes Feb 4, 2025

View reviewed changes

AndrewEichmann-NOAA approved these changes Feb 4, 2025

View reviewed changes

guillaumevernieres merged commit fa09326 into develop Feb 4, 2025
35 checks passed

guillaumevernieres deleted the feature/lowres-b branch February 4, 2025 16:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add code changes for the implementation of the low-resolution B #1441

Add code changes for the implementation of the low-resolution B #1441

Uh oh!

guillaumevernieres commented Jan 13, 2025 •

edited

Loading

Uh oh!

emcbot commented Jan 13, 2025

Uh oh!

shlyaeva left a comment

Uh oh!

emcbot commented Jan 15, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

guillaumevernieres commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

RussTreadon-NOAA left a comment

Uh oh!

AndrewEichmann-NOAA left a comment

Uh oh!

Uh oh!

shlyaeva commented Feb 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add code changes for the implementation of the low-resolution B #1441

Add code changes for the implementation of the low-resolution B #1441

Uh oh!

Conversation

guillaumevernieres commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependencies

Work done in this PR

Testing

Automated CI tests to run in Global Workflow

Uh oh!

emcbot commented Jan 13, 2025

Uh oh!

shlyaeva left a comment

Choose a reason for hiding this comment

Uh oh!

emcbot commented Jan 15, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

guillaumevernieres commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

emcbot commented Feb 4, 2025

Uh oh!

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shlyaeva commented Feb 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

guillaumevernieres commented Jan 13, 2025 •

edited

Loading