C48mx500_3DVarAOWCDA failure due to turning on waves

[RussTreadon-NOAA]

The following jobs failed in the 20250129 run of feature/stable-nightly

        2044 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 (Failed)
        2046 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 (Failed)
        2047 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
        2048 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
        2049 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
        2050 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)

@CoryMartin-NOAA identified the problem.

g-w PR #3249 added waves to the C48mx500_3DVarAOWCDA test. Wave files need to be staged to COMROOT in order for the forecast model to successfully run. Specifically, the waveinit job needs to be added to test/gw-ci/CMakeLists.txt

@@ -359,6 +359,7 @@ if (WORKFLOW_TESTS)
     set(YAML_PATH ${HOMEgfs}/ci/cases/pr/${pslot}.yaml)
     set(HALF_CYCLE_TASKS
       "gdas_stage_ic"
+      "gdas_waveinit"
       "gdas_fcst")
     set(FULL_CYCLE_TASKS
       "gdas_prepoceanobs"

This issue is opened to document this problem and its fix.

[RussTreadon-NOAA]

Work for this issue will be done in bugfix/waveinit

[RussTreadon-NOAA]

Add gdas_waveinit to test/gw-ci/CMakeLists.txt. Build GDASApp and run C48mx500_3DVarAOWCDA suite with the following results

Test project /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250129/global-workflow/sorc/gdas.cd/build
      Start 2087: test_gdasapp_C48mx500_3DVarAOWCDA
 1/10 Test #2087: test_gdasapp_C48mx500_3DVarAOWCDA ....................................   Passed    7.85 sec
      Start 2088: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800
 2/10 Test #2088: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800 .........   Passed   86.25 sec
      Start 2089: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800
 3/10 Test #2089: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800 .........   Passed   52.70 sec
      Start 2090: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800
 4/10 Test #2090: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 .............   Passed  452.87 sec
      Start 2091: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000
 5/10 Test #2091: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 .....   Passed  300.35 sec
      Start 2092: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000
 6/10 Test #2092: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 .......   Passed  113.63 sec
      Start 2093: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000
 7/10 Test #2093: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 ....   Passed   96.43 sec
      Start 2094: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000
 8/10 Test #2094: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 .....   Passed  115.44 sec
      Start 2095: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000
 9/10 Test #2095: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 ...   Passed   70.02 sec
      Start 2096: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000
10/10 Test #2096: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 ...   Passed   83.18 sec

100% tests passed, 0 tests failed out of 10

Label Time Summary:
gdasapp    = 1378.72 sec*proc (10 tests)
script     = 1378.72 sec*proc (10 tests)

Total Test time (real) = 1381.41 sec

Also run C48mx500_hybAOWCDA. All tests pass.

Test project /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250129/global-workflow/sorc/gdas.cd/build
      Start 2097: test_gdasapp_C48mx500_hybAOWCDA
 1/12 Test #2097: test_gdasapp_C48mx500_hybAOWCDA ....................................   Passed    8.98 sec
      Start 2098: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800
 2/12 Test #2098: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800 .........   Passed  309.12 sec
      Start 2099: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800
 3/12 Test #2099: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800 .............   Passed  429.73 sec
      Start 2100: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800
 4/12 Test #2100: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800 .....   Passed  250.11 sec
      Start 2101: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800
 5/12 Test #2101: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800 .........   Passed  446.13 sec
      Start 2102: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000
 6/12 Test #2102: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 .....   Passed  299.83 sec
      Start 2103: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000
 7/12 Test #2103: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 ...   Passed  166.44 sec
      Start 2104: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000
 8/12 Test #2104: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000 .......   Passed  120.31 sec
      Start 2105: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000
 9/12 Test #2105: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 ....   Passed   83.50 sec
      Start 2106: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000
10/12 Test #2106: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 .....   Passed  106.83 sec
      Start 2107: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000
11/12 Test #2107: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 ...   Passed   84.45 sec
      Start 2108: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000
12/12 Test #2108: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 ...   Passed   85.99 sec

100% tests passed, 0 tests failed out of 12

Label Time Summary:
gdasapp    = 2391.42 sec*proc (12 tests)
script     = 2391.42 sec*proc (12 tests)

Total Test time (real) = 2394.22 sec