Skip to content

Commit

Permalink
Expand parallel job submissions (NOAA-EMC#28)
Browse files Browse the repository at this point in the history
Expand the parallel job submissions by submitting minimization and
conventional obs plots as separate jobs using `ush/splitPlotYaml.py`.
All GFS plot jobs now to completion on hera in ~45 min using this
parallel strategy.

The `conTime.yaml` and `conVert.yaml` files have been generalized to use
DATATYPES, and use of 'all' for datatypes is now supported using the
parm/observation_datatypes.yaml master list.

The change to `ush/diagnosticAverager.py` is a pycode fix and is
unrelated to the parallel job changes.

Note that this has not been tested yet on wcoss2. I figured to do that
as a separate issue, and likely next.

Closes NOAA-EMC#25
  • Loading branch information
EdwardSafford-NOAA authored Apr 26, 2024
1 parent 68cccf7 commit 461137b
Show file tree
Hide file tree
Showing 8 changed files with 552 additions and 184 deletions.
10 changes: 6 additions & 4 deletions parm/gfs/conTime.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,8 @@ datasets:

regions: 1

datatypes: {{DATATYPES}}

groups:
- name: ges

Expand Down Expand Up @@ -278,8 +280,8 @@ datasets:

regions: 1

datatypes: &datatypes ['q120_0', 'q130_0']

datatypes: {{DATATYPES}}
groups:
- name: anl

Expand Down Expand Up @@ -452,7 +454,7 @@ graphics:
# 4 plot ConMon time series
# -------------------------
- batch figure:
datatypes: *datatypes
datatypes: {{DATATYPES}}
variables: ['count1_allev']

figure:
Expand Down Expand Up @@ -576,7 +578,7 @@ graphics:
label: 'final outloop'

- batch figure:
datatypes: *datatypes
datatypes: {{DATATYPES}}
variables: ['bias1_allev']

figure:
Expand Down
6 changes: 4 additions & 2 deletions parm/gfs/conVert.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ datasets:

regions: 1

datatypes: {{DATATYPES}}

groups:
- name: ges

Expand All @@ -37,7 +39,7 @@ datasets:

regions: 1

datatypes: &datatypes ['q120_0', 'q130_0']
datatypes: {{DATATYPES}}

groups:
- name: anl
Expand All @@ -59,7 +61,7 @@ graphics:
# 4 plot ConMon time series
# -------------------------
- batch figure:
datatypes: *datatypes
datatypes: {{DATATYPES}}
variables: ['count1', 'count2', 'count_vqc1', 'count3']

figure:
Expand Down
Loading

0 comments on commit 461137b

Please sign in to comment.