Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 #3309

Open
wants to merge 19 commits into
base: develop
Choose a base branch
from
Open

Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 #3309

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
87db086
Updates for cycling C1152 ATM
CatherineThomas-NOAA Dec 19, 2024
7c3a155
Merge branch 'NOAA-EMC:develop' into c1152_cyc
CatherineThomas-NOAA Dec 21, 2024
9c1b856
Merge branch 'NOAA-EMC:develop' into c1152_cyc
CatherineThomas-NOAA Jan 6, 2025
c7b3010
Update gsi_utils hash for C1152 cycling
CatherineThomas-NOAA Jan 6, 2025
a161bf2
Update gsi_utils for C1152 cycling
CatherineThomas-NOAA Jan 8, 2025
e90db09
Merge branch 'NOAA-EMC:develop' into c1152_cyc
CatherineThomas-NOAA Jan 8, 2025
a11f4d2
aerosol c1152 cycling changes
Jan 24, 2025
d9c1c5b
more resource changes
Jan 24, 2025
6db13da
extend walltime for aerovar
Jan 27, 2025
27dcbbb
Merge branch 'develop' into feature/aero1152
Jan 27, 2025
c29244f
Merge branch 'NOAA-EMC:develop' into feature/aero1152
CoryMartin-NOAA Jan 28, 2025
c9ef39b
update config.aeroanlgenb
CoryMartin-NOAA Jan 28, 2025
c0f96d1
Merge branch 'NOAA-EMC:develop' into feature/aero1152
CoryMartin-NOAA Jan 31, 2025
4e48fa2
Merge branch 'NOAA-EMC:develop' into feature/aero1152
CoryMartin-NOAA Feb 6, 2025
dd9823c
update GDAS hash
CoryMartin-NOAA Feb 7, 2025
0f4dd95
appease the lint gods
CoryMartin-NOAA Feb 7, 2025
dc36516
address russ's comments
CoryMartin-NOAA Feb 7, 2025
9a601a0
module setup line
CoryMartin-NOAA Feb 7, 2025
2529457
update gdas hash
Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ step=$1
export launcher="mpiexec -l"
export mpmd_opt="--cpu-bind verbose,core cfp"

# Add path to GDASApp libraries
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOMEgfs}/sorc/gdas.cd/build/lib"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"

# Calculate common resource variables
# Check first if the dependent variables are set
if [[ -n "${ntasks:-}" && -n "${max_tasks_per_node:-}" && -n "${tasks_per_node:-}" ]]; then
Expand Down
9 changes: 4 additions & 5 deletions parm/config/gfs/config.aeroanlgenb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,16 @@ export JEDI_CONFIG_YAML="${PARMgfs}/gdas/aero_bmat_jedi_config.yaml.j2"
export JCB_BASE_YAML="${PARMgfs}/gdas/aero/jcb-base.yaml.j2"
export AERO_BMATRIX_STAGE_TMPL="${PARMgfs}/gdas/aero_stage_bmatrix_bkg.yaml.j2"
export AERO_BMATRIX_FINALIZE_TMPL="${PARMgfs}/gdas/aero_finalize_bmatrix_bkg.yaml.j2"
export aero_diffusion_iter=10
export aero_diffusion_horiz_len=2500e3
export aero_diffusion_fixed_val=1.0
export aero_diffusion_iter=200
export aero_diffusion_horiz_len=300e3
export aero_diffusion_fixed_val=20.0
export npx_clim_b=97
export npy_clim_b=97
export aero_diagb_weight=0.9
export aero_staticb_rescaling_factor=2.0
export aero_diagb_rescale=20.0
export aero_diagb_n_halo=4
export aero_diagb_n_neighbors=16
export aero_diagb_smooth_horiz_iter=0
export aero_diagb_smooth_horiz_iter=200
export aero_diagb_smooth_vert_iter=0

echo "END: config.aeroanlgenb"
12 changes: 6 additions & 6 deletions parm/config/gfs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,7 @@ case ${step} in

export layout_x
export layout_y
walltime="00:10:00"
walltime="00:30:00"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should walltime be CASE-specific?

ntasks=1
threads_per_task=1
tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
Expand Down Expand Up @@ -454,9 +454,9 @@ case ${step} in
export layout_x
export layout_y

walltime="00:30:00"
walltime="00:45:00"
ntasks=$(( layout_x * layout_y * 6 ))
threads_per_task=1
threads_per_task=5
Comment on lines +457 to +459
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, should threads and walltime be specific to the resolution?

tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
export is_exclusive=True
;;
Expand Down Expand Up @@ -498,11 +498,11 @@ case ${step} in


"aeroanlfinal")
walltime="00:10:00"
walltime="00:30:00"
ntasks=1
threads_per_task=1
tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
memory="3072M"
memory="13072M"
;;

"marineanlinit")
Expand Down Expand Up @@ -903,7 +903,7 @@ case ${step} in
;;
"C768" | "C1152")
# Not valid resolutions for ensembles
declare -x "walltime_gdas"="00:50:00"
declare -x "walltime_gdas"="01:20:00"
declare -x "walltime_gfs"="06:00:00"
;;
*)
Expand Down
1 change: 1 addition & 0 deletions parm/config/gfs/config.resources.WCOSS2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
case ${step} in
"prep")
export is_exclusive=True
export tasks_per_node=5
export memory="480GB"
;;

Expand Down
2 changes: 1 addition & 1 deletion sorc/gdas.cd
Open in desktop
Submodule gdas.cd updated 31 files
+1 −1 modulefiles/EVA/hera.lua
+1 −1 modulefiles/GDAS/hera.intel.lua
+2 −2 parm/aero/jcb-base.yaml.j2
+1 −1 parm/jcb-gdas
+1 −1 parm/soca/berror/soca_ensrecenter.yaml
+5 −11 parm/soca/fms/input.nml.j2
+5 −0 parm/soca/soca_fix_stage_025.yaml.j2
+5 −0 parm/soca/soca_fix_stage_100.yaml.j2
+5 −0 parm/soca/soca_fix_stage_500.yaml.j2
+1 −1 sorc/oops
+1 −1 sorc/saber
+1 −1 sorc/soca
+1 −1 sorc/ufo
+5 −0 test/gw-ci/CMakeLists.txt
+6 −1 ush/ioda/bufr2ioda/marine/b2i/bufr2ioda_insitu_profile_glider.py
+1 −1 ush/soca/soca_vrfy.py
+38 −0 utils/soca/fig_gallery/README.md
+53 −16 utils/soca/fig_gallery/gdassoca_obsstats.py
+23 −24 utils/soca/fig_gallery/gdassoca_obsstats_template.html
+19 −53 utils/soca/fig_gallery/marine_vrfy_display/default.css
+586 −205 utils/soca/fig_gallery/marine_vrfy_display/index_vrfy_marine.html
+0 −45 utils/soca/fig_gallery/run_marine_analysis_vrfy_manual.job
+71 −0 utils/soca/fig_gallery/run_vrfy.py
+16 −0 utils/soca/fig_gallery/vrfy_config.yaml
+58 −0 utils/soca/fig_gallery/vrfy_jobcard.sh.j2
+92 −66 utils/soca/fig_gallery/vrfy_script.py
+29 −18 utils/soca/gdas_ens_handler.h
+8 −1 utils/soca/gdas_incr_handler.h
+75 −43 utils/soca/gdas_postprocincr.h
+10 −1 utils/soca/gdas_soca_diagb.h
+20 −8 utils/soca/gdas_socahybridweights.h
2 changes: 1 addition & 1 deletion ush/module-setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ elif [[ ${MACHINE_ID} = s4* ]] ; then
elif [[ ${MACHINE_ID} = wcoss2 ]]; then
# We are on WCOSS2
# Ignore default modules of the same version lower in the search path (req'd by spack-stack)
#export LMOD_TMOD_FIND_FIRST=yes #TODO: Uncomment this when using spack-stack
export LMOD_TMOD_FIND_FIRST=yes
CoryMartin-NOAA marked this conversation as resolved.
Show resolved Hide resolved
module reset

elif [[ ${MACHINE_ID} = cheyenne* ]] ; then
Expand Down