Skip to content

Add capability to create aerosol AOD analysis products in GRIB2 format via UPP #3889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: develop
Choose a base branch
from
Draft

Add capability to create aerosol AOD analysis products in GRIB2 format via UPP #3889

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3da2797
add method to process aerosol analysis to grib2
Jul 21, 2025
3333a7e
fix coding norms
Jul 21, 2025
63e3233
add upp processing to job script
Jul 21, 2025
6c67cf1
add atmos history path
Jul 21, 2025
5e4a341
stage files to run UPP
Jul 21, 2025
33daaf2
update job resource
Jul 22, 2025
6e6ea86
update namelist template with 'nasa_on' parameter
Jul 22, 2025
3757474
set running environment
Jul 22, 2025
2f23293
save the output grib2 file to COMOUT_CHEM_ANALYSIS
Jul 22, 2025
8b26fb8
norm fix
Jul 22, 2025
d27ab35
Merge branch 'develop' into feature/aero_postprocess
ypwang19 Jul 22, 2025
0323953
remove unnecessary lines
Jul 23, 2025
c5333e4
Merge branch 'NOAA-EMC:develop' into feature/aero_postprocess
ypwang19 Jul 29, 2025
51fd3b3
reset to analysis time in input file, and save output to ROTDIR
Jul 30, 2025
91c3338
save grib2 file to ROTDIR
Jul 30, 2025
4332bf5
update gdasapp hash
Jul 30, 2025
3e5e9a2
Merge remote-tracking branch 'upstream/develop' into feature/aero_pos…
Jul 30, 2025
5b34437
Merge branch 'develop' into feature/aero_postprocess
ypwang19 Aug 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev/parm/config/gcafs/config.aeroanl.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ export CRTM_FIX_YAML="${PARMgfs}/gdas/aero_crtm_coeff.yaml.j2"
export JEDI_FIX_YAML="${PARMgfs}/gdas/aero_jedi_fix.yaml.j2"

export JEDI_CONFIG_YAML="${PARMgfs}/gdas/aeroanl_jedi_config.yaml.j2"
export UPP_CONFIG_YAML="${PARMgfs}/gdas/aeroanl_upp_config.yaml.j2"

export AERO_STAGE_VARIATIONAL_TMPL="${PARMgfs}/gdas/aero_stage_variational.yaml.j2"
export AERO_FINALIZE_VARIATIONAL_TMPL="${PARMgfs}/gdas/aero_finalize_variational.yaml.j2"
Expand Down
4 changes: 2 additions & 2 deletions dev/parm/config/gcafs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,10 +516,10 @@ case ${step} in
echo "FATAL ERROR: Resources not defined for job ${step} at resolution ${CASE}"
exit 4
esac
ntasks=1
ntasks=120
threads_per_task=1
tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
memory="13072M"
memory=${mem_node_max}
;;

"marineanlinit")
Expand Down
1 change: 1 addition & 0 deletions dev/parm/config/gfs/config.com
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,5 +110,6 @@ declare -rx COM_CHEM_HISTORY_TMPL=${COM_BASE}'/model/chem/history'
declare -rx COM_CHEM_ANALYSIS_TMPL=${COM_BASE}'/analysis/chem'
declare -rx COM_CHEM_BMAT_TMPL=${COM_CHEM_ANALYSIS_TMPL}'/bmatrix'
declare -rx COM_CHEM_ANLMON_TMPL=${COM_BASE}'/products/chem/anlmon'
declare -rx COM_CHEM_GRIB_TMPL=${COM_BASE}'/products/chem/grib2'

declare -rx COM_MED_RESTART_TMPL=${COM_BASE}'/model/med/restart'
7 changes: 7 additions & 0 deletions env/GAEAC5.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,13 @@ case ${step} in
export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANL}"
;;
"aeroanlfinal")

export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANLFINAL}"
;;
"aeroanlgenb")

export NTHREADS_AEROANLGENB=${NTHREADSmax}
Expand Down
7 changes: 7 additions & 0 deletions env/GAEAC6.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,13 @@ case ${step} in
export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANL}"
;;
"aeroanlfinal")

export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANLFINAL}"
;;
"aeroanlgenb")

export NTHREADS_AEROANLGENB=${NTHREADSmax}
Expand Down
7 changes: 7 additions & 0 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,13 @@ elif [[ "${step}" = "aeroanlvar" ]]; then
export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANL}"

elif [[ "${step}" = "aeroanlfinal" ]]; then

export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANLFINAL}"

elif [[ "${step}" = "aeroanlgenb" ]]; then

export NTHREADS_AEROANLGENB=${NTHREADSmax}
Expand Down
7 changes: 7 additions & 0 deletions env/HERCULES.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,13 @@ case ${step} in

export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANL}"
;;
"aeroanlfinal")

export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANLFINAL}"
;;
"aeroanlgenb")

Expand Down
7 changes: 7 additions & 0 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@ elif [[ "${step}" = "aeroanlvar" ]]; then
export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANL}"

elif [[ "${step}" = "aeroanlfinal" ]]; then

export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default} --cpus-per-task=${NTHREADS_AEROANLFINAL}"

elif [[ "${step}" = "aeroanlgenb" ]]; then

export NTHREADS_AEROANLGENB=${NTHREADSmax}
Expand Down
7 changes: 7 additions & 0 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,13 @@ elif [[ "${step}" = "aeroanlvar" ]]; then
export NTHREADS_AEROANL=${NTHREADSmax}
export APRUN_AEROANL="${APRUN_default}"

elif [[ "${step}" = "aeroanlfinal" ]]; then

export APRUNCFP="${launcher} -np \$ncmd ${mpmd_opt}"

export NTHREADS_AEROANLFINAL=${NTHREADSmax}
export APRUN_AEROANLFINAL="${APRUN_default}"

elif [[ "${step}" = "aeroanlgenb" ]]; then

export NTHREADS_AEROANLGENB=${NTHREADSmax}
Expand Down
3 changes: 2 additions & 1 deletion jobs/JGLOBAL_AERO_ANALYSIS_FINALIZE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "aeroanlfinal" -c "base aeroanl aeroan
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \
COMOUT_CHEM_ANALYSIS:COM_CHEM_ANALYSIS_TMPL \
COMOUT_CONF:COM_CONF_TMPL \
COMOUT_ATMOS_RESTART:COM_ATMOS_RESTART_TMPL
COMOUT_ATMOS_RESTART:COM_ATMOS_RESTART_TMPL \
COMOUT_CHEM_GRIB:COM_CHEM_GRIB_TMPL

###############################################################
# Run relevant script
Expand Down
2 changes: 2 additions & 0 deletions jobs/JGLOBAL_AERO_ANALYSIS_INITIALIZE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@ GDUMP="${GDUMP/gcafs/gcdas}"
# Generate COM variables from templates
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \
COMIN_OBS:COM_OBS_TMPL \
COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL \
COMOUT_CHEM_ANALYSIS:COM_CHEM_ANALYSIS_TMPL

RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} declare_from_tmpl -rx \
COMIN_ATMOS_RESTART_PREV:COM_ATMOS_RESTART_TMPL \
COMIN_ATMOS_HISTORY_PREV:COM_ATMOS_HISTORY_TMPL \
COMIN_CHEM_BMAT_PREV:COM_CHEM_BMAT_TMPL \
COMIN_CHEM_ANALYSIS_PREV:COM_CHEM_ANALYSIS_TMPL

Expand Down
5 changes: 5 additions & 0 deletions parm/gdas/aero_finalize_variational.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ mkdir:
- "{{ COMOUT_CHEM_ANALYSIS }}"
- "{{ COMOUT_ATMOS_RESTART }}"
- "{{ COMOUT_CONF }}"
- "{{ COMOUT_CHEM_GRIB }}"
copy:
## copy variational YAML to ROTDIR
- ["{{ DATA }}/aeroanlvar.yaml", "{{ COMOUT_CONF }}/{{ APREFIX }}aerovar.yaml"]
Expand All @@ -23,3 +24,7 @@ copy:
{% for tile in range(1,ntiles+1) %}
- ["{{ DATA }}/anl/{{ bkgtime | to_fv3time }}.fv_tracer.res.tile{{ tile }}.nc", "{{ COMOUT_ATMOS_RESTART }}/{{ bkgtime | to_fv3time }}.aeroanl_fv_tracer.res.tile{{ tile }}.nc"]
{% endfor %}
## copy grib2 analysis
- ["{{ DATA }}/upp/GFSPRS.GrbF00", "{{ COMOUT_CHEM_GRIB }}/{{ APREFIX }}aeroanl.grib2"]
- ["{{ DATA }}/upp/GFSPRS.GrbF00.idx", "{{ COMOUT_CHEM_GRIB }}/{{ APREFIX }}aeroanl.grib2.idx"]

9 changes: 9 additions & 0 deletions parm/gdas/aero_stage_variational.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ mkdir:
- "{{ DATA }}/diags"
- "{{ DATA }}/berror"
- "{{ DATA }}/bkg"
- "{{ DATA }}/upp"
copy:
######################################
## copy backgrounds
Expand Down Expand Up @@ -48,3 +49,11 @@ copy:
### copy diffusion files
- ["{{ COMIN_CHEM_BMAT_PREV }}/{{ GPREFIX }}aero_diffusion_hz.nc", "{{ DATA }}/berror/diffusion_hz.nc"]
- ["{{ COMIN_CHEM_BMAT_PREV }}/{{ GPREFIX }}aero_diffusion_vt.nc", "{{ DATA }}/berror/diffusion_vt.nc"]

######################################
### copy backgrounds from COMIN_ATMOS_HISTORY_PREV
- ["{{ COMIN_ATMOS_HISTORY_PREV }}/{{ GPREFIX }}atmf006.nc", "{{ DATA }}/upp/atm_{{ current_cycle | to_YMDH }}0000.nc"]
- ["{{ COMIN_ATMOS_HISTORY_PREV }}/{{ GPREFIX }}sfcf006.nc", "{{ DATA }}/upp/sfc_{{ current_cycle | to_YMDH }}0000.nc"]
### copy atmos increment from COMIN_ATMOS_ANALYSIS
- ["{{ COMIN_ATMOS_ANALYSIS }}/{{ APREFIX }}atminc.nc", "{{ DATA }}/upp/{{ APREFIX }}atminc.nc"]

60 changes: 60 additions & 0 deletions parm/gdas/aeroanl_upp_config.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
upp:
rundir: '{{ DATA }}/upp'
exe_src: '{{ EXECgfs }}/upp.x'
mpi_cmd: '{{ APRUN_AEROANLFINAL }}'
jcb_base_yaml: '{{ PARMgfs }}/gdas/aero/jcb-base.yaml.j2'
config:
grib_version: "grib2"
ioform: "netcdfpara"
po: [1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.7,0.4,0.2,0.1,0.07,0.04,0.02,0.01]
rdaod: False
nasa_on: True
fix_data:
mkdir:
- "{{ DATA }}/upp"
copy:
- ["{{ PARMgfs }}/post/gefs/postxconfig-NT-gefs-gcafs.txt", "{{ DATA }}/upp/postxconfig-NT.txt"]
#- ["{{ PARMgfs }}/post/postxconfig-NT-gfs-chem.txt", "{{ DATA }}/upp/postxconfig-NT.txt"]
- ["{{ PARMgfs }}/post/params_grib2_tbl_new", "{{ DATA }}/upp/params_grib2_tbl_new"]
- ["{{ PARMgfs }}/post/nam_micro_lookup.dat", "{{ DATA }}/upp/eta_micro_lookup.dat"]
- ["{{ EXECgfs }}/upp.x", "{{ DATA }}/upp/"]
- ["{{ PARMgfs }}/post/itag.jinja", "{{ DATA }}/upp/"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_DUST_nasa.dat", "{{ DATA }}/upp/optics_luts_DUST_nasa.dat"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_NITR_nasa.dat", "{{ DATA }}/upp/optics_luts_NITR_nasa.dat"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_SALT_nasa.dat", "{{ DATA }}/upp/optics_luts_SALT_nasa.dat"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_SOOT_nasa.dat", "{{ DATA }}/upp/optics_luts_SOOT_nasa.dat"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_SUSO_nasa.dat", "{{ DATA }}/upp/optics_luts_SUSO_nasa.dat"]
- ["{{ HOMEgfs }}/sorc/upp.fd/fix/chem/optics_luts_WASO_nasa.dat", "{{ DATA }}/upp/optics_luts_WASO_nasa.dat"]

analysis:
config:
nasa_on: True
data_in:
none:

atmincvars: [
['ugrd', 'u_inc'],
['vgrd', 'v_inc'],
['dpres', 'delp_inc'],
['delz', 'delz_inc'],
['tmp', 'T_inc'],
['spfh', 'sphum_inc'],
['rwmr', 'liq_wat_inc'],
['o3mr', 'o3mr_inc'],
['icmr', 'icmr_inc']]

aeroincvars: [
['dust1', 'mass_fraction_of_dust001_in_air'],
['dust2', 'mass_fraction_of_dust002_in_air'],
['dust3', 'mass_fraction_of_dust003_in_air'],
['dust4', 'mass_fraction_of_dust004_in_air'],
['dust5', 'mass_fraction_of_dust005_in_air'],
['seas1', 'mass_fraction_of_sea_salt001_in_air'],
['seas2', 'mass_fraction_of_sea_salt002_in_air'],
['seas3', 'mass_fraction_of_sea_salt003_in_air'],
['seas4', 'mass_fraction_of_sea_salt004_in_air'],
['so4', 'mass_fraction_of_sulfate_in_air'],
['oc1', 'mass_fraction_of_hydrophobic_organic_carbon_in_air'],
['oc2', 'mass_fraction_of_hydrophilic_organic_carbon_in_air'],
['bc1', 'mass_fraction_of_hydrophobic_black_carbon_in_air'],
['bc2', 'mass_fraction_of_hydrophilic_black_carbon_in_air']]
1 change: 1 addition & 0 deletions parm/post/itag.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@
kpo = {{ po | length }},
po = {{ po | join(', ') }},
rdaod = {{ rdaod | to_f90bool }}
nasa_on = {{ nasa_on | to_f90bool }}
/

1 change: 1 addition & 0 deletions parm/post/upp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ upp:
ioform: "netcdfpara"
po: [1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.7,0.4,0.2,0.1,0.07,0.04,0.02,0.01]
rdaod: False
nasa_on: False
fix_data:
mkdir:
- "{{ DATA }}"
Expand Down
4 changes: 4 additions & 0 deletions scripts/exglobal_aero_analysis_finalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

from wxflow import Logger, cast_strdict_as_dtypedict
from pygfs.task.aero_analysis import AerosolAnalysis
from pygfs.task.upp import UPP


# Initialize root logger
Expand All @@ -22,5 +23,8 @@
# Instantiate the aerosol analysis task
AeroAnl = AerosolAnalysis(config)

# Process aerosal products through UPP
AeroAnl.upp_anlproc()

# Finalize JEDI aerosol variational analysis
AeroAnl.finalize()
2 changes: 1 addition & 1 deletion sorc/gdas.cd
Open in desktop
Submodule gdas.cd updated 15 files
+0 −1 .gitignore
+0 −4 .gitmodules
+0 −1 bundle/CMakeLists.txt
+1 −0 modulefiles/GDAS/gaeac5.intel.lua
+1 −0 modulefiles/GDAS/gaeac6.intel.lua
+1 −0 modulefiles/GDAS/hera.intel.lua
+1 −0 modulefiles/GDAS/hercules.gnu.lua
+1 −0 modulefiles/GDAS/hercules.intel.lua
+1 −0 modulefiles/GDAS/noaacloud.intel.lua
+1 −0 modulefiles/GDAS/orion.intel.lua
+1 −0 modulefiles/GDAS/ursa.intel.lua
+1 −0 modulefiles/GDAS/wcoss2.intel.lua
+1 −1 parm/jcb-gdas
+1 −0 parm/soca/letkf/letkf.yaml.j2
+0 −1 sorc/femps
Loading