Improve messaging to display clear warning when missing snogrb file

[KateFriedman-NOAA]

Description

This PR resolves bugzilla 1373 (Display clear warning when missing snogrb file). The following changes are included in this PR to resolve the bugzilla:

• add "INFO" message when final snogrb file is decided in script
• add "FATAL ERROR" message when decided-upon snogrb file is missing, exit if so

Also included in this PR are a related bug fix uncovered during work to resolve bugzilla and needed COMIN/COMOUT updates, which are part of a larger initiative. Those changes are:

• fix bug with res variable setting in scripts/exgdas_enkf_sfc.sh
• COMIN/COMOUT updates for the sfcanl jobs

Refs #2451
Refs #1041
Resolves #2500

Type of change

Bugzilla resolution and bug fix

Change characteristics

Error handling and messaging updates in related workflow scripts.

How has this been tested?

Ran C96_atm3DVar CI test on Hercules.

[CoryMartin-NOAA]

one suggestion in several places just to make things clearer in the logs

[GeorgeGayno-NOAA]

After looking at the script logic more closely, I think there is a way to avoid a FATAL error. Note this logic. New snow is only created once per day (around 00z), but there are snow files every six hours. In most cases, these are simply copies of the 00z file with a new file name. Snow is only applied in the cycling if there is new snow. Hence, the check here on the internal date/time stamp. If FNSNOA and FNSNOG contain the same time stamp, the snow update is shut off by setting FNSNOW="" and CYCLVARS to 99999.

if [[ $(${WGRIB} -4yr "${FNSNOA}" 2>/dev/null | grep -i snowc | awk -F: '{print $3}' | awk -F= '{print $2}') -le \
  $(${WGRIB} -4yr "${snoprv}" 2>/dev/null | grep -i snowc | awk -F: '{print $3}' | awk -F= '{print $2}') ]] ; then
  export FNSNOA=" "
  export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"
else

If FNSNOA or FNSNOG are missing, the same settings could be done and a WARNING message be printed. The script will continue, and the cycling will not update snow.

The snow data is created by an upstream job which has a logic to always populate FNSNOA and FNSNOG files. So I can't envision this ever happening. But if it does, we can continue running and have the SPA investigate the warning.

[KateFriedman-NOAA]

If I understand the discussion above, we don't want to FATAL ERROR and exit if either snogrb file is missing, instead we want to print a WARNING but continue on. Is that correct? If so:

export FNSNOA=${FNSNOA:-${COMIN_OBS}/${OPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOA exists, if not use t1534 version
[[ ! -f ${FNSNOA} ]] && export FNSNOA="${COMIN_OBS}/${OPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOA} ]]; then
  echo "WARNING: Current cycle snow file ${FNSNOA} is not available. Job will continue but not update snow."
else
  echo "INFO: Current cycle snow file is ${FNSNOA}"
fi
export FNSNOG=${FNSNOG:-${COMIN_OBS_PREV}/${GPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOG exists, if not use t1534 version
[[ ! -f ${FNSNOG} ]] && export FNSNOG="${COMIN_OBS_PREV}/${GPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOG} ]]; then
  echo "WARNING: Previous cycle snow file ${FNSNOG} is not available. Job will continue but not update snow."
else
  echo "INFO: Previous cycle snow file is ${FNSNOG}"
fi

[GeorgeGayno-NOAA]

If I understand the discussion above, we don't want to FATAL ERROR and exit if either snogrb file is missing, instead we want to print a WARNING but continue on. Is that correct? If so:

export FNSNOA=${FNSNOA:-${COMIN_OBS}/${OPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOA exists, if not use t1534 version
[[ ! -f ${FNSNOA} ]] && export FNSNOA="${COMIN_OBS}/${OPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOA} ]]; then
  echo "WARNING: Current cycle snow file ${FNSNOA} is not available. Job will continue but not update snow."
else
  echo "INFO: Current cycle snow file is ${FNSNOA}"
fi
export FNSNOG=${FNSNOG:-${COMIN_OBS_PREV}/${GPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOG exists, if not use t1534 version
[[ ! -f ${FNSNOG} ]] && export FNSNOG="${COMIN_OBS_PREV}/${GPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOG} ]]; then
  echo "WARNING: Previous cycle snow file ${FNSNOG} is not available. Job will continue but not update snow."
else
  echo "INFO: Previous cycle snow file is ${FNSNOG}"
fi

Yes, print a warning, but these variables also must be set:

 export FNSNOA=" "
 export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"

[KateFriedman-NOAA]

export FNSNOA=" "
export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"

Gotcha. Set those for both the missing current cycle and previous cycle snogrb files? Should these if-blocks get merged for any reason?

Like:

export FNSNOA=${FNSNOA:-${COMIN_OBS}/${OPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOA exists, if not use t1534 version
[[ ! -f ${FNSNOA} ]] && export FNSNOA="${COMIN_OBS}/${OPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOA} ]]; then
  echo "WARNING: Current cycle snow file ${FNSNOA} is not available. Job will continue but not update snow."
  export FNSNOA=" "
  export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"
else
  echo "INFO: Current cycle snow file is ${FNSNOA}"
fi
export FNSNOG=${FNSNOG:-${COMIN_OBS_PREV}/${GPREFIX}snogrb_t${JCAP_CASE}.${LONB_CASE}.${LATB_CASE}}
# Check if resolution specific FNSNOG exists, if not use t1534 version
[[ ! -f ${FNSNOG} ]] && export FNSNOG="${COMIN_OBS_PREV}/${GPREFIX}snogrb_t1534.3072.1536"
if [[ ! -f ${FNSNOG} ]]; then
  echo "WARNING: Previous cycle snow file ${FNSNOG} is not available. Job will continue but not update snow."
  export FNSNOA=" "
  export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"
else
  echo "INFO: Previous cycle snow file is ${FNSNOG}"
fi

[GeorgeGayno-NOAA]

Something like (not sure the syntax is exactly correct):

# one or both files dont exist. turn off cycling of snow.
if [[ -f ${FNSNOA} ]] || [ -f ${snoprv} ]]; then
    export FNSNOA=" "
    export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"
# both files exist. check grib date of each file. if the current data it not newer than the previous data, turn off cycling.
elif [ $($WGRIB -4yr $FNSNOA 2>/dev/null | grep -i snowc | awk -F: '{print $3}' | awk -F= '{print $2}') -le \
     $($WGRIB -4yr $snoprv 2>/dev/null | grep -i snowc | awk -F: '{print $3}' | awk -F= '{print $2}') ] ; then
    export FNSNOA=" "
    export CYCLVARS="FSNOL=99999.,FSNOS=99999.,"
# current data is newer than previous data. apply snow.
else
    export SNOW_NUDGE_COEFF=${SNOW_NUDGE_COEFF:-0.}
    export CYCLVARS="FSNOL=${SNOW_NUDGE_COEFF},$CYCLVARS"
fi

I think that will work. There may be a way to combine the first two branches. Also, how is $snoprv set when GETGES is called? Can we do a '-f' check on it?

[KateFriedman-NOAA]

how is $snoprv set when GETGES is called?

When RUN_GETGES=NO it uses the value of FNSNOG:

+ exgdas_enkf_sfc.sh[108]: snoprv=/scratch1/NCEPDEV/stmp4/Kate.Friedman/RUNTESTS_test/COMROOT/C96C48_hybatmDA/gdas.20211221/00/obs/gdas.t00z.snogrb_t1534.3072.1536

If RUN_GETGES=YES then it would invoke this line:

snoprv=$($GETGESSH -q -t snogrb_$JCAP_CASE -e $gesenvir -n $GDUMP -v $GDATE)

I tried that by hardcoding RUN_GETGES=YES and hit a bunch of unbound variables and issues. Can tell we never invoke getges.sh in our v17 testing. I think we should rework this part and/or consider updating getges.sh or doing away with it. Thoughts?