In the HAND 4.9.20.1 stage-based CatFIM run, 6 sites errored out with the NOAA VDatum failed but no error message returned from VDatum API message. When these sites were re-run, they finished without error. This suggests that the issue that occurred was that the NOAA VDatum API was temporarily stalled/overwhelmed by the amount of queries that CatFIM was making and thus did not provide the proper outputs.
There are already some protections in place to retry the API calls if there's an error. However, it seems like if none of those protections work, the NOAA VDatum failed but no error message returned from VDatum API message will be returned.
There might be a good opportunity to adjust the retries/backoff factors etc. It might also be helpful to add a longer cool-off time after initial VDatum API calls fail. And then it could be good to add another cool-off period and perhaps one more retry if that NOAA VDatum failed but no error message returned from VDatum API error occurs. (And if nothing else, should add to this error message that it is likely that the API got overwhelmed and remove the 'lid' input while we're at it.)
Retry code to potentially update:
from tools/tools_shared_functions.py, run_vdatum_for_region(), approx. line 1706:
retry = Retry(connect=5, backoff_factor=5, total=4, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
Could also increase the sleep time a few lines above (currently reads time.sleep(1))
In the HAND 4.9.20.1 stage-based CatFIM run, 6 sites errored out with the
NOAA VDatum failed but no error message returned from VDatum APImessage. When these sites were re-run, they finished without error. This suggests that the issue that occurred was that the NOAA VDatum API was temporarily stalled/overwhelmed by the amount of queries that CatFIM was making and thus did not provide the proper outputs.There are already some protections in place to retry the API calls if there's an error. However, it seems like if none of those protections work, the
NOAA VDatum failed but no error message returned from VDatum APImessage will be returned.There might be a good opportunity to adjust the retries/backoff factors etc. It might also be helpful to add a longer cool-off time after initial VDatum API calls fail. And then it could be good to add another cool-off period and perhaps one more retry if that
NOAA VDatum failed but no error message returned from VDatum APIerror occurs. (And if nothing else, should add to this error message that it is likely that the API got overwhelmed and remove the 'lid' input while we're at it.)Retry code to potentially update:
from
tools/tools_shared_functions.py,run_vdatum_for_region(), approx. line 1706:retry = Retry(connect=5, backoff_factor=5, total=4, status_forcelist=[429, 500, 502, 503, 504])adapter = HTTPAdapter(max_retries=retry)Could also increase the sleep time a few lines above (currently reads
time.sleep(1))