[5pt] PR: Incorporate Lidar-Informed elevations for bridges

[AliForghani-NOAA]

This PR closes the issue #1242.
This PR incorporates lidar-derived elevations for OSM bridges into the FIM. The workflow consists of:

• Extracting lidar-derived elevations for bridges
• Calculating DEM required corrections (elevation differences between lidar elevations and 3DEP DEMs within bridge areas)
• Adding the DEM corrections into the final HAND grid

Additions

• data/bridges/make_rasters_using_lidar.py. Generates bridge elevation raster files using lidar data, as described here. This script has to be run on Windows machines. This script must be run twice, each time with a different OSM bridge-lines GPKG file: once for the CONUS and separately for Alaska. The raster files are generated using EPSG:5070 for CONUS bridges and EPSG:3338 for Alaska bridges.

• data/bridges/make_dem_dif_for_bridges.py Generates DEM correction files by subtracting USGS 3DEP DEMs from lidar-elevation raster files at HUC levels, as described here. Non-bridge locations (which make up most of the HUC domain) have zero values, while OSM bridge locations typically show positive values. This script also needs to be run twice: once for the CONUS and a separate time specifically for Alaska.

• data/bridges/conda_fim_bridges_enviro.yml A conda environment file to install conda env needed for running data/bridges/make_rasters_using_lidar.py

• data/bridges/setup_conda_for_make_rasters.txt Provides instructions for running the data/bridges/make_rasters_using_lidar.py script on a Windows machine using a Conda environment.

Changes

• data/bridges/pull_osm_bridges.py The updates include:

  • This script now generates two separate OSM bridge centerline files: one for CONUS and another for Alaska. That was a very critically required update for Lidar data. With pull_osm_bridges not running AK correctly as it own wrong resulting in the OSM files having the wrong CRS which would affect Lidar bridges.
  • It was also pulling in some bad bridge HUCs as well as some pulling in a bunch of invalid date. While it originally had code to drop bridge type of "abandoned", it was not working. We also realized we were missing code to drop a bunch of other invalid data based on bridge types such as "razed, dismantled, destroyed, etc". We also made a decision to drop "proposed" bridge types as we don't know its status on being built. We spot checked a bunch of the "proposed" and none yet existed. Maybe they will appear in later pulls. We also found more invalid columns that triggered dropping of some bridge data.
  • We also upgraded logging a bit to find errors of what bridge data was being dropped.

• src/bash_variables.env - Ingests links to two new VRT files for DEM correction rasters (CONUS and Alaska) and the new Alaska OSM bridges GPKG file, plus new pre-clip directory.

• data/wbd/generate_pre_clip_fim_huc8.py Pre-clips the new Alaska OSM bridges GPKG file.

• src/run_unit_wb.sh Crops the DEM correction VRT file to the buffered HUC boundary and create a copy for branch 0.

• src/run_by_branch.sh Clips HUC-level DEM correction rasters for branches.

• src/heal_bridges_osm.py Now implements two distinct workflows for healing the REM at OSM bridge locations, depending on the availability of lidar data:

  • If lidar data is unavailable: The existing workflow is followed, creating a 10m buffer around the bridge centerline and applying the maximum REN value within this buffer to the entire area.
  • If lidar data is available: The script reads DEM correction rasters and add their values into the REM, effectively healing it at bridge locations (DEM correction files have zero value for non-bridge locations). Additionally, it reports the median healed REM values within a 1.5m buffer around bridge centerlines for use in determining bridge risk status.

• src/delineate_hydros_and_produce_HAND.sh Updated to align with the revised input arguments of src/heal_bridges_osm.py

• src/aggregate_by_huc.py Applies 'threshold' HAND/discharge terminology instead of 'max' HAND/discharge terminology for determining bridges risk status. Also included information on the presence or absence of lidar data in the bridge risk status report.

• tools/bridge_inundation.py. Applies 'threshold' HAND/discharge terminology instead of 'max' HAND/discharge terminology for determining bridges risk status. Also added the 'evaluated_discharge' field to show the given flow for determining the risk status.

• data/usgs/acquire_and_preprocess_3dep_dems.py : minor updates to inline comments

• data/create_vrt_file.py: small updates to text and changed timezones to be UTC instead of local to match much of our other conventions. Note: Many of our files are not yet UTC, but we hope to change them as we work on related files in the future.

• .gitignore / pyproject.toml - With the addition of the new conda_fim_bridges_enviro.yml and setup_conda_for_make_rasters.txt, our current linting system was consistantly failing linting tests. Adjustments were made to try and have linting ignore the two files.

Removals

Testing

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Current files / folders to be deployed

The new paths to be copied to all enviros are:

• data/inputs/osm/bridges/20250207/10m_dem_diffs/ (covers AK and conus)
• data/inputs/osm/bridges/20250207/alaska_osm_bridges.gpkg
• data/inputs/osm/bridges/20250207/conus_osm_bridges.gpkg
• data/inputs/pre_clip_huc8/20250218/

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

We have a fix we need to do.. but not quite yet:

data/inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg
and a conus version
/data//inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg

but bash variables is looking only for:

osm_bridges=${inputsDir}/osm/bridges/250102/huc6_dem_diff/osm_all_bridges_modified.gpkg
and
input_bridge_elev_diff=${inputsDir}/osm/bridges/250102/huc6_dem_diff/bridge_elev_diff.vrt

Shouldn't there be two sets of these bash variables? one for conus and one for ak?
which would affect run_unit_wb.sh and also run_by_branch.sh

Keep an eye on your heal_bridges_osm.py file too.

It would also affect generate_pre_clip_fim_huc8.py and clip_vectors_to_wbd.py. I can help with that if you like. Just do small subset tests on pre-clip as I am going to be merging in some Matt's fixes for pre-clips.

To do a full re-gen of conus pre-commits, I need both branches.

I think I will approve and get Matt's 1178 merged to dev first, but hold on doing a pre-clip on his, then we can merge dev into yours and update your pre-clip. Those new pre-clips can now be used to do a full clean test on fim-pipeline and your code chagnes. I will do the new pre-clips when we get there on my prod machines.

When we merge dev back to yours.. we will need to watch the delinate_hydros...sh as both of you have changes to the exact same heal calls.

We will have to put this on hold until Matt's is merged in, then we can fix yours up, including pre-clips. That will affect the files being given / loaded for fim_pipeline to use.

[AliForghani-NOAA]

Rob,
I could understand two topics:
1- Regarding bash variables links for Alaska.... Yes, this is correct and I am still working on that --have not performed an Alaska run yet.

2- Regarding pre-clipping of osm vector data.... we need to pre-clip below two files which are just some new input data and they should not have any conflict with anything else. I could not follow if there is any limitation for this data pre-clipping or not.

• CONUS: /inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg
• Alaska: /inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg

Please let me know if I missed anything else.

[RobHanna-NOAA]

Rob, I could understand two topics: 1- Regarding bash variables links for Alaska.... Yes, this is correct and I am still working on that --have not performed an Alaska run yet.

2- Regarding pre-clipping of osm vector data.... we need to pre-clip below two files which are just some new input data and they should not have any conflict with anything else. I could not follow if there is any limitation for this data pre-clipping or not.

• CONUS: /inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg
• Alaska: /inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg

Please let me know if I missed anything else.

Yes.

1. absolutely correct
2. We know we need to fix pre-clip for this which we can do after Matt's merge. We will need to pre-clip before we can accurately test the full lidar system. I can help with the pre-clip adjustments and testing as I know it pretty well. Well.. I can talk you through it if you want to do it too. Matt's pending PR has pre-clip changes we need first.

[RobHanna-NOAA]

• Next steps updated Jan 28
  • Rob: make all new DEMS (conus and AK) at the current HUC6 Conus / HUC8 AK. Something seems wrong with Matt's earlier PR for dems from earlier for layer name changes in AK. I need to check this. Just to make sure. Seeing as the WBD appears to have changed ???
    - Update: Jan 31: DEMS are good but I am changing the name and updating one of the WBD files to make it more/less have only the FIM domain WBD_National_EPSG_5070_WBDHU8_clip_dem_domain.gpkg will be replace with WBD_National_HUC8_EPSG_5070_HAND_domain.gpkg. It is a smaller version including our domain HUC8s but a few other strays. ak) removed North Alaska, some Pacific ones, a few non US ones. I didn't' fully reconcile it to our included_huc8_withAlaska.lst but much closer the the previous one.
  • With new DEM's/vrt's being created, run/test Ali's tool for doing dem diff rasters. Run the dem diff regen at scale.
    • Updated: Jan 31: No longer needed
  • Update Feb 7, 2025: - new osm bridge data completed.
  • Update Feb 7, 2025: Repull lidar data
    • Update: Feb 16: Repull Lidar Data completed.
  • Update Feb 7, 2025: - Re-run make dem differences for lidar.
  • Test pre-clips: Ali is testing one conus and one AK huc for pre-clip and will take it through fim-pipeline. (Done)
  • Run pre-clip at scale.
  • Approve my part of Ali's PR (the data/bridges code). Have someone else test / approve fim-pipeline more in depth with this full new dataset. ie) testing changes to the files in the repo "src" directory.
  • Copy new PR inputs to all enviros. PS. some already copied, but reconcile it in all 5 enviros.

This lets me test Matt's DEM, WBD change, Ali's new LIDAR DEMs tool, and pre-clip which may/may not have an error. The result of that chain gives us valid exact data going into production for this PR and some retro, already merged, PR's.

In a separate PR, make the fixes for get_sample_data ensuring the new DEMS, pre-clips and LIDAR DEMS are coming in correctly. Look for other changed / new inputs. Might only need the new LIDAR DEMs. TBD.

[RobHanna-NOAA]

Done testing pull_osm_bridges, make_rasters_using_lidar.py, make_dem_dif_for_bridges.py and pre-clip. All are done and running a scale. Ready for someone else to test fim-pipeline.

[ZahraGhahremani]

I ran this branch for HUC 10280103. For most cases, results look as expected. I found a bridge where it does not have lidar tif but the inundation does not match the current dev.
Current dev for osm id 1295421986, threatened:

Lidar bridge for osm id 1295421986, at risk:

Looking at the bridge elevation difference: