Refactor open data part 2

[tiffanychu90]

open data

Bring in columns needed so we can move fully into fct_monthly_scheduled_stops and fct_monthly_routes and change the underlying dataset from publishing a daily sample to everything we observed that month.

• open_data/create_routes_data.py -> published as ca_transit_routes in Geoportal
• open_data/create_stops_data.py -> published as ca_transit_stops in Geoportal
• Refactor open data part 1 #1842 surfaced need for useful bits we typically want from scheduled trips (route related columns), stops, stop_times
  • add some of these joins - Refactor fct_daily_scheduled_stops by using dim_stop_arrivals data-infra#4687
  • bring the columns needed into daily stops and monthly stops, also common columns needed caltrans_district_full Add Daily RT service summary + adjust mart_gtfs_rollup tables data-infra#4710
• Add the SHN derived columns
• Deprecate unused functions in shared_utils and unused paths in gtfs_analytics_data.yml
• Completes Research Request - Refactor open data scripts #1840

TODOs for bridge table in the warehouse

Bridge table can benefit from:

• handling missing NTD ID or RTPA. Similar issue as when analysis_name is introduced, some versioned key has it, but prior versioned keys don't
• be more resilient to pick up variations in schedule_gtfs_dataset_name (Trinity Schedule vs Trinity Remix Schedule)

[github-actions]

nbviewer URLs for impacted notebooks:

• _shared_utils/shared_utils/export_portfolio_display_yml_airtable.ipynb

[github-actions]

nbviewer URLs for impacted notebooks:

• _shared_utils/shared_utils/export_portfolio_display_yml_airtable.ipynb

[edasmalchi]

@tiffanychu90 can we keep more of portfolio_utils around for the time being? It's used in HQTA and rt_segment_speeds: https://github.com/search?q=repo%3Acal-itp%2Fdata-analyses+portfolio_utils.standardize_operator_info_for_exports&type=code

[edasmalchi]

Excited for this though, should I try doing the ArcGIS process starting from these for CA Transit Routes/Stops?

[tiffanychu90]

Excited for this though, should I try doing the ArcGIS process starting from these for CA Transit Routes/Stops?

@edasmalchi: Yes, you can start! I think I can merge tomorrow.

Other notes:

• I only used a certain subset of columns from fct_monthly_scheduled_stops to replicate what was there, but obviously take a look at the columns that would be available now. I tried to make it obvious where adjustments can be made, like setting the columns I read, and also the renaming function, so both scripts follow a similar flow.
• I also only brought in a subset of columns from the bridge table...also another place to see if there's new columns to include.
• Since month_first_day is the effective filter now, the 1st of each month is when the last month's data is available in full. I have a line in those models that doesn't add the row until the full month is available.
• Also noticed that for maybe the Nov 2025 month, one of the Amtrak routes (California Zephyr) looked weird, but just 1 of the 2 shapes present, but seemed to disappear in Dec 2025. I don't think there's a way to detect beyond visual inspection. the gdf.geometry.is_simple was False for both rows, so it'd either drop both or keep both.

[github-actions]

nbviewer URLs for impacted notebooks:

• _shared_utils/shared_utils/export_portfolio_display_yml_airtable.ipynb