Test Buckets Cleanup

[erikamov]

User story / feature request

As part of the Cal-ITP Cost Reduction Plan, we need to clear test buckets to reduce Goggle Cloud monthly costs.
A Lifecycle Rule will be added to these buckets to keep deleting objects (files/data) after 30 days of its creation.

Cal-ITP Test Buckets Cleanup.pdf

Acceptance Criteria

See a lower Cloud Storage Bill and Test Buckets cleared.

Notes

Share the Bucket list on Slack channels to alert people in case someone needs any content of those buckets.

[erikamov]

Post message about the cleanup and attached PDF on slack channel calitp-data-infra. Should I post in any other channel?

[tiffanychu90]

• One things I'm noticing for my set of BQ tables for dbt (sheet Pivot Table 1), I always follow the instructions here to run poetry run dbt run --full-refresh if you haven't been creating new tables for a couple weeks.
  • I only really work in one section of the warehouse though (mart_gtfs), so I probably could have a lot of tables in tiffany_* portions of cal-itp-data-infra-staging deleted. What would be the command I should run to just bring in mart_gtfs?
  • If the analysts can always just bring in the specific portion of the tables they need for adding new tables in dbt, which seems fairly infrequent nowadays except for me + Vivek, we can delete christian, eric, mine (and I'll repopulate with the portion I need).
• Let's clean up other users who aren't working on adding warehouse tables in the foreseeable future: soren, anyone else?