feat(ingest/snowflake): apply table name normalization for queries_v2 #12566
Conversation
Adds config `table_name_normalization_rules` to snowflake. The tables identified by these rules should typically be temporary or transient. This helps in uniform query_id generation for queries using random table names for each ETL run for tools like DBT, Segment, Fivetran
github-actions
bot
added
the
ingestion
| @@ -300,6 +301,18 @@ class SnowflakeV2Config( | |||
| "to ignore the temporary staging tables created by known ETL tools.", | |||
| ) | |||
|
|
|||
| table_name_normalization_rules: Dict[re.Pattern, str] = pydantic.Field( | |||
There was a problem hiding this comment.
where are the defaults injected in?
| @@ -300,6 +301,18 @@ class SnowflakeV2Config( | |||
| "to ignore the temporary staging tables created by known ETL tools.", | |||
| ) | |||
|
|
|||
| table_name_normalization_rules: Dict[re.Pattern, str] = pydantic.Field( | |||
| default={}, | |||
| description="[Advanced] Regex patterns for table names to normalize in lineage ingestion. " | |||
There was a problem hiding this comment.
does it make sense for us to have table_name_normalization_rules and extra_table_name_normalization_rules - setting the former overrides the defaults, but the latter lets you extend it?
the ruff linter uses a similar pattern which works pretty well https://docs.astral.sh/ruff/settings/#lint_extend-select
There was a problem hiding this comment.
That's an excellent idea. Have not injected defaults yet. That's exactly the problem I was looking to solve - how to let users extend defaults without re-declaring them, and at the same time, allow users to change defaults if they do not suit them.
| @@ -8,6 +8,7 @@ | |||
| from datahub.sql_parsing.sql_parsing_common import QueryType | |||
| from datahub.sql_parsing.sqlglot_lineage import _UPDATE_ARGS_NOT_SUPPORTED_BY_SELECT | |||
| from datahub.sql_parsing.sqlglot_utils import ( | |||
| _TABLE_NAME_NORMALIZATION_RULES, | |||
There was a problem hiding this comment.
we should remove the _ prefix if this is a public thing that other sources depend on
Adds config
table_name_normalization_rulesto snowflake. The tables identified by these rules should typically be temporary or transient. This helps in uniform query_id generation for queries using random table names for each ETL run for tools like DBT, Segment, FivetranChecklist