fix(ingest/powerbi): strip Athena catalog prefix from ODBC navigation lineage URNs#17729
Open
zlosim wants to merge 1 commit into
Open
Conversation
… lineage URNs PowerBI reports that connect to Athena via Odbc.DataSource with HierarchicalNavigation surface the Athena catalog (e.g. "AwsDataCatalog") as the Database navigation level, producing 3-part catalog.database.table qualified names. The standalone Athena connector omits the catalog and uses database.table, so these navigation-based upstream URNs never match the Athena entities and lineage silently dangles. query_lineage() already normalized Athena lineage (catalog stripping + federated _apply_table_platform_override). This extracts that into a shared _apply_athena_post_processing() helper and applies it to the navigation path in expression_lineage() too, threading the DSN through so federated overrides work there as well. Tests: add a navigation catalog-stripping test plus a navigation integration test asserting catalog stripping + athena->mysql override run together. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
the
ingestion
Contributor
|
Linear: ING-2807 Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned. |
github-actions
Bot
added
the
community-contribution
Bundle ReportBundle size has no change ✅ |
maggiehays
added
the
needs-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PowerBI reports that connect to Athena via
Odbc.DataSource(..., [HierarchicalNavigation=true])and navigateCatalog → Schema → Tableproduce upstream lineage URNs that keep the Athena catalog as a third name segment, e.g.:The standalone Athena connector emits 2-part
athena,dimensions.country,PROD, so these navigation-based PowerBI edges dangle to orphan stub datasets and no lineage shows in the UI.Problem
OdbcLineagehas two paths:Odbc.Query(SQL) →query_lineage()— already strips the Athena catalog (added in fix(ingest/powerbi): strip Athena catalog prefix from ODBC column lineage URNs #15746).Odbc.DataSource(hierarchical navigation) →expression_lineage()— buildsdatabase.schema.tablefrom the navigation accessors wheredatabaseis the catalog (AwsDataCatalog) and never strips it.So catalog stripping only ran for the SQL path. Reports using navigation (the common case for Athena-via-ODBC) got mismatched URNs.
Solution
_apply_table_platform_override) into a sharedOdbcLineage._apply_athena_post_processing(lineage, platform_pair, dsn)helper, gated onplatform == athena(no-op otherwise).query_lineage()andexpression_lineage()(the latter now threadsdsnso federated overrides work there too). Single source of truth — the two paths can't drift.Known limitation
Only 3-part navigation names are catalog-stripped. A degenerate 2-level navigation (
Database+Table, noSchema) keeps the catalog because the single level is ambiguous (catalog vs glue database) and the shared stripper can't safely strip 2-part names (the SQL path uses 2-part for a legitimatedatabase.table). This matches the native Athena connector, which requires a fullcatalog.schema.tablehierarchy. Documented in code comments.Testing
test_odbc_expression_lineage_strips_athena_catalog— navigationAwsDataCatalog → dimensions → country→athena,dimensions.country,PROD.test_odbc_expression_lineage_integration_catalog_stripping_and_platform_override— navigation + override config →mysql,federated.orders,PROD(exercises both strip and override throughexpression_lineage)../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_powerbi_parser.py→ 29 passed../gradlew :metadata-ingestion:lintFix→ clean.Checklist