feat(ingest/sigma): add default_db/schema config and optimize SQL parsing performance

[kyungsoo-datahub]

• Add default_db / default_schema config to generate fully qualified URNs (e.g., prod.public.table)
• Add generate_column_lineage config (default: false) to skip expensive column lineage computation
• Add retry logic for 429/503 errors with exponential backoff
• Handle 400 Bad Request for elements without lineage support (text boxes, images)

Why generate_column_lineage=false by default

Sigma only uses in_tables (table-level lineage) - see sigma.py:411:

sql_parser_in_tables = create_lineage_sql_parsed_result(...).in_tables
column_lineage is computed but never used. Skipping it improves performance without functional impact.

Example Config

chart_sources_platform_mapping:
  "*":
    data_source_platform: "redshift"
    default_db: "prod"
    default_schema: "public"
generate_column_lineage: false  # default

[github-actions]

Linear: ING-1487

[codecov]

Codecov Report

❌ Patch coverage is 65.78947% with 13 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...on/src/datahub/ingestion/source/sigma/sigma_api.py	44.44%	10 Missing ⚠️
...gestion/src/datahub/sql_parsing/sqlglot_lineage.py	87.50%	2 Missing ⚠️
...estion/src/datahub/ingestion/source/sigma/sigma.py	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (65.78%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.

📢 Thoughts on this report? Let us know!

[alwaysmeticulous]

✅ Meticulous spotted 0 visual differences across 969 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 2390383. This comment will update as new commits are pushed.}

[github-actions]

Linear: ING-1508

[github-actions]

Linear: ING-1542

[ligfx]

• Adding default_db/default_schema makes sense to me, though unfortunate that it's needed 🤔 I can't see anywhere in the Sigma REST API that it exposes the default database/schema for a connection, even though it's user-configurable via the UI. (IMO, also we should really re-architect how we handle connections and get rid of chart_sources_platform_mapping, but that's for another time.)

• Is there a reason to add generate_column_lineage as a user-configurable option, if column lineage is never used by the ingestion source anyways? Seems like it should just pass it to the lineage parser automatically (although I'm also not convinced that it's that much of a performance benefit).

• Also, a lot of separate behavior changes in this PR. I wonder if we could break them out into different PRs.

[kyungsoo-datahub]

• Adding default_db/default_schema makes sense to me, though unfortunate that it's needed 🤔 I can't see anywhere in the Sigma REST API that it exposes the default database/schema for a connection, even though it's user-configurable via the UI. (IMO, also we should really re-architect how we handle connections and get rid of chart_sources_platform_mapping, but that's for another time.)

Yeah, I looked into the Sigma connections API (GET /v2/connections/{connectionId}) but the response fields vary by connection type and don't expose default db/schema directly. Would need connection-type-specific parsing to extract them. Static config felt like the practical choice for now.

• Is there a reason to add generate_column_lineage as a user-configurable option, if column lineage is never used by the ingestion source anyways? Seems like it should just pass it to the lineage parser automatically (although I'm also not convinced that it's that much of a performance benefit).

Thank you for the suggesstion. Updated. In terms of performance benefit, it was aborted without this skipping. Also, we can reduce our cost for the CLL computation which doesn't work for Sigma anyways.

• Also, a lot of separate behavior changes in this PR. I wonder if we could break them out into different PRs.

Yes, now, it's divided to

PR1: #16243 (Retry 429/503 errors)
PR2: #16244 (Skip column level lineage)
PR3: #16245 (default db/default schema)