Fix SourceScanOptimizer generating duplicated columns
#1494
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds a check to `can_combine`, which is called by the `SourceScanOptimizer` when determining whether it can combine two branches into one. The new check makes sure we only combine branches if: - both branches don't share aliases - if they do share aliases, the aliases refer to different things
This commit adds dataflow and SQL snapshot tests that verify we're not generating duplicated aliases when combining branches. I created 3 new dummy metrics for these tests: `derived_sharedalias_1a`, `derived_sharedalias_1b` and `derived_sharedalias_2`. The `1a` and `1b` variant both make `shared_alias` refer to `bookings`, while the `2` variant makes it refer to `instant_bookings`. This way we can test the 2 cases: - If the alias refers to the same thing (`1a` vs `1b`), we can merge branches safely - If the alias refers to different things (`1a` vs `2`), we cannot merge branches safely
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
This commit adds a new integration test that uses `derived_sharedalias_1a` and `derived_sharedalias_2`. I manually optimized the SQL from the output of the optimized SQL snapshot.
serramatutu
force-pushed
the
serramatutu/validation-fix
branch
from
726f86b to
ba207b9
Compare
courtneyholcomb
added
the
Run Tests With Other SQL Engines
courtneyholcomb
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
courtneyholcomb
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
courtneyholcomb
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
courtneyholcomb
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
github-actions
bot
removed
the
Run Tests With Other SQL Engines
...ntics/metricflow_semantics/test_helpers/semantic_manifest_yamls/simple_manifest/metrics.yaml
Outdated
Show resolved
Hide resolved
...ntics/metricflow_semantics/test_helpers/semantic_manifest_yamls/simple_manifest/metrics.yaml
Outdated
Show resolved
Hide resolved
tests_metricflow/query_rendering/test_derived_metric_rendering.py
Outdated
Show resolved
Hide resolved
...test_derived_metric_that_defines_the_same_alias_in_different_components__plan0_optimized.sql
Outdated
Show resolved
Hide resolved
|
Since the last review I added some fixup commits to fix the PR comments. |
serramatutu
added
the
Run Tests With Other SQL Engines
serramatutu
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
serramatutu
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
serramatutu
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
serramatutu
had a problem deploying
to
DW_INTEGRATION_TESTS
GitHub Actions
Failure
courtneyholcomb
approved these changes
Nov 4, 2024
github-actions
bot
removed
the
Run Tests With Other SQL Engines
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes the issue with
SourceScanOptimizergenerating duplicated columns. The problem was happening because we didn't check for the same alias referring to different things before merging nodes.The fix for it was pretty simple, and I added a few tests to catch this issue.
I recommend reviewing commit by commit :)