fix: re-enable null-equal join dynamic filters with an IS NULL predicate#23106
Open
mdashti wants to merge 6 commits into
Open
fix: re-enable null-equal join dynamic filters with an IS NULL predicate#23106mdashti wants to merge 6 commits into
mdashti wants to merge 6 commits into
Conversation
The hash-join dynamic filter pushed `key IN build_keys` down to the probe scan for null-aware anti joins too. That drops the probe-side NULL, but `NOT IN` three-valued logic needs it to collapse the result to zero rows, so the join silently returned rows. OR `probe_key IS NULL` into the pushed predicate. Non-NULL probe rows still get filtered; only the NULL additionally survives.
Exercises the pushdown path the existing in-memory tests miss: parquet with row-level filtering, so the pushed dynamic filter actually drops rows. Without the fix `id NOT IN (SELECT eid ...)` returns 1 and 3 instead of zero rows.
github-actions
Bot
added
sqllogictest
mdashti
force-pushed
the
moe/null-equal-dynamic-filter
branch
2 times, most recently
from
64e1820 to
3721aa9
Compare
Author
|
@adriangb Can you please take a look? |
mdashti
changed the title
mdashti
force-pushed
the
moe/null-equal-dynamic-filter
branch
from
3721aa9 to
9f4e40c
Compare
RatulDawar
reviewed
Jun 23, 2026
| let any_key_is_null = self | ||
| .on_right | ||
| .iter() | ||
| .filter(|key| key.nullable(&self.probe_schema).unwrap_or(true)) |
Contributor
There was a problem hiding this comment.
Should we widen when we are unable to check column nullability ? i.e. unwrap_or(true).
From what I see this can only happen when on_right and schema are out of sync which seems to be an invalid state ?
Author
There was a problem hiding this comment.
This is a should-never-happen (as you said: keys out of sync with the probe schema), so I kept unwrap_or(true) as the safe degradation: over-widening only loses a little selectivity, while false could drop a NULL the join needs. Documented it in 9620b97.
Contributor
There was a problem hiding this comment.
I was thinking how an invalid state if achieved somehow should be handled, instead of silently handling it shouldn't we propagate the error further.
The fail safe check was added here #3238
Though I am not sure what's the consensus for things like these, so a commiter's input would be helpful here.
It's cheap, so it short-circuits NULL rows before the costlier filter. The `debug_assert` pins the single-key invariant the `on_right[0]` indexing relies on.
RESET restores the defaults at the end instead of re-setting explicit values. A probe can hold several NULLs, so the comments read as plural.
build-side predicate prunes a probe-side NULL that can null-match a build-side NULL. Push the filter with `OR key IS NULL` over the nullable probe keys instead, the way apache#23104 does for null-aware anti joins. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity.
The `unwrap_or(true)` widening on an unresolved nullability check wasn't obvious. An extra NULL row is safe; dropping a needed one isn't.
mdashti
changed the title
mdashti
force-pushed
the
moe/null-equal-dynamic-filter
branch
from
9f4e40c to
9620b97
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #23104, which adds the shared probe-NULL helper this builds on. That one should land first; until it does, this PR's diff includes its two commits.
Which issue does this close?
Re-enables the dynamic filter that #22965 disabled (#22964), with the proper null-equal semantics.
Rationale for this change
#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with
OR IS NULL". #23104 does that for null-aware anti joins; this re-enables the null-equal case the same way.What changes are included in this PR?
return falseinallow_join_dynamic_filter_pushdown.key IS NULLfor every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity.Are these changes tested?
Yes. #22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and
preserve_probe_nullsunit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening).Are there any user-facing changes?
Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results.