Kept null-aware anti-join NULLs in the pushed dynamic filter.

[mdashti]

What

A null-aware anti join (NOT IN) wrongly returned rows when its inner side had a NULL. This PR makes the pushed-down dynamic filter keep the probe's NULL rows, so the result collapses to zero as it should.

Why

A hash join pushes a build-side filter (key IN build_keys) down to the probe scan. That's fine for most joins, but the probe's NULL is special for NOT IN: one NULL makes every comparison unknown, so the result has to be empty. The pushed filter dropped that NULL at the scan, before the join's null-aware check ran, so rows leaked through.

How

Threaded null_aware into SharedBuildAccumulator. When it's set, build_filter ORs probe_key IS NULL into the predicate before updating the dynamic filter. Non-NULL probe rows still get filtered, so the optimization stays. HashJoinExec already validates that a null-aware join has a single probe key.

Tests

Added a parquet repro to null_aware_anti_join.slt. The existing cases use in-memory VALUES, whose scans never apply the pushed filter, so they passed despite the bug. The new one sets parquet.pushdown_filters = true so the filter runs row-level. Without the fix it returns 1, 3; with it, zero rows.

[philippemnoel]

Is this different from what Mithun fixed? apache#22104

[mdashti]

@philippemnoel interesting! Didn't see that. It showed up in our tests in paradedb/paradedb#5363 . Should check if it's the same fix or not.

[mdashti]

@philippemnoel AFAI understand, PR apache#22104 is a different bug: it preserves null_aware on the logical JoinNode proto round-trip (a serialization drop). The current PR fixes the physical dynamic filter dropping the probe NULL at runtime. Different layers.

[philippemnoel]

Cool. Awesome then :)

[mdashti]

Superseded by the upstream PR apache#23104.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 103.959s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 100.176s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.607s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 206.814s] datafusion
    Building datafusion-datasource v54.0.0 (current)
       Built [  35.997s] (current)
     Parsing datafusion-datasource v54.0.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-datasource v54.0.0 (baseline)
       Built [  35.898s] (baseline)
     Parsing datafusion-datasource v54.0.0 (baseline)
      Parsed [   0.031s] (baseline)
    Checking datafusion-datasource v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.249s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure enum_missing: pub enum removed or renamed ---

Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_missing.ron

Failed in:
  enum datafusion_datasource::RangeCalculation, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/340b07beb964ef4ccc14e3c5b2e76a96e456b41e/datafusion/datasource/src/mod.rs:427

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/function_missing.ron

Failed in:
  function datafusion_datasource::calculate_range, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/340b07beb964ef4ccc14e3c5b2e76a96e456b41e/datafusion/datasource/src/mod.rs:442

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  73.776s] datafusion-datasource
    Building datafusion-datasource-csv v54.0.0 (current)
       Built [  36.232s] (current)
     Parsing datafusion-datasource-csv v54.0.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-datasource-csv v54.0.0 (baseline)
       Built [  36.052s] (baseline)
     Parsing datafusion-datasource-csv v54.0.0 (baseline)
      Parsed [   0.011s] (baseline)
    Checking datafusion-datasource-csv v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.111s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  73.823s] datafusion-datasource-csv
    Building datafusion-datasource-json v54.0.0 (current)
       Built [  36.388s] (current)
     Parsing datafusion-datasource-json v54.0.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-datasource-json v54.0.0 (baseline)
       Built [  35.742s] (baseline)
     Parsing datafusion-datasource-json v54.0.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-datasource-json v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.089s] 223 checks: 220 pass, 3 fail, 0 warn, 30 skip

--- failure module_missing: pub module removed or renamed ---

Description:
A publicly-visible module cannot be imported by its prior path. A `pub use` may have been removed, or the module may have been renamed, removed, or made non-public.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/module_missing.ron

Failed in:
  mod datafusion_datasource_json::boundary_stream, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/340b07beb964ef4ccc14e3c5b2e76a96e456b41e/datafusion/datasource-json/src/boundary_stream.rs:18

--- failure pub_module_level_const_missing: pub module-level const is missing ---

Description:
A public const is missing or renamed
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/pub_module_level_const_missing.ron

Failed in:
  END_SCAN_LOOKAHEAD in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/340b07beb964ef4ccc14e3c5b2e76a96e456b41e/datafusion/datasource-json/src/boundary_stream.rs:36

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_datasource_json::boundary_stream::AlignedBoundaryStream, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/340b07beb964ef4ccc14e3c5b2e76a96e456b41e/datafusion/datasource-json/src/boundary_stream.rs:64

     Summary semver requires new major version: 3 major and 0 minor checks failed
    Finished [  73.705s] datafusion-datasource-json
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  34.795s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.125s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  35.110s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.134s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.592s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  72.235s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 174.747s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 173.951s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.088s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 352.566s] datafusion-sqllogictest