Poc: expose group hash feedback from GroupValues by Rachelint · Pull Request #23229 · apache/datafusion

Rachelint · 2026-06-29T01:09:25Z

Which issue does this PR close?

Part of hash aggregate repartition POC.

Rationale for this change

This POC prepares GroupValues for partition-aware partial aggregate output. The partial aggregate output path needs to know which input rows created new groups, and the hash for those rows, so it can later route newly-created group ids to target partitions without relying on a separate repartition + coalesce pipeline.

What changes are included in this PR?

Extend GroupValues::intern to fill per-row hashes and optionally return the input rows that created new groups.
Keep hash calculation inside GroupValues implementations.
Wire reusable hash/new-group-row buffers through aggregate hash table and row-hash paths.
Update existing call sites and benchmarks to the new intern signature.

Are these changes tested?

Yes:

cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-physical-plan group_values --lib

github-actions · 2026-06-29T01:11:45Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-common v54.0.0 (current)
       Built [  33.665s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.056s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  29.254s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.055s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.935s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ExecutionOptions.enable_partial_aggregation_local_repartition in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:723

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  65.818s] datafusion-common
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  32.700s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.125s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  33.045s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.125s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.848s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_parameter_count_changed: pub trait method parameter count changed ---

Description:
A trait method now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_parameter_count_changed.ron

Failed in:
  GroupValues::intern now takes 4 instead of 2 parameters, in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/group_values/mod.rs:102

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  68.422s] datafusion-physical-plan

Rachelint · 2026-06-30T12:50:20Z

run benchmarks clickbench_partitioned

adriangbot · 2026-06-30T12:53:05Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4843647256-752-r4jz6 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fuse-aggr-repart-poc-2 (d82f05b) to 32d3d3a (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-30T13:19:55Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and fuse-aggr-repart-poc-2
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Query     ┃                                       HEAD ┃                fuse-aggr-repart-poc-2 ┃         Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ QQuery 0  │               1.23 / 3.95 ±5.41 / 14.78 ms │          1.24 / 4.05 ±5.48 / 15.02 ms │      no change │
│ QQuery 1  │             12.84 / 13.05 ±0.20 / 13.41 ms │        12.74 / 12.83 ±0.06 / 12.92 ms │      no change │
│ QQuery 2  │             36.80 / 37.13 ±0.25 / 37.55 ms │        35.57 / 35.79 ±0.19 / 36.09 ms │      no change │
│ QQuery 3  │             30.75 / 31.41 ±0.76 / 32.79 ms │        30.34 / 30.85 ±0.42 / 31.50 ms │      no change │
│ QQuery 4  │      1641.91 / 1729.24 ±64.24 / 1835.36 ms │     230.71 / 233.36 ±2.60 / 237.88 ms │  +7.41x faster │
│ QQuery 5  │     1573.18 / 1762.93 ±185.86 / 2092.25 ms │     270.25 / 276.34 ±4.55 / 282.17 ms │  +6.38x faster │
│ QQuery 6  │                1.26 / 1.41 ±0.23 / 1.86 ms │           1.27 / 1.44 ±0.24 / 1.91 ms │      no change │
│ QQuery 7  │             13.70 / 13.92 ±0.16 / 14.17 ms │        13.92 / 14.02 ±0.10 / 14.21 ms │      no change │
│ QQuery 8  │      2033.24 / 2055.11 ±18.34 / 2083.82 ms │     340.73 / 344.57 ±2.80 / 348.74 ms │  +5.96x faster │
│ QQuery 9  │          479.21 / 485.59 ±5.34 / 492.40 ms │    453.60 / 470.04 ±13.29 / 493.25 ms │      no change │
│ QQuery 10 │           75.79 / 86.69 ±17.00 / 120.57 ms │        69.50 / 73.22 ±5.41 / 83.94 ms │  +1.18x faster │
│ QQuery 11 │             90.62 / 91.84 ±1.10 / 93.50 ms │        80.40 / 81.62 ±1.47 / 84.39 ms │  +1.13x faster │
│ QQuery 12 │      1634.55 / 1701.81 ±77.06 / 1850.32 ms │     269.88 / 272.97 ±3.64 / 279.33 ms │  +6.23x faster │
│ QQuery 13 │        565.80 / 798.94 ±147.53 / 958.94 ms │     360.25 / 373.10 ±9.75 / 384.03 ms │  +2.14x faster │
│ QQuery 14 │          559.23 / 568.91 ±8.28 / 581.13 ms │     283.82 / 290.73 ±4.16 / 295.32 ms │  +1.96x faster │
│ QQuery 15 │      1948.90 / 2028.62 ±63.16 / 2130.34 ms │     276.97 / 281.58 ±3.57 / 286.66 ms │  +7.20x faster │
│ QQuery 16 │      4162.27 / 4335.65 ±90.20 / 4419.39 ms │     630.48 / 639.52 ±7.84 / 652.10 ms │  +6.78x faster │
│ QQuery 17 │     4153.91 / 4306.26 ±118.49 / 4441.64 ms │    628.67 / 650.03 ±22.64 / 692.45 ms │  +6.62x faster │
│ QQuery 18 │  17861.22 / 18714.80 ±535.28 / 19436.77 ms │ 1271.81 / 1295.27 ±17.46 / 1320.70 ms │ +14.45x faster │
│ QQuery 19 │             28.17 / 30.17 ±2.96 / 36.05 ms │        27.85 / 28.08 ±0.12 / 28.17 ms │  +1.07x faster │
│ QQuery 20 │          514.46 / 520.09 ±5.05 / 529.05 ms │    512.82 / 523.76 ±10.82 / 544.27 ms │      no change │
│ QQuery 21 │          511.95 / 517.24 ±5.77 / 526.93 ms │    510.28 / 525.06 ±10.41 / 541.68 ms │      no change │
│ QQuery 22 │         979.12 / 987.22 ±9.88 / 1006.20 ms │  989.27 / 1012.37 ±20.85 / 1043.08 ms │      no change │
│ QQuery 23 │      3010.41 / 3042.56 ±24.52 / 3064.73 ms │ 3084.83 / 3124.51 ±33.99 / 3169.72 ms │      no change │
│ QQuery 24 │             41.02 / 41.59 ±0.76 / 43.02 ms │        41.71 / 43.44 ±1.82 / 46.93 ms │      no change │
│ QQuery 25 │          110.73 / 113.76 ±2.50 / 116.27 ms │     111.19 / 114.68 ±5.41 / 125.43 ms │      no change │
│ QQuery 26 │             41.66 / 42.86 ±0.74 / 43.79 ms │        41.83 / 45.82 ±3.63 / 50.12 ms │   1.07x slower │
│ QQuery 27 │         665.88 / 676.57 ±11.10 / 696.07 ms │     671.79 / 677.52 ±3.91 / 682.32 ms │      no change │
│ QQuery 28 │     3493.90 / 3798.99 ±158.00 / 3948.21 ms │  3072.35 / 3077.99 ±4.90 / 3085.57 ms │  +1.23x faster │
│ QQuery 29 │             40.46 / 40.95 ±0.39 / 41.54 ms │       40.84 / 57.80 ±15.82 / 76.97 ms │   1.41x slower │
│ QQuery 30 │         549.65 / 570.50 ±11.31 / 583.08 ms │     307.59 / 312.38 ±4.06 / 318.83 ms │  +1.83x faster │
│ QQuery 31 │          299.87 / 305.00 ±3.00 / 309.13 ms │     284.44 / 298.14 ±9.00 / 309.72 ms │      no change │
│ QQuery 32 │        939.71 / 966.12 ±37.59 / 1040.32 ms │ 1054.98 / 1064.72 ±11.10 / 1080.27 ms │   1.10x slower │
│ QQuery 33 │ 25445.84 / 27426.47 ±2060.28 / 31321.90 ms │ 1469.87 / 1505.13 ±20.49 / 1530.42 ms │ +18.22x faster │
│ QQuery 34 │ 28373.67 / 30066.24 ±1095.85 / 31809.92 ms │ 1511.09 / 1538.69 ±26.70 / 1572.55 ms │ +19.54x faster │
│ QQuery 35 │     1053.64 / 1122.79 ±102.45 / 1326.72 ms │    280.69 / 328.90 ±66.63 / 460.86 ms │  +3.41x faster │
│ QQuery 36 │         158.22 / 177.87 ±18.60 / 209.54 ms │        70.90 / 75.27 ±3.66 / 79.72 ms │  +2.36x faster │
│ QQuery 37 │            38.90 / 49.04 ±18.40 / 85.76 ms │       36.14 / 42.06 ±11.17 / 64.39 ms │  +1.17x faster │
│ QQuery 38 │             42.36 / 43.71 ±0.90 / 44.79 ms │        43.45 / 46.90 ±3.62 / 53.03 ms │   1.07x slower │
│ QQuery 39 │         169.06 / 193.35 ±15.99 / 218.83 ms │     135.89 / 150.06 ±8.90 / 160.42 ms │  +1.29x faster │
│ QQuery 40 │             14.98 / 15.48 ±0.54 / 16.48 ms │        14.54 / 15.07 ±0.69 / 16.41 ms │      no change │
│ QQuery 41 │             13.83 / 14.03 ±0.14 / 14.23 ms │        13.95 / 14.22 ±0.32 / 14.65 ms │      no change │
│ QQuery 42 │             13.55 / 13.78 ±0.14 / 13.98 ms │        13.33 / 18.08 ±5.71 / 27.79 ms │   1.31x slower │
└───────────┴────────────────────────────────────────────┴───────────────────────────────────────┴────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 109543.66ms │
│ Total Time (fuse-aggr-repart-poc-2)   │  20021.94ms │
│ Average Time (HEAD)                   │   2547.53ms │
│ Average Time (fuse-aggr-repart-poc-2) │    465.63ms │
│ Queries Faster                        │          21 │
│ Queries Slower                        │           5 │
│ Queries with No Change                │          17 │
│ Queries with Failure                  │           0 │
└───────────────────────────────────────┴─────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	550.1s
Peak memory	11.9 GiB
Avg memory	6.5 GiB
CPU user	4861.6s
CPU sys	312.6s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	105.0s
Peak memory	13.2 GiB
Avg memory	4.4 GiB
CPU user	1031.6s
CPU sys	71.2s
Peak spill	0 B

File an issue against this benchmark runner

Dandandan · 2026-06-30T14:25:26Z

Huh what's HEAD doing?

Dandandan · 2026-06-30T14:25:34Z

run benchmarks clickbench_partitioned

adriangbot · 2026-06-30T14:27:43Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4844538345-753-x75z6 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fuse-aggr-repart-poc-2 (bc70b9d) to 32d3d3a (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

Rachelint · 2026-06-30T14:33:01Z

Huh what's HEAD doing?

Can see
#23250

adriangbot · 2026-06-30T14:56:29Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and fuse-aggr-repart-poc-2
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Query     ┃                                      HEAD ┃                 fuse-aggr-repart-poc-2 ┃         Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ QQuery 0  │              1.23 / 3.96 ±5.40 / 14.76 ms │           1.25 / 4.06 ±5.55 / 15.16 ms │      no change │
│ QQuery 1  │            12.58 / 12.88 ±0.17 / 13.10 ms │         13.31 / 13.83 ±0.54 / 14.66 ms │   1.07x slower │
│ QQuery 2  │            35.61 / 35.95 ±0.24 / 36.31 ms │         36.30 / 36.49 ±0.16 / 36.69 ms │      no change │
│ QQuery 3  │            30.48 / 31.39 ±0.94 / 33.07 ms │         30.61 / 30.85 ±0.21 / 31.19 ms │      no change │
│ QQuery 4  │     1649.38 / 1727.72 ±54.68 / 1775.51 ms │  1171.38 / 1192.61 ±14.95 / 1214.30 ms │  +1.45x faster │
│ QQuery 5  │     1681.09 / 1749.40 ±44.57 / 1807.88 ms │     842.19 / 874.40 ±30.95 / 928.91 ms │  +2.00x faster │
│ QQuery 6  │               1.30 / 1.48 ±0.27 / 2.02 ms │            1.29 / 1.45 ±0.26 / 1.95 ms │      no change │
│ QQuery 7  │            13.85 / 14.08 ±0.13 / 14.22 ms │         14.94 / 15.70 ±1.22 / 18.13 ms │   1.12x slower │
│ QQuery 8  │     2050.58 / 2119.62 ±62.00 / 2229.22 ms │      345.95 / 351.80 ±3.62 / 356.40 ms │  +6.03x faster │
│ QQuery 9  │        463.95 / 494.32 ±21.45 / 529.75 ms │  1061.05 / 1113.73 ±28.10 / 1141.00 ms │   2.25x slower │
│ QQuery 10 │            76.43 / 77.14 ±0.67 / 78.29 ms │         74.42 / 75.60 ±1.29 / 78.11 ms │      no change │
│ QQuery 11 │           87.28 / 93.49 ±9.97 / 113.36 ms │         85.33 / 85.68 ±0.24 / 86.06 ms │  +1.09x faster │
│ QQuery 12 │     1648.92 / 1803.18 ±88.83 / 1914.26 ms │   968.58 / 1001.73 ±19.55 / 1023.72 ms │  +1.80x faster │
│ QQuery 13 │       582.93 / 710.85 ±116.35 / 863.04 ms │     425.52 / 455.07 ±33.13 / 518.38 ms │  +1.56x faster │
│ QQuery 14 │         541.54 / 555.41 ±8.46 / 566.38 ms │     290.45 / 302.97 ±17.52 / 336.95 ms │  +1.83x faster │
│ QQuery 15 │     1907.57 / 1966.29 ±46.84 / 2041.00 ms │  1479.19 / 1555.23 ±41.25 / 1596.65 ms │  +1.26x faster │
│ QQuery 16 │     4364.79 / 4460.91 ±77.97 / 4576.34 ms │     629.73 / 659.41 ±23.69 / 687.98 ms │  +6.77x faster │
│ QQuery 17 │    4269.36 / 4391.60 ±120.37 / 4549.93 ms │     634.90 / 648.42 ±10.94 / 662.18 ms │  +6.77x faster │
│ QQuery 18 │ 17761.38 / 18597.37 ±489.11 / 19253.41 ms │  1251.26 / 1293.08 ±31.85 / 1348.34 ms │ +14.38x faster │
│ QQuery 19 │            28.54 / 31.94 ±5.89 / 43.68 ms │         28.10 / 31.45 ±5.55 / 42.48 ms │      no change │
│ QQuery 20 │         515.30 / 523.53 ±9.02 / 535.77 ms │     514.94 / 530.28 ±15.71 / 559.54 ms │      no change │
│ QQuery 21 │         515.14 / 518.61 ±3.57 / 525.37 ms │      517.45 / 522.79 ±5.60 / 533.16 ms │      no change │
│ QQuery 22 │       996.15 / 1002.36 ±6.68 / 1011.86 ms │    990.05 / 1003.37 ±9.44 / 1016.11 ms │      no change │
│ QQuery 23 │     3066.49 / 3084.19 ±15.47 / 3105.13 ms │  3107.51 / 3148.50 ±29.36 / 3191.77 ms │      no change │
│ QQuery 24 │            40.74 / 41.26 ±0.52 / 42.22 ms │         41.74 / 43.90 ±2.70 / 48.94 ms │   1.06x slower │
│ QQuery 25 │         110.31 / 112.48 ±2.65 / 117.32 ms │      112.96 / 114.11 ±0.92 / 115.20 ms │      no change │
│ QQuery 26 │            41.66 / 42.64 ±0.73 / 43.92 ms │         42.01 / 46.51 ±3.55 / 52.28 ms │   1.09x slower │
│ QQuery 27 │        667.13 / 679.88 ±12.13 / 702.44 ms │      677.27 / 681.03 ±5.53 / 691.95 ms │      no change │
│ QQuery 28 │    3443.23 / 3649.18 ±149.01 / 3829.98 ms │  3742.55 / 3837.57 ±82.61 / 3950.28 ms │   1.05x slower │
│ QQuery 29 │           40.51 / 52.89 ±15.30 / 76.77 ms │         40.44 / 40.69 ±0.22 / 41.03 ms │  +1.30x faster │
│ QQuery 30 │        554.77 / 569.66 ±13.29 / 589.31 ms │      305.36 / 316.78 ±7.84 / 329.53 ms │  +1.80x faster │
│ QQuery 31 │        289.92 / 303.12 ±18.96 / 340.69 ms │      283.19 / 292.80 ±7.03 / 302.07 ms │      no change │
│ QQuery 32 │      949.81 / 1007.50 ±33.07 / 1048.45 ms │   1061.06 / 1074.22 ±7.63 / 1083.88 ms │   1.07x slower │
│ QQuery 33 │ 25867.68 / 27037.94 ±995.91 / 28314.21 ms │ 4548.33 / 4793.03 ±167.98 / 5042.23 ms │  +5.64x faster │
│ QQuery 34 │ 27679.77 / 28987.55 ±876.01 / 30135.41 ms │ 4685.79 / 5028.50 ±225.99 / 5309.30 ms │  +5.76x faster │
│ QQuery 35 │     1088.53 / 1161.10 ±88.40 / 1330.31 ms │  1335.53 / 1399.27 ±64.02 / 1479.84 ms │   1.21x slower │
│ QQuery 36 │         162.86 / 172.00 ±7.97 / 183.72 ms │         81.10 / 86.86 ±4.80 / 93.05 ms │  +1.98x faster │
│ QQuery 37 │           38.33 / 49.38 ±17.99 / 85.15 ms │         37.14 / 38.33 ±1.18 / 40.51 ms │  +1.29x faster │
│ QQuery 38 │            44.75 / 45.63 ±0.71 / 46.87 ms │        40.76 / 51.23 ±16.00 / 82.88 ms │   1.12x slower │
│ QQuery 39 │         185.96 / 194.54 ±6.27 / 204.49 ms │     135.12 / 149.71 ±14.88 / 176.01 ms │  +1.30x faster │
│ QQuery 40 │            14.28 / 14.71 ±0.26 / 15.10 ms │         14.32 / 16.75 ±4.13 / 24.99 ms │   1.14x slower │
│ QQuery 41 │            13.67 / 14.18 ±0.43 / 14.90 ms │         13.89 / 15.00 ±1.74 / 18.47 ms │   1.06x slower │
│ QQuery 42 │            12.90 / 14.30 ±2.45 / 19.19 ms │         13.17 / 13.46 ±0.28 / 14.00 ms │  +1.06x faster │
└───────────┴───────────────────────────────────────────┴────────────────────────────────────────┴────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 108157.60ms │
│ Total Time (fuse-aggr-repart-poc-2)   │  32988.25ms │
│ Average Time (HEAD)                   │   2515.29ms │
│ Average Time (fuse-aggr-repart-poc-2) │    767.17ms │
│ Queries Faster                        │          19 │
│ Queries Slower                        │          11 │
│ Queries with No Change                │          13 │
│ Queries with Failure                  │           0 │
└───────────────────────────────────────┴─────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	545.1s
Peak memory	12.3 GiB
Avg memory	6.4 GiB
CPU user	4842.8s
CPU sys	322.8s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	170.0s
Peak memory	12.5 GiB
Avg memory	4.7 GiB
CPU user	1024.4s
CPU sys	86.4s
Peak spill	0 B

File an issue against this benchmark runner

github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 29, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jun 29, 2026

Expose group hash feedback from GroupValues

03c7079

Rachelint force-pushed the fuse-aggr-repart-poc-2 branch from f501156 to 03c7079 Compare June 29, 2026 01:29

github-actions Bot added the physical-expr Changes to the physical-expr crates label Jun 29, 2026

Rachelint marked this pull request as draft June 29, 2026 05:37

Optimize skip partial aggregation group values

0e09f71

Rachelint force-pushed the fuse-aggr-repart-poc-2 branch from 8e3137d to 0e09f71 Compare June 29, 2026 08:45

Rachelint added 2 commits June 29, 2026 17:17

Reset feedback in non-column group values

c6f8933

Revert unused binary map hash feedback

25229c2

github-actions Bot removed the physical-expr Changes to the physical-expr crates label Jun 29, 2026

Rachelint added 7 commits June 29, 2026 17:35

Minimize primitive group values intern change

49eabfa

Always support partial repartition in column groups

e013580

Allow skip partial through either path

da15c1e

Track partial aggregation repartition groups

0432f05

Relax partial repartition grouping check

510d7b3

Minimize row group values intern change

98d0a67

Use append-only group values for skip partial

5db2863

Rachelint force-pushed the fuse-aggr-repart-poc-2 branch from 1d3e71a to 3fc7753 Compare June 30, 2026 00:55

Materialize append-only partial output

1583cd4

Rachelint force-pushed the fuse-aggr-repart-poc-2 branch 8 times, most recently from 13a67ae to 6a5284a Compare June 30, 2026 07:33

Emit partial repartition batches

1f6edfd

Rachelint force-pushed the fuse-aggr-repart-poc-2 branch from 6a5284a to 1f6edfd Compare June 30, 2026 07:44

Rachelint added 3 commits June 30, 2026 18:17

Bypass coalesce for partitioned aggregation

7d3846c

Move partial aggregation repartition into emit

8cfa875

Optimize local partial repartition

d82f05b

Simplify partial local repartition

bc70b9d

github-actions Bot added the common Related to common crate label Jun 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Poc: expose group hash feedback from GroupValues#23229

Poc: expose group hash feedback from GroupValues#23229
Rachelint wants to merge 17 commits into
apache:mainfrom
Rachelint:fuse-aggr-repart-poc-2

Rachelint commented Jun 29, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026 •

edited

Loading

Uh oh!

Rachelint commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Dandandan commented Jun 30, 2026

Uh oh!

Dandandan commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Rachelint commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Rachelint commented Jun 29, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Uh oh!

github-actions Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rachelint commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Dandandan commented Jun 30, 2026

Uh oh!

Dandandan commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Rachelint commented Jun 30, 2026

Uh oh!

adriangbot commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 29, 2026 •

edited

Loading