Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Selection array & fused compute + filter operations #2063

Draft
wants to merge 13 commits into
base: develop
Choose a base branch
from
Draft

Selection array & fused compute + filter operations #2063

wants to merge 13 commits into from

Conversation

joseph-isaacs
Copy link
Member

No description provided.

@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label Jan 23, 2025
@joseph-isaacs joseph-isaacs marked this pull request as draft January 23, 2025 16:48
@joseph-isaacs
Merge branch 'develop' into aduffy/selection-vec
a9656f7 
# Conflicts:
#	encodings/fsst/src/compute/mod.rs
#	vortex-array/src/array/varbin/compute/filter.rs
#	vortex-scan/src/range_scan.rs
@joseph-isaacs joseph-isaacs added benchmark Run benchmarks on this branch and removed benchmark Run benchmarks on this branch labels Jan 23, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Jan 23, 2025
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label Jan 23, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Jan 23, 2025
@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: random_access

Table of Results
name PR 763fb7a base 4d82c2e ratio (PR/base) unit
random-access/vortex-tokio-local-disk 2.59851e+06 2.62699e+06 0.989158 ns
random-access/vortex-local-fs 3.29804e+06 3.27346e+06 1.00751 ns
random-access/parquet-tokio-local-disk 2.36304e+08 2.21467e+08 1.06699 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: datafusion

Table of Results
name PR 763fb7a base 4d82c2e ratio (PR/base) unit
arrow/planning 931852 935574 0.996021 ns
arrow/exec 1.97202e+06 1.96644e+06 1.00284 ns
vortex-pushdown-compressed/planning 575528 581088 0.990433 ns
vortex-pushdown-compressed/exec 2.7261e+06 2.68109e+06 1.01679 ns
vortex-pushdown-uncompressed/planning 582613 587468 0.991736 ns
vortex-pushdown-uncompressed/exec 1.55468e+06 1.56046e+06 0.996293 ns
vortex-nopushdown-compressed/planning 941246 948696 0.992147 ns
vortex-nopushdown-compressed/exec 3.15583e+06 3.09812e+06 1.01863 ns
vortex-nopushdown-uncompressed/planning 942678 944340 0.99824 ns
vortex-nopushdown-uncompressed/exec 4.99228e+06 5.03123e+06 0.99226 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 23, 2025 •
edited by joseph-isaacs
Loading

Benchmarks: TPC-H

Table of Results
name PR 763fb7a base 4d82c2e ratio (PR/base) unit
tpch_q01/arrow 537749625 5.36595e+08 1.00215 ns
tpch_q01/parquet 751103749 7.5425e+08 0.995829 ns
tpch_q01/vortex-file-compressed 517215514 5.21059e+08 $$ \color{red} 0.992623 $$ ns
tpch_q02/arrow 142638006 1.41044e+08 1.0113 ns
tpch_q02/parquet 172481961 1.75158e+08 0.984724 ns
tpch_q02/vortex-file-compressed 146662814 1.51021e+08 0.971139 ns
tpch_q03/arrow 169774006 1.78197e+08 0.952732 ns
tpch_q03/parquet 367993777 3.75854e+08 0.979086 ns
tpch_q03/vortex-file-compressed 212364086 2.31551e+08 0.917136 ns
tpch_q04/arrow 182689713 1.76613e+08 1.03441 ns
tpch_q04/parquet 218269435 2.15979e+08 1.01061 ns
tpch_q04/vortex-file-compressed 168550613 1.71132e+08 0.984914 ns
tpch_q05/arrow 321642152 3.34465e+08 0.961661 ns
tpch_q05/parquet 516604109 5.0492e+08 1.02314 ns
tpch_q05/vortex-file-compressed 394091176 3.75612e+08 1.0492 ns
tpch_q06/arrow 27579899 2.57312e+07 1.07185 ns
tpch_q06/parquet 155353584 1.49498e+08 1.03917 ns
tpch_q06/vortex-file-compressed 78578248 6.55908e+07 1.19801 ns
tpch_q07/arrow 617892314 6.28731e+08 0.982761 ns
tpch_q07/parquet 726915416 7.61966e+08 0.954 ns
tpch_q07/vortex-file-compressed 658512251 6.39881e+08 1.02912 ns
tpch_q08/arrow 285826589 2.63615e+08 1.08426 ns
tpch_q08/parquet 539808896 5.4738e+08 0.986168 ns
tpch_q08/vortex-file-compressed 361070139 3.64072e+08 0.991755 ns
tpch_q09/arrow 487207148 4.84935e+08 1.00469 ns
tpch_q09/parquet 756692030 7.86226e+08 0.962436 ns
tpch_q09/vortex-file-compressed 584878702 6.05118e+08 0.966552 ns
tpch_q10/arrow 265869220 2.63511e+08 1.00895 ns
tpch_q10/parquet 498216915 5.03518e+08 0.989471 ns
tpch_q10/vortex-file-compressed 332993140 2.80186e+08 1.18847 ns
tpch_q11/arrow 152771633 1.41723e+08 1.07796 ns
tpch_q11/parquet 159537356 1.46868e+08 1.08626 ns
tpch_q11/vortex-file-compressed 152102307 1.28913e+08 1.17989 ns
tpch_q12/arrow 187463926 1.82167e+08 1.02908 ns
tpch_q12/parquet 323445541 3.26764e+08 0.989844 ns
tpch_q12/vortex-file-compressed 367028093 2.52849e+08 1.45157 ns
tpch_q13/arrow 165144253 1.68981e+08 0.977293 ns
tpch_q13/parquet 297177207 3.04662e+08 0.975431 ns
tpch_q13/vortex-file-compressed 174449960 1.79993e+08 0.969204 ns
tpch_q14/arrow 38467826 3.80065e+07 1.01214 ns
tpch_q14/parquet 230155607 2.32304e+08 0.990754 ns
tpch_q14/vortex-file-compressed 86299223 7.91279e+07 1.09063 ns
tpch_q15/arrow 71146399 6.85518e+07 1.03785 ns
tpch_q15/parquet 314034249 3.26914e+08 0.960603 ns
tpch_q15/vortex-file-compressed 161158953 1.36004e+08 1.18496 ns
tpch_q16/arrow 95118280 1.00563e+08 0.945858 ns
tpch_q16/parquet 106584091 1.15265e+08 0.924684 ns
tpch_q16/vortex-file-compressed 99746053 1.03993e+08 0.959164 ns
tpch_q17/arrow 622798442 6.1488e+08 1.01288 ns
tpch_q17/parquet 652326915 6.8914e+08 0.946581 ns
tpch_q17/vortex-file-compressed 598468011 6.01563e+08 0.994855 ns
tpch_q18/arrow 1262795479 1.12834e+09 1.11916 ns
tpch_q18/parquet 1444968985 1.32644e+09 1.08936 ns
tpch_q18/vortex-file-compressed 1172865293 1.15699e+09 1.01372 ns
tpch_q19/arrow 155100679 1.48207e+08 1.04652 ns
tpch_q19/parquet 437181205 4.12352e+08 1.06021 ns
tpch_q19/vortex-file-compressed 236413753 1.50122e+08 1.57481 ns
tpch_q20/arrow 177865461 1.7369e+08 1.02404 ns
tpch_q20/parquet 314913084 3.08537e+08 1.02067 ns
tpch_q20/vortex-file-compressed 222287288 2.05197e+08 1.08329 ns
tpch_q21/arrow 1016672696 1.0077e+09 1.0089 ns
tpch_q21/parquet 1108392924 1.124e+09 0.986117 ns
tpch_q21/vortex-file-compressed 1146748199 9.87621e+08 1.16112 ns
tpch_q22/arrow 83073669 7.74696e+07 1.07234 ns
tpch_q22/parquet 115498505 1.08777e+08 1.0618 ns
tpch_q22/vortex-file-compressed 88682604 8.40972e+07 1.05452 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: Clickbench

Table of Results
name PR 763fb7a base 4d82c2e ratio (PR/base) unit
clickbench_q00/parquet 1830927 1.84803e+06 0.990743 ns
clickbench_q01/parquet 59969846 6.05121e+07 0.991039 ns
clickbench_q02/parquet 116536987 1.19232e+08 0.977394 ns
clickbench_q03/parquet 84039885 8.51468e+07 0.987 ns
clickbench_q04/parquet 648430278 6.80707e+08 0.952584 ns
clickbench_q05/parquet 822245143 8.10245e+08 1.01481 ns
clickbench_q06/parquet 1946354 1.91497e+06 1.01639 ns
clickbench_q07/parquet 62378467 6.29376e+07 0.991116 ns
clickbench_q08/parquet 741693758 7.42396e+08 0.999054 ns
clickbench_q09/parquet 1047037565 1.04101e+09 1.00579 ns
clickbench_q10/parquet 257007413 2.54068e+08 1.01157 ns
clickbench_q11/parquet 301670112 2.96881e+08 1.01613 ns
clickbench_q12/parquet 848755568 8.39954e+08 1.01048 ns
clickbench_q13/parquet 1079365028 1.09922e+09 0.981934 ns
clickbench_q14/parquet 828740436 8.28898e+08 0.99981 ns
clickbench_q15/parquet 755258739 7.49588e+08 1.00757 ns
clickbench_q16/parquet 1680224128 1.62771e+09 1.03226 ns
clickbench_q17/parquet 1447716558 1.45057e+09 0.998033 ns
clickbench_q18/parquet 3062438152 3.0419e+09 1.00675 ns
clickbench_q19/parquet 64889638 6.57873e+07 0.986355 ns
clickbench_q20/parquet 1210728111 1.20567e+09 1.00419 ns
clickbench_q21/parquet 1399943939 1.40539e+09 0.996122 ns
clickbench_q22/parquet 2447128736 2.44036e+09 1.00277 ns
clickbench_q23/parquet 8248863743 8.22688e+09 1.00267 ns
clickbench_q24/parquet 532096087 5.22033e+08 1.01928 ns
clickbench_q25/parquet 512170548 5.05294e+08 1.01361 ns
clickbench_q26/parquet 592791174 5.98289e+08 0.990811 ns
clickbench_q27/parquet 1628496105 1.61232e+09 1.01003 ns
clickbench_q28/parquet 11479454954 1.13172e+10 1.01433 ns
clickbench_q29/parquet 427622228 4.24697e+08 1.00689 ns
clickbench_q30/parquet 760659772 7.67668e+08 0.990871 ns
clickbench_q31/parquet 791510706 7.88696e+08 1.00357 ns
clickbench_q32/parquet 2791963230 2.76638e+09 1.00925 ns
clickbench_q33/parquet 2828347019 2.89145e+09 0.978177 ns
clickbench_q34/parquet 2757318948 2.81974e+09 0.977863 ns
clickbench_q35/parquet 841999216 8.47197e+08 0.993865 ns
clickbench_q36/parquet 171934998 1.67406e+08 1.02705 ns
clickbench_q37/parquet 87341879 8.57019e+07 1.01914 ns
clickbench_q38/parquet 112397178 1.11307e+08 1.00979 ns
clickbench_q39/parquet 325630252 3.21578e+08 1.0126 ns
clickbench_q40/parquet 50236436 4.85722e+07 1.03426 ns
clickbench_q41/parquet 48370450 4.618e+07 1.04743 ns
clickbench_q42/parquet 68728735 6.59882e+07 1.04153 ns
clickbench_q00/vortex-file-compressed 1244287 2.06818e+06 0.601633 ns
clickbench_q01/vortex-file-compressed 27932924 2.86411e+07 0.975274 ns
clickbench_q02/vortex-file-compressed 91851218 8.80802e+07 1.04281 ns
clickbench_q03/vortex-file-compressed 81977733 7.93399e+07 1.03325 ns
clickbench_q04/vortex-file-compressed 619504045 6.06496e+08 1.02145 ns
clickbench_q05/vortex-file-compressed 650697683 6.33353e+08 1.02738 ns
clickbench_q06/vortex-file-compressed 64641490 2.124e+06 30.4338 ns
clickbench_q07/vortex-file-compressed 67220588 5.46896e+07 1.22913 ns
clickbench_q08/vortex-file-compressed 750959022 7.31922e+08 1.02601 ns
clickbench_q09/vortex-file-compressed 946369964 9.37393e+08 1.00958 ns
clickbench_q10/vortex-file-compressed 421751758 2.42138e+08 1.74178 ns
clickbench_q11/vortex-file-compressed 490945547 3.05826e+08 1.60531 ns
clickbench_q12/vortex-file-compressed 624838661 5.79467e+08 1.0783 ns
clickbench_q13/vortex-file-compressed 963303096 8.86788e+08 1.08628 ns
clickbench_q14/vortex-file-compressed 649290067 5.91904e+08 1.09695 ns
clickbench_q15/vortex-file-compressed 739312522 7.42848e+08 0.99524 ns
clickbench_q16/vortex-file-compressed 1448633083 1.41937e+09 1.02062 ns
clickbench_q17/vortex-file-compressed 1363696835 1.29779e+09 1.05079 ns
clickbench_q18/vortex-file-compressed 2990582560 2.89089e+09 1.03448 ns
clickbench_q19/vortex-file-compressed 54235892 4.33281e+07 1.25175 ns
clickbench_q20/vortex-file-compressed 615491191 4.76568e+08 1.29151 ns
clickbench_q21/vortex-file-compressed 869683654 7.41099e+08 1.17351 ns
clickbench_q22/vortex-file-compressed 1883047446 1.83749e+09 1.02479 ns
clickbench_q23/vortex-file-compressed 4629605731 3.88133e+09 1.19279 ns
clickbench_q24/vortex-file-compressed 426654186 3.5492e+08 1.20211 ns
clickbench_q25/vortex-file-compressed 349396519 3.18949e+08 1.09546 ns
clickbench_q26/vortex-file-compressed 477181239 4.09981e+08 1.16391 ns
clickbench_q27/vortex-file-compressed 1621514467 1.34995e+09 1.20117 ns
clickbench_q28/vortex-file-compressed 10879885078 1.06396e+10 1.02258 ns
clickbench_q29/vortex-file-compressed 704986981 7.18318e+08 0.981441 ns
clickbench_q30/vortex-file-compressed 667278565 5.83158e+08 1.14425 ns
clickbench_q31/vortex-file-compressed 707549415 6.05745e+08 1.16807 ns
clickbench_q32/vortex-file-compressed 2768852941 2.76541e+09 1.00124 ns
clickbench_q33/vortex-file-compressed 2278044575 2.16428e+09 1.05256 ns
clickbench_q34/vortex-file-compressed 2271149915 2.16698e+09 1.04807 ns
clickbench_q35/vortex-file-compressed 973055340 9.32895e+08 1.04305 ns
clickbench_q36/vortex-file-compressed 142506691 5.86194e+07 2.43105 ns
clickbench_q37/vortex-file-compressed 77076779 5.19247e+07 1.4844 ns
clickbench_q38/vortex-file-compressed 81312708 4.4365e+07 1.83281 ns
clickbench_q39/vortex-file-compressed 292960133 1.00987e+08 2.90097 ns
clickbench_q40/vortex-file-compressed 39311329 2.94133e+07 1.33652 ns
clickbench_q41/vortex-file-compressed 45423564 2.93945e+07 1.54531 ns
clickbench_q42/vortex-file-compressed 60344288 3.97413e+07 1.51843 ns

@gatesn
Copy link
Contributor

gatesn commented Jan 23, 2025

This is an experiment right? Don't intend to merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@joseph-isaacs @gatesn @a10y @danking