feat(rust): first pass at v2 implementation - don't merge

[nyurik]

just some vibed v2 code for discussion purposes for now, to be cleaned up

[github-actions]

Performance Comparison main → v2-impl

Total Elapsed Time: 4.58s → 4.54s (-0.8%)
CPU Baseline: 91.06µs → 90.54µs (-0.6%)
Benchmark ID: mlt-convert

timing - Function execution time metrics.

+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| Function                             | Calls                        | Avg                            | P95                            | Total                          | % Total                      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode                    | 9505 → 9515 (+0.1%)          | 1.04ms → 1.03ms (-1.0%)        | 4.37ms → 4.42ms (+1.1%)        | 9.84s → 9.76s (-0.8%)          | 215.14% → 214.98% (-0.1%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode_into               | 38025 → 38065 (+0.1%)        | 176.04µs → 172.17µs (-2.2%)    | 549.38µs → 541.18µs (-1.5%)    | 6.69s → 6.55s (-2.1%)          | 146.28% → 144.34% (-1.3%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| writer::with                         | 2974127 → 2975906 (+0.1%)    | 1.68µs → 1.64µs (-2.4%)        | 4.46µs → 4.45µs (-0.2%)        | 5.01s → 4.89s (-2.4%)          | 109.43% → 107.71% (-1.6%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_to                     | 38027 → 38068 (+0.1%)        | 101.91µs → 99.03µs (-2.8%)     | 377.34µs → 372.74µs (-1.2%)    | 3.88s → 3.77s (-2.8%)          | 84.68% → 83.03% (-1.9%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_properties             | 38025 → 38065 (+0.1%)        | 69.75µs → 69.13µs (-0.9%)      | 318.98µs → 318.98µs (+0.0%)    | 2.65s → 2.63s (-0.8%)          | 57.95% → 57.95% (+0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_prop                   | 155655 → 155800 (+0.1%)      | 16.84µs → 16.69µs (-0.9%)      | 32.56µs → 32.46µs (-0.3%)      | 2.62s → 2.60s (-0.8%)          | 57.27% → 57.26% (-0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| physical::encode_u32s                | 2689108 → 2690617 (+0.1%)    | 810.00ns → 776.00ns (-4.2%)    | 330.00ns → 311.00ns (-5.8%)    | 2.18s → 2.09s (-4.1%)          | 47.62% → 46.01% (-3.4%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::group_string_properties | 9509 → 9519 (+0.1%)          | 136.69µs → 144.86µs (+6.0%)    | 680.45µs → 714.24µs (+5.0%)    | 1.30s → 1.38s (+6.2%)          | 28.40% → 30.37% (+6.9%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::write_shared_dict       | 31911 → 31935 (+0.1%)        | 40.31µs → 40.44µs (+0.3%)      | 110.40µs → 109.06µs (-1.2%)    | 1.29s → 1.29s (+0.0%)          | 28.11% → 28.45% (+1.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| tile::from_tile                      | 38029 → 38069 (+0.1%)        | 34.28µs → 33.28µs (-2.9%)      | 136.96µs → 135.81µs (-0.8%)    | 1.30s → 1.27s (-2.3%)          | 28.49% → 27.91% (-2.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::write_str_col               | 76393 → 76476 (+0.1%)        | 15.35µs → 15.03µs (-2.1%)      | 23.05µs → 22.88µs (-0.7%)      | 1.17s → 1.15s (-1.7%)          | 25.62% → 25.32% (-1.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::select_vertex_strategy       | 38028 → 38068 (+0.1%)        | 14.42µs → 14.69µs (+1.9%)      | 57.70µs → 57.76µs (+0.1%)      | 548.32ms → 559.08ms (+2.0%)    | 11.98% → 12.31% (+2.8%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::write_str_plain             | 76394 → 76477 (+0.1%)        | 4.65µs → 4.63µs (-0.4%)        | 8.93µs → 8.86µs (-0.8%)        | 355.11ms → 354.26ms (-0.2%)    | 7.76% → 7.80% (+0.5%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| sort::sort                           | 38029 → 38069 (+0.1%)        | 9.58µs → 8.64µs (-9.8%)        | 37.31µs → 36.03µs (-3.4%)      | 364.35ms → 328.92ms (-9.7%)    | 7.96% → 7.24% (-9.0%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| logical::encode_u32s                 | 2606308 → 2607735 (+0.1%)    | 82.00ns → 87.00ns (+6.1%)      | 230.00ns → 231.00ns (+0.4%)    | 215.79ms → 227.31ms (+5.3%)    | 4.72% → 5.01% (+6.1%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| Function                             | Calls                        | Avg                            | P95                            | Total                          | % Total                     |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| optimizer::encode                    | 9505 → 9515 (+0.1%)          | 199.4 KB → 199.3 KB (-0.1%)    | 413.2 KB → 413.2 KB (+0.0%)    | 1.8 GB → 1.8 GB (+0.0%)        | 33.55% → 33.59% (+0.1%)     |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| tile::from_tile                      | 38029 → 38069 (+0.1%)        | 44.1 KB → 43.9 KB (-0.5%)      | 204.4 KB → 202.2 KB (-1.1%)    | 1.6 GB → 1.6 GB (+0.0%)        | 29.65% → 29.63% (-0.1%)     |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| writer::with                         | 2974127 → 2975906 (+0.1%)    | 206 B → 206 B (+0.0%)          | 11.0 KB → 11.0 KB (+0.0%)      | 586.5 MB → 586.3 MB (-0.0%)    | 10.63% → 10.63% (+0.0%)     |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| strings::fsst_try_train              | 27085 → 27115 (+0.1%)        | 15.9 KB → 15.9 KB (+0.0%)      | 3.6 MB → 3.6 MB (+0.0%)        | 420.8 MB → 421.0 MB (+0.0%)    | 7.63% → 7.63% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| physical::encode_u32s                | 2689108 → 2690617 (+0.1%)    | 136 B → 136 B (+0.0%)          | 38.6 KB → 38.6 KB (+0.0%)      | 349.1 MB → 349.0 MB (-0.0%)    | 6.33% → 6.33% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| shared_dict::group_string_properties | 9509 → 9519 (+0.1%)          | 29.5 KB → 29.4 KB (-0.3%)      | 164.6 KB → 163.0 KB (-1.0%)    | 274.4 MB → 273.0 MB (-0.5%)    | 4.97% → 4.95% (-0.4%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| encode::write_to                     | 38027 → 38068 (+0.1%)        | 4.0 KB → 4.0 KB (+0.0%)        | 16.1 KB → 16.1 KB (+0.0%)      | 148.1 MB → 148.0 MB (-0.1%)    | 2.68% → 2.68% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| shared_dict::write_shared_dict       | 31911 → 31935 (+0.1%)        | 3.2 KB → 3.2 KB (+0.0%)        | 10.8 KB → 10.8 KB (+0.0%)      | 99.3 MB → 99.3 MB (+0.0%)      | 1.80% → 1.80% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| strings::write_str_col               | 76393 → 76476 (+0.1%)        | 788 B → 787 B (-0.1%)          | 3.5 KB → 3.5 KB (+0.0%)        | 57.5 MB → 57.5 MB (+0.0%)      | 1.04% → 1.04% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| logical::encode_u32s                 | 2606308 → 2607735 (+0.1%)    | 11 B → 11 B (+0.0%)            | 224 B → 224 B (+0.0%)          | 29.0 MB → 29.0 MB (+0.0%)      | 0.53% → 0.53% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| strings::write_raw_str_data          | 184199 → 184388 (+0.1%)      | 126 B → 121 B (-4.0%)          | 18.8 KB → 18.5 KB (-1.6%)      | 22.3 MB → 21.4 MB (-4.0%)      | 0.40% → 0.39% (-2.5%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| encode::build_morton_dict            | 472 → 472 (+0.0%)            | 44.5 KB → 44.5 KB (+0.0%)      | 89.7 KB → 89.7 KB (+0.0%)      | 20.5 MB → 20.5 MB (+0.0%)      | 0.37% → 0.37% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| strings::write_str_plain             | 76394 → 76477 (+0.1%)        | 64 B → 64 B (+0.0%)            | 256 B → 256 B (+0.0%)          | 4.7 MB → 4.7 MB (+0.0%)        | 0.09% → 0.09% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| encode::write_prop                   | 155655 → 155800 (+0.1%)      | 24 B → 24 B (+0.0%)            | 256 B → 256 B (+0.0%)          | 3.7 MB → 3.6 MB (-2.7%)        | 0.07% → 0.07% (+0.0%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| 🆕 encode::write_id_value_stream     | 0 → 44168 (+100.0%) ⚠️       | 0 B → 70 B (+100.0%) ⚠️        | 0 B → 318 B (+100.0%) ⚠️       | 0 B → 3.0 MB (+100.0%) ⚠️      | 0.00% → 0.05% (+100.0%) ⚠️  |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| 🗑️ strings::write_fsst_data          | 500 → 0 (-100.0%) 🚀         | 5.7 KB → 0 B (-100.0%) 🚀      | 34.8 KB → 0 B (-100.0%) 🚀     | 2.8 MB → 0 B (-100.0%) 🚀      | 0.05% → 0.00% (-100.0%) 🚀  |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+

Threads

No threads to compare

---

Generated with hotpath-rs

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.71%. Comparing base (1ed745b) to head (b63cf3a).

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #1328       +/-   ##
=============================================
+ Coverage     63.99%   74.71%   +10.72%     
- Complexity        0     1002     +1002     
=============================================
  Files            88      203      +115     
  Lines         14445    13255     -1190     
  Branches          0     2553     +2553     
=============================================
+ Hits           9244     9904      +660     
+ Misses         5201     2617     -2584     
- Partials          0      734      +734     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.