feat: support rangebitmap read and write by fafacao86 · Pull Request #185 · alibaba/paimon-cpp

fafacao86 · 2026-03-19T03:42:16Z

Purpose

Linked issue: close #146

Tests

UT in rangebtimap_file_index_test.cpp

functional tests: single-chunk, multi-chunk, different types rangebitmap read and write
different query patterns: EQ GT LT GTE LTE ISNULL, query existing data and non-existing data.
different data patterns: normal numbers mixed with NULLs, floats with -0.0, +0.0, NaN etc.
edge cases: empty rangebitmap, rangebitmap with ONLY NULLs.

IT in paimon::test::ReadInteWithIndexTest::CheckResultForRangeBitmap
data is generated using paimon-java v1.3.1.
Same data, same queries, with single-chunk and multi-chunk, result should be the same.

tests are mainly written by AI, reviewed by human.
test coverage:
range_bitmap_file_index.cpp is a little low(82.7%) is because No write integration test to cover CreateWriter Method.

[range_bitmap.cpp](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap.cpp.gcov.html)	
96.2%96.2%
96.2 %	203 / 211	100.0 %	19 / 19
[range_bitmap.h](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap.h.gcov.html)	
100.0%
100.0 %	5 / 5	100.0 %	2 / 2
[range_bitmap_file_index.cpp](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap_file_index.cpp.gcov.html)	
82.7%82.7%
82.7 %	110 / 133	96.6 %	28 / 29
[range_bitmap_file_index.h](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap_file_index.h.gcov.html)	
100.0%
100.0 %	1 / 1	100.0 %	2 / 2
[range_bitmap_file_index_factory.cpp](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap_file_index_factory.cpp.gcov.html)	
100.0%
100.0 %	4 / 4	100.0 %	4 / 4
[range_bitmap_file_index_factory.h](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap_file_index_factory.h.gcov.html)	
100.0%
100.0 %	2 / 2	100.0 %	1 / 1
[range_bitmap_file_index_test.cpp](http://172.16.25.167:8000/coverage/common/file_index/rangebitmap/range_bitmap_file_index_test.cpp.gcov.html)	
99.7%99.7%
99.7 %	317 / 318	100.0 %	61 / 61

API and Format

Documentation

Generative AI tooling

Generated-by: Kimi K2.5

src/paimon/common/file_index/rangebitmap/range_bitmap_file_index.cpp

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp

Copilot

Pull request overview

Adds RangeBitmap file index support (read/write) and validates it with new unit/integration tests plus embedded test datasets to close #146.

Changes:

Implement RangeBitmap file index reader/writer and factory registration.
Add comprehensive UTs for RangeBitmap behavior across types and edge-cases, plus IT coverage using PaIOn-generated datasets.
Add ORC/Parquet test datasets (single-chunk and multi-chunk) with range-bitmap index metadata.

Reviewed changes

Copilot reviewed 31 out of 51 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
test/test_data/parquet/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/snapshot-1	Adds Parquet multi-chunk snapshot metadata for ITs.
test/test_data/parquet/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/LATEST	Adds Parquet multi-chunk latest snapshot pointer.
test/test_data/parquet/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/EARLIEST	Adds Parquet multi-chunk earliest snapshot pointer.
test/test_data/parquet/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/schema/schema-0	Adds schema/options enabling range-bitmap with small chunk size for multi-chunk behavior.
test/test_data/parquet/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/README	Documents Parquet multi-chunk dataset rows and index config.
test/test_data/parquet/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/snapshot-1	Adds Parquet single-chunk snapshot metadata for ITs.
test/test_data/parquet/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/LATEST	Adds Parquet single-chunk latest snapshot pointer.
test/test_data/parquet/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/EARLIEST	Adds Parquet single-chunk earliest snapshot pointer.
test/test_data/parquet/append_with_rangebitmap.db/append_with_rangebitmap/schema/schema-0	Adds schema/options enabling range-bitmap for single-chunk case.
test/test_data/parquet/append_with_rangebitmap.db/append_with_rangebitmap/README	Documents Parquet single-chunk dataset rows and index config.
test/test_data/orc/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/snapshot-1	Adds ORC multi-chunk snapshot metadata for ITs.
test/test_data/orc/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/LATEST	Adds ORC multi-chunk latest snapshot pointer.
test/test_data/orc/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/snapshot/EARLIEST	Adds ORC multi-chunk earliest snapshot pointer.
test/test_data/orc/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/schema/schema-0	Adds schema/options enabling range-bitmap + ORC format + small chunk size for multi-chunk behavior.
test/test_data/orc/append_with_rangebitmap_multi_chunk.db/append_with_rangebitmap_multi_chunk/README	Documents ORC multi-chunk dataset rows and index config.
test/test_data/orc/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/snapshot-1	Adds ORC single-chunk snapshot metadata for ITs.
test/test_data/orc/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/LATEST	Adds ORC single-chunk latest snapshot pointer.
test/test_data/orc/append_with_rangebitmap.db/append_with_rangebitmap/snapshot/EARLIEST	Adds ORC single-chunk earliest snapshot pointer.
test/test_data/orc/append_with_rangebitmap.db/append_with_rangebitmap/schema/schema-0	Adds schema/options enabling range-bitmap + ORC format for single-chunk case.
test/test_data/orc/append_with_rangebitmap.db/append_with_rangebitmap/README	Documents ORC single-chunk dataset rows and index config.
test/inte/read_inte_with_index_test.cpp	Adds IT assertions for RangeBitmap index across predicates and data patterns (single/multi-chunk).
src/paimon/common/file_index/rangebitmap/range_bitmap_file_index_test.cpp	Adds UTs that roundtrip writer/reader and validate predicate behavior and edge cases.
src/paimon/common/file_index/rangebitmap/range_bitmap_file_index_factory.h	Declares factory for RangeBitmap file index.
src/paimon/common/file_index/rangebitmap/range_bitmap_file_index_factory.cpp	Implements factory creation and registration.
src/paimon/common/file_index/rangebitmap/range_bitmap_file_index.h	Declares RangeBitmap file index, reader, and writer APIs.
src/paimon/common/file_index/rangebitmap/range_bitmap_file_index.cpp	Implements index reader/writer based on RangeBitmap serialization.
src/paimon/common/file_index/rangebitmap/range_bitmap.h	Declares RangeBitmap query API and append/serialize builder.
src/paimon/common/file_index/rangebitmap/range_bitmap.cpp	Implements RangeBitmap read path and serialization format.
src/paimon/common/file_index/rangebitmap/dictionary/key_factory.h	Renames default chunk size constant to match naming conventions.
src/paimon/common/file_index/CMakeLists.txt	Adds RangeBitmap sources to build.
src/paimon/CMakeLists.txt	Registers new RangeBitmap unit test in test build.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/common/file_index/rangebitmap/range_bitmap_file_index_test.cpp

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp

src/paimon/common/file_index/rangebitmap/range_bitmap_file_index.cpp

test/inte/read_inte_with_index_test.cpp

src/paimon/common/file_index/rangebitmap/range_bitmap_file_index_test.cpp

lxy-9602 · 2026-03-22T02:37:48Z

@fafacao86 Thanks for the contribution!
We're currently improving our code quality standards, and we'd appreciate it if you could add an I/O exception test for the range index before the code is merged.

You can refer to issue #188 and take a look at this example test as a pattern.

The goal is to ensure the range index gracefully handles I/O failures (e.g., read errors, file not found, etc.), similar to how SstFileIoTest does.

Let me know if you need any help — happy to assist! Thanks!

fafacao86 · 2026-03-22T04:53:10Z

@fafacao86 Thanks for the contribution! We're currently improving our code quality standards, and we'd appreciate it if you could add an I/O exception test for the range index before the code is merged.

Sure, let me take a look.

fafacao86 · 2026-03-23T06:22:56Z

Is my understanding of the test purpose of TEST_F(SstFileIOTest, TestIOException) correct?
Triggers I/O failures at every possible point in the code (1st, 2nd, 3rd... I/O operation) to ensure that when a disk read fails or a network connection drops, the code returns an error gracefully instead of crashing? The test expectation is not to crash, and error message contains IO information?
@lxy-9602

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

src/paimon/common/file_index/rangebitmap/range_bitmap_io_test.cpp

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

lxy-9602

+1

lxy-9602 · 2026-03-24T01:02:07Z

Great contribution — the range index is high quality and well-designed! 👍
I wonder are there plans to support timestamp and variable-length types like string in the future?

fafacao86 · 2026-03-24T01:48:28Z

I wonder are there plans to support timestamp and variable-length types like string in the future?

I plan to first support String type in the near future, rangebitmap not only has the ability to do range query on numerics, but also supports string type EQ lookup which is quite useful I think.Timestamp will be later work.

lszskye · 2026-03-24T06:18:06Z

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp

+Result<PAIMON_UNIQUE_PTR<Bytes>> RangeBitmap::Appender::Serialize() const {
+    int32_t code = 0;
+    const int32_t max_code = bitmaps_.empty() ? 0 : static_cast<int32_t>(bitmaps_.size() - 1);
+    PAIMON_ASSIGN_OR_RAISE(auto bsi, BitSliceIndexBitmap::Appender::Create(0, max_code, pool_));


please change "auto" to "std::unique_ptrBitSliceIndexBitmap::Appender"

lszskye · 2026-03-24T06:22:08Z

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp

+        return Status::Invalid(fmt::format(
+            "Chunk size cannot be larger than 2GB, current bytes: {}", chunk_size_bytes_limit_));
+    }
+    PAIMON_ASSIGN_OR_RAISE(auto dictionary,


fafacao86 and others added 3 commits March 19, 2026 11:25

feat: support rangebitmap read and write

7bb9498

Merge branch 'main' into rangebitmap-inte

2be9260

Merge branch 'main' into rangebitmap-inte

8e1432c

lxy-9602 reviewed Mar 20, 2026

View reviewed changes

src/paimon/common/file_index/rangebitmap/range_bitmap_file_index.cpp Outdated Show resolved Hide resolved

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp Show resolved Hide resolved

src/paimon/common/file_index/rangebitmap/range_bitmap.cpp Show resolved Hide resolved

fafacao86 added 3 commits March 20, 2026 10:28

feat: support rangebitmap read and write

892d811

feat: support rangebitmap read and write

1d7bfde

feat: support rangebitmap read and write

53d1245

zjw1111 requested a review from Copilot March 20, 2026 10:02

Copilot AI reviewed Mar 20, 2026

View reviewed changes

Copilot started reviewing on behalf of zjw1111 March 20, 2026 10:25 View session

Merge branch 'main' into rangebitmap-inte

f898dab

fafacao86 and others added 2 commits March 23, 2026 17:11

add IO exception test

8f4b217

Apply suggestions from code review

048b89c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fafacao86 commented Mar 23, 2026

View reviewed changes

src/paimon/common/file_index/rangebitmap/range_bitmap_io_test.cpp Show resolved Hide resolved

fafacao86 and others added 4 commits March 23, 2026 17:21

Merge branch 'main' into rangebitmap-inte

dde6506

Merge branch 'main' into rangebitmap-inte

0171ef4

Apply suggestions from code review

c2ffd19

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

chunk size check

b2be075

lxy-9602 approved these changes Mar 24, 2026

View reviewed changes

lszskye reviewed Mar 24, 2026

View reviewed changes

Conversation

fafacao86 commented Mar 19, 2026

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 commented Mar 22, 2026

Uh oh!

fafacao86 commented Mar 22, 2026

Uh oh!

fafacao86 commented Mar 23, 2026

Uh oh!

Uh oh!

lxy-9602 left a comment

Choose a reason for hiding this comment

Uh oh!

lxy-9602 commented Mar 24, 2026

Uh oh!

fafacao86 commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lszskye Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lszskye Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fafacao86 commented Mar 24, 2026 •

edited

Loading

lszskye Mar 24, 2026 •

edited

Loading