feat: support external path for global index#52
Merged
lszskye merged 7 commits intoalibaba:mainfrom Jan 13, 2026
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds support for storing global index files in an external path, allowing users to separate index storage from the main table directory.
Changes:
- Added new configuration option
global-index.external-pathto specify an external directory for global index files - Renamed
GlobalIndexIOMeta::file_nametofile_pathto support full path tracking for external indices - Updated
IndexFileMetaconstructor to acceptdv_rangesandexternal_pathparameters consistently across all usages
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| include/paimon/defs.h | Added new option constant GLOBAL_INDEX_EXTERNAL_PATH with documentation |
| src/paimon/common/defs.cpp | Implemented the new option constant |
| src/paimon/core/core_options.h | Added getter and creator methods for global index external path |
| src/paimon/core/core_options.cpp | Implemented parsing and validation logic for the new option |
| src/paimon/core/utils/file_store_path_factory.h | Added parameter and getter for global index external path |
| src/paimon/core/utils/file_store_path_factory.cpp | Implemented path resolution logic that respects external paths |
| include/paimon/global_index/global_index_io_meta.h | Renamed field from file_name to file_path for clarity |
| include/paimon/global_index/io/global_index_file_writer.h | Added ToPath method to interface |
| src/paimon/core/global_index/global_index_file_manager.h | Enhanced file manager to handle external paths |
| src/paimon/core/global_index/global_index_write_task.cpp | Updated to extract file name from full path and store external path in metadata |
| src/paimon/core/global_index/row_range_global_index_scanner_impl.cpp | Updated to use full file path instead of just file name |
| src/paimon/core/index/index_file_meta.h | Removed deprecated 4-parameter constructor that omitted dv_ranges and external_path |
| src/paimon/global_index/lumina/lumina_global_index.cpp | Updated to use file_path field and extract file name for path operations |
| src/paimon/common/global_index/bitmap/bitmap_global_index.cpp | Updated to use file_path field |
| src/paimon/common/global_index/wrap/file_index_writer_wrapper.h | Updated to store full path in metadata |
| test/inte/global_index_test.cpp | Added comprehensive test TestDataEvolutionBatchScanWithExternalPath and updated existing tests |
| Multiple test files | Updated all test files with new constructor signatures for IndexFileMeta |
| Multiple operation files | Updated all file store operations to pass the new global index external path parameter |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lucasfang
approved these changes
Jan 13, 2026
Collaborator
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Enable global index store in external path
Linked issue: #5
Tests
GlobalIndexTest.TestDataEvolutionBatchScanWithExternalPath