feat(metrics): add histogram impl && add table scan metric#171
Merged
lxy-9602 merged 5 commits intoalibaba:mainfrom Mar 11, 2026
Merged
feat(metrics): add histogram impl && add table scan metric#171lxy-9602 merged 5 commits intoalibaba:mainfrom
lxy-9602 merged 5 commits intoalibaba:mainfrom
Conversation
Contributor
Author
|
@lucasfang @lxy-9602 PTAL, thanks! |
lxy-9602
reviewed
Mar 9, 2026
lxy-9602
reviewed
Mar 9, 2026
d233337 to
e6215ab
Compare
There was a problem hiding this comment.
Pull request overview
Adds histogram support to Paimon’s public Metrics API and wires a new scan-plan duration histogram metric into FileStoreScan so repeated plan creations can be analyzed via percentiles/stddev.
Changes:
- Extend public
MetricsAPI with histogram operations andHistogramStats. - Implement histogram collection (bucketed) plus windowed aggregation and integrate into
MetricsImpl(merge/overwrite/JSON serialization). - Record scan plan build duration as both a counter (
lastScanDuration) and histogram sample (scanDuration), and add unit tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| include/paimon/metrics.h | Public API extension: HistogramStats + histogram methods on Metrics. |
| src/paimon/common/metrics/metrics_impl.h / .cpp | Implements histogram storage, observation, merge/overwrite, and JSON flattening. |
| src/paimon/common/metrics/histogram.h / .cpp | Adds internal bucketed histogram implementation and stats estimation. |
| src/paimon/common/metrics/histogram_windowing.h / .cpp | Adds time-windowed histogram wrapper for recent-window aggregation. |
| src/paimon/core/operation/metrics/scan_metrics.h | Adds ScanMetrics::SCAN_DURATION histogram metric key. |
| src/paimon/core/operation/file_store_scan.cpp | Records scan plan build duration into counter + histogram. |
| src/paimon/core/operation/key_value_file_store_scan_test.cpp | Verifies scan duration counter + histogram stats behavior. |
| src/paimon/core/operation/append_only_file_store_scan_test.cpp | Verifies scan duration counter + histogram stats behavior (append-only). |
| src/paimon/common/metrics/histogram_test.cpp | Unit tests for histogram correctness + MetricsImpl histogram integration. |
| src/paimon/common/metrics/histogram_windowing_test.cpp | Unit tests for windowing advancement/reset/merge/clone behavior. |
| src/paimon/CMakeLists.txt | Adds new histogram sources and tests to the build. |
| LICENSE | Notes RocksDB inspiration for histogram utilities. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
e6215ab to
cd826ca
Compare
Contributor
Author
|
@lxy-9602 updated,ptal,thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
• Add histogram metric support to the public Metrics API by introducing HistogramStats and histogram operations (ObserveHistogram, GetHistogramStats, GetAllHistogramStats).
• Implement histogram collection and windowed aggregation (percentiles and stddev estimation), and integrate histogram storage/merge/serialization in MetricsImpl.
• Enhance FileStoreScan scan metrics during plan creation (FileStoreScan::CreatePlan):
• Capture plan-building duration in milliseconds and store it as a counter (lastScanDuration) and a histogram sample (scanDuration) for repeated observations
(src/paimon/core/operation/file_store_scan.cpp:166).
• Introduce the histogram metric key ScanMetrics::SCAN_DURATION (src/paimon/core/operation/metrics/scan_metrics.h:24).
Tests
• UT: histogram and scan-metrics related unit tests are updated/covered.
• IT: N/A
API and Format
• API: Yes. Public metrics API is extended in include/paimon/metrics.h:31.
• Storage format / protocol: No.
Documentation
• Introduces a new metrics capability (histogram). No additional documentation is added in this PR.