Skip to content

feat(metrics): add histogram impl && add table scan metric#171

Merged
lxy-9602 merged 5 commits intoalibaba:mainfrom
SGZW:zhangwei/histogram
Mar 11, 2026
Merged

feat(metrics): add histogram impl && add table scan metric#171
lxy-9602 merged 5 commits intoalibaba:mainfrom
SGZW:zhangwei/histogram

Conversation

@SGZW
Copy link
Contributor

@SGZW SGZW commented Mar 7, 2026

Purpose

• Add histogram metric support to the public Metrics API by introducing HistogramStats and histogram operations (ObserveHistogram, GetHistogramStats, GetAllHistogramStats).
• Implement histogram collection and windowed aggregation (percentiles and stddev estimation), and integrate histogram storage/merge/serialization in MetricsImpl.
• Enhance FileStoreScan scan metrics during plan creation (FileStoreScan::CreatePlan):
• Capture plan-building duration in milliseconds and store it as a counter (lastScanDuration) and a histogram sample (scanDuration) for repeated observations
(src/paimon/core/operation/file_store_scan.cpp:166).
• Introduce the histogram metric key ScanMetrics::SCAN_DURATION (src/paimon/core/operation/metrics/scan_metrics.h:24).

Tests

• UT: histogram and scan-metrics related unit tests are updated/covered.
• IT: N/A

API and Format

• API: Yes. Public metrics API is extended in include/paimon/metrics.h:31.
• Storage format / protocol: No.

Documentation

• Introduces a new metrics capability (histogram). No additional documentation is added in this PR.

@SGZW
Copy link
Contributor Author

SGZW commented Mar 7, 2026

@lucasfang @lxy-9602 PTAL, thanks!

@SGZW SGZW requested a review from lxy-9602 March 9, 2026 10:39
@SGZW SGZW force-pushed the zhangwei/histogram branch 3 times, most recently from d233337 to e6215ab Compare March 9, 2026 15:04
@lucasfang lucasfang requested a review from Copilot March 10, 2026 01:02
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds histogram support to Paimon’s public Metrics API and wires a new scan-plan duration histogram metric into FileStoreScan so repeated plan creations can be analyzed via percentiles/stddev.

Changes:

  • Extend public Metrics API with histogram operations and HistogramStats.
  • Implement histogram collection (bucketed) plus windowed aggregation and integrate into MetricsImpl (merge/overwrite/JSON serialization).
  • Record scan plan build duration as both a counter (lastScanDuration) and histogram sample (scanDuration), and add unit tests.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
include/paimon/metrics.h Public API extension: HistogramStats + histogram methods on Metrics.
src/paimon/common/metrics/metrics_impl.h / .cpp Implements histogram storage, observation, merge/overwrite, and JSON flattening.
src/paimon/common/metrics/histogram.h / .cpp Adds internal bucketed histogram implementation and stats estimation.
src/paimon/common/metrics/histogram_windowing.h / .cpp Adds time-windowed histogram wrapper for recent-window aggregation.
src/paimon/core/operation/metrics/scan_metrics.h Adds ScanMetrics::SCAN_DURATION histogram metric key.
src/paimon/core/operation/file_store_scan.cpp Records scan plan build duration into counter + histogram.
src/paimon/core/operation/key_value_file_store_scan_test.cpp Verifies scan duration counter + histogram stats behavior.
src/paimon/core/operation/append_only_file_store_scan_test.cpp Verifies scan duration counter + histogram stats behavior (append-only).
src/paimon/common/metrics/histogram_test.cpp Unit tests for histogram correctness + MetricsImpl histogram integration.
src/paimon/common/metrics/histogram_windowing_test.cpp Unit tests for windowing advancement/reset/merge/clone behavior.
src/paimon/CMakeLists.txt Adds new histogram sources and tests to the build.
LICENSE Notes RocksDB inspiration for histogram utilities.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@SGZW SGZW force-pushed the zhangwei/histogram branch from e6215ab to cd826ca Compare March 10, 2026 02:27
@SGZW
Copy link
Contributor Author

SGZW commented Mar 10, 2026

@lxy-9602 updated,ptal,thanks

Copy link
Collaborator

@lucasfang lucasfang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602 lxy-9602 merged commit 7c89b2e into alibaba:main Mar 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants