RFC: Introduce typed distribution metrics and native Prometheus histograms

[lyang24]

Summary

Vespa’s metrics framework primarily models counters and aggregated values such as count, sum, min, max, average, and last.

This RFC proposes a typed telemetry model and a first-class histogram metric type for the core metrics framework. The initial public output is Prometheus/OpenMetrics-compatible explicit-bucket histograms, with correct cumulative temporality, metadata, and aggregation semantics.

The first producer will be Proton matching latency. The design is intended to become the common foundation for metrics from Proton, Container, storage, distributor, controllers, and future OpenTelemetry export.

Motivation

Vespa has many operational metrics, but its current model has important limitations for large-scale Prometheus/Grafana operations:

• Averages hide tail latency.
• Maximum values are too sensitive to isolated outliers.
• Per-node p95/p99 gauges cannot be mathematically aggregated across nodes, replicas, or time windows.
• Proton matching latency currently has aggregate values only, so it cannot derive a latency distribution.
• Metrics-proxy currently flattens metrics into scalar samples and does not preserve typed Prometheus histogram semantics.
• Existing metrics snapshots represent an interval, while Prometheus counters and histograms require cumulative state for correct rate() calculations.

The desired query should be possible and correct:

histogram_quantile(
  0.99,
  sum by (le, clusterid, documenttype) (
    rate(content_proton_documentdb_matching_query_latency_seconds_bucket[5m])
  )
)

This RFC introduces the required platform capability.

Goals

• Introduce a typed metric contract for Vespa.

• Add first-class histogram support to the C++ metrics framework.

• Preserve histogram state through metric recording, snapshotting, state APIs, metrics-proxy, and Prometheus/OpenMetrics exposition.

• Expose standard Prometheus histogram series:

  • _bucket{le="..."}
  • _sum
  • _count

• Use correct cumulative temporality for Prometheus/OpenMetrics output.

• Preserve backwards compatibility for existing metrics and APIs.

• Define stable naming, unit, bucket, and cardinality rules.

• Add Proton matching latency as the initial producer.

• Reserve support for exemplars and trace correlation.

• Add end-to-end compatibility tests using Prometheus and OpenTelemetry data models.

Non-goals

• Replacing every existing Vespa metric in one change.
• Removing existing .average, .sum, .count, .max, or .95percentile metrics.
• Requiring OpenTelemetry in all Vespa deployments.
• Implementing distributed tracing in this RFC.
• Automatically converting all existing percentile gauges into histograms.
• Replacing the existing state API immediately.
• Shipping Prometheus native histograms or OpenTelemetry exponential histograms in the first implementation.

Terminology

This RFC distinguishes metric type from aggregation temporality.

Metric types:

• Monotonic counter: only increases, except after restart.
• Up-down counter: may increase and decrease.
• Gauge: current or observed value.
• Histogram: distribution of non-negative observations.
• Information or state metric: low-cardinality identity or lifecycle state.

Aggregation temporality:

• Cumulative: values accumulate for the lifetime of a metric instance.
• Delta: values represent observations since the previous collection interval.

Prometheus/OpenMetrics histogram output must use cumulative temporality. Existing Vespa snapshot APIs may continue to expose interval-oriented values for backwards compatibility.

Proposed metric contract

Every canonical metric instrument must declare:

• Name.
• Description.
• Unit.
• Metric type.
• Aggregation temporality.
• Allowed dimensions.
• Runtime cardinality limit.
• Histogram bucket schema, if applicable.
• Stability level and compatibility version.

Canonical metric identities must not encode aggregation suffixes such as .average, .sum, .95percentile, or .last.

For example:

  Canonical identity:
  content.proton.documentdb.matching.query_latency

  Type:
  histogram

  Unit:
  s

  Prometheus name:
  content_proton_documentdb_matching_query_latency_seconds

Legacy metric names remain supported through an explicit compatibility mapping.

First-class histogram metric

Introduce HistogramMetric as a first-class metric alongside CountMetric and ValueMetric.

Conceptually:

  metrics::DurationHistogramMetric queryLatency(
      "query_latency",
      {},
      "Latency in seconds when matching and ranking a query",
      this);

On the hot path:

  queryLatency.observe(latency_seconds);

The implementation must provide:

• Thread-safe observation.
• Bounded memory usage.
• Low allocation and low synchronization overhead on hot paths.
• Merge across metric sets.
• Snapshot support.
• Separate delta and cumulative views where required.
• Reset behavior for interval snapshots.
• Validation of observation values.
• Stable bucket layout for the metric lifetime.

Duration histograms accept finite, non-negative values only. Invalid values such as NaN, negative values, and infinity must be rejected and counted through telemetry self-metrics.

Histogram representation

The initial representation uses explicit buckets.

For duration metrics, define a shared standard bucket schema in seconds, for example:

  0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05,
  0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf

The final bucket schema should be reviewed against observed Vespa query, feed, persistence, and maintenance latency distributions.

Bucket schemas are immutable for a metric family and label set. Changing boundaries requires a new metric name or a versioned metric family.

The internal distribution abstraction must not be limited to classic Prometheus histograms. It should leave room for future support of:

• Explicit-bucket histograms.
• OpenTelemetry exponential histograms.
• Prometheus native histograms.
• Exemplars.

The first implementation only needs explicit buckets.

Temporality and reset semantics

Histogram bucket semantics and aggregation temporality are separate concerns.

Within one histogram point, bucket counts are cumulative by upper bound:

  count(le="0.01") <= count(le="0.05") <= ... <= count(le="+Inf")

For Prometheus/OpenMetrics output:

• _bucket, _sum, and _count must be cumulative across collection intervals.
• They must remain monotonic for the lifetime of the metric instance.
• +Inf bucket count must equal _count.
• Process restart or metric-instance recreation is a counter reset.
• Prometheus output must not expose a one-minute delta snapshot as though it were a cumulative counter.

For legacy state APIs:

• Existing interval/snapshot semantics remain unchanged.
• Histogram interval data may be exposed as a delta distribution.
• The new JSON representation must explicitly identify temporality.

For OTLP export:

• Temporality must be explicit.
• Prometheus readers require cumulative temporality.
• OTLP exporters may select delta or cumulative temporality.

Naming, units, and compatibility

A native histogram must not reuse a legacy metric base name if the legacy metric already produces _count or _sum samples.

For example, retain existing legacy metrics:

content.proton.documentdb.matching.query_latency.average
content.proton.documentdb.matching.query_latency.sum
content.proton.documentdb.matching.query_latency.count
content.proton.documentdb.matching.query_latency.max

Introduce the native histogram under a distinct, unit-bearing Prometheus family:

content.proton.documentdb.matching.query_latency_seconds

Prometheus/OpenMetrics output:

content_proton_documentdb_matching_query_latency_seconds_bucket
content_proton_documentdb_matching_query_latency_seconds_sum
content_proton_documentdb_matching_query_latency_seconds_count

The metrics system must maintain a registry that validates:

• One metric family has one type.
• One metric family has one unit.
• One metric family has one description.
• One histogram family has one bucket schema.
• Reserved Prometheus suffixes are not used incorrectly.
• Metric and label names are valid after translation.

Metrics should use canonical OpenTelemetry-compatible units where possible:

s
By
1
{request}
{document}

Prometheus/OpenMetrics output should include HELP, TYPE, and UNIT metadata.

Prometheus/OpenMetrics exposition

The new endpoint must expose typed OpenMetrics-compatible families.

Example:

  # HELP content_proton_documentdb_matching_query_latency_seconds Latency in seconds when matching and ranking a query.
  # TYPE content_proton_documentdb_matching_query_latency_seconds histogram
  # UNIT content_proton_documentdb_matching_query_latency_seconds seconds
  content_proton_documentdb_matching_query_latency_seconds_bucket{le="0.01",documenttype="music"} 123
  content_proton_documentdb_matching_query_latency_seconds_bucket{le="0.05",documenttype="music"} 456
  content_proton_documentdb_matching_query_latency_seconds_bucket{le="+Inf",documenttype="music"} 500
  content_proton_documentdb_matching_query_latency_seconds_sum{documenttype="music"} 12.34
  content_proton_documentdb_matching_query_latency_seconds_count{documenttype="music"} 500
  # EOF

The implementation must support:

• Correct content negotiation.
• HELP, TYPE, and UNIT metadata.
• Histogram bucket ordering.
• Cumulative buckets.
• +Inf bucket.
• Correct escaping and validation.
• OpenMetrics EOF where applicable.
• Legacy Prometheus text output where required for compatibility.

State API and metrics-proxy

The complete pipeline must preserve typed metric semantics.

Required work:

• Extend MetricVisitor with histogram visitation.
• Add histogram value state to the core metrics model.
• Extend MetricSnapshot to merge and expose distributions.
• Extend the state API JSON format with a versioned histogram representation.
• Extend JsonWriter.
• Extend PrometheusWriter.
• Extend metrics-proxy parsing and metric model.
• Extend metrics-proxy aggregation so it aggregates bucket, sum, and count correctly.
• Ensure metrics consumers select a histogram as a complete metric family rather than selecting arbitrary individual bucket series.

The old JSON/state API format remains available. New typed distribution output should be versioned rather than silently changing old response semantics.

Cardinality protection

Cardinality must be enforced at runtime, not only documented.

For every metric family, define:

• Maximum active label sets per process.
• Maximum bucket count.
• Allowed label keys.
• Expected cardinality budget.
• Overflow behavior.

When the cardinality limit is exceeded, further observations must aggregate into a bounded overflow series, for example:

otel.metric.overflow="true"

The system must emit self-observability metrics for:

• Active series per metric family.
• Cardinality overflows.
• Rejected label keys.
• Dropped observations.
• Invalid observations.
• Histogram memory usage.
• Histogram merge failures.
• Metrics exporter queue depth.
• Export retries.
• Export failures.
• Telemetry dropped because of backpressure.

Dimensions such as query text, document id, user id, request id, trace id, and raw URL must never be ordinary metric labels.

Initial producer: Proton matching latency

The first producer is Proton matching latency.

The initial histogram should cover:

content.proton.documentdb.matching.query_latency_seconds

Dimensions:

• documenttype.
• Existing document DB/service dimensions.
• rankProfile for rank-profile-specific metrics.

The metric should be emitted at:

• Document DB aggregate level.
• Rank-profile level.

Rank-profile cardinality must be bounded. The RFC implementation should define whether rank-profile histograms are enabled by default, configurable, or capped by a maximum number of active rank profiles.

Existing Proton latency metrics remain unchanged for compatibility.

Exemplars and trace correlation

Histogram design must reserve support for exemplars from the beginning.

A sampled observation may attach:

trace_id
span_id

and a bounded set of filtered attributes.

Exemplars are not normal metric labels. They are sparse references from a histogram observation to a trace.

Requirements:

• Exemplar storage must be bounded.
• Exemplar attributes must be filtered and privacy-safe.
• Trace and span identifiers must be preferred over arbitrary attributes.
• Exemplar label size must respect Prometheus limits.
• The initial implementation may omit exemplars, but the metric model and exposition format must support adding them without changing histogram identity.

This enables a future Grafana workflow from a p99 spike to a representative query trace.

Resource attributes and dimensions

Telemetry must distinguish resource identity from metric dimensions.

Resource attributes identify the emitting service or node:

service.name
service.instance.id
service.version
host.name
cloud.region
vespa.cluster
vespa.cluster.type
vespa.zone

Metric dimensions describe the measured operation:

documenttype
rankProfile
operation
threadpool
status_code

The platform should avoid duplicating the same identity as both resource attributes and per-point labels unless required by a specific compatibility endpoint.

Future OpenTelemetry integration

The first histogram implementation must work independently of OpenTelemetry export.

A later phase should add typed OTLP metrics export using the same canonical metric model:

• Counter → OTLP Sum.
• Gauge → OTLP Gauge.
• Explicit histogram → OTLP Histogram.
• Future exponential histogram → OTLP ExponentialHistogram.
• Resource attributes → OTLP Resource.
• Metric dimensions → OTLP point attributes.
• Exemplars → OTLP exemplars.

This avoids maintaining separate Prometheus and OpenTelemetry metric implementations.

Alternatives considered

Export p95/p99 gauges

Rejected as the primary design.

Per-node percentile gauges are useful for local diagnosis but cannot be correctly aggregated across nodes or time windows.

Use KLL or DDSketch and export quantiles

Rejected for the initial Prometheus-facing output.

Mergeable sketches may be useful internally or for future OTLP exponential histogram support. However, Prometheus/Grafana supports histogram buckets directly through histogram_quantile.

Add a direct Prometheus endpoint in Proton

Rejected.

A Proton-specific endpoint would bypass metrics consumers, state APIs, metrics-proxy, service aggregation, and common operational behavior.

Only implement histograms in Container Java metrics

Rejected.

The immediate serving-engine observability gap includes Proton and other C++ services. A Java-only solution does not address the most important search-node latency paths.

Replace all metrics in one migration

Rejected.

The current metric surface is large and operationally important. The migration must be additive and incremental.

Rollout plan

1. Define typed metric contract, naming rules, units, cardinality policy, and compatibility registry.
2. Add histogram value state and visitor support to the C++ metrics framework.
3. Add snapshot, state API JSON, and Prometheus/OpenMetrics support.
4. Add metrics-proxy typed parsing and aggregation.
5. Add Proton matching latency as the first producer.
6. Add built-in metric consumer support and custom-consumer documentation.
7. Publish Grafana dashboards, recording rules, and alerts.
8. Add histograms to Container, feed, docsum, ANN, persistence, bucket movement, and controller paths.
9. Add OTLP metrics export based on the same canonical metric model.
10. Add exemplars and end-to-end trace correlation in a follow-up tracing RFC.

Testing and compatibility

The implementation must include:

• Unit tests for observation, bucket assignment, merge, reset, snapshotting, and cumulative state.
• Tests for invalid observations.
• Tests for histogram cardinality overflow.
• Tests that verify +Inf equals count.
• Tests that verify bucket counts are cumulative.
• Tests that verify bucket schema stability.
• Tests that verify process restart and metric recreation reset semantics.
• State API JSON serialization tests.
• Prometheus/OpenMetrics exposition tests.
• Metrics-proxy parsing and aggregation tests.
• Proton integration tests.
• Hot-path benchmarks for observe() latency, allocations, memory use, and merge cost.
• Compatibility tests that parse generated exposition with Prometheus' actual text/OpenMetrics parser.
• OTLP compatibility tests for metric type, unit, description, temporality, resource attributes, and explicit bucket boundaries.
• Metrics-consumer tests that verify a complete histogram family is selected.

Documentation and operational deliverables

The implementation is incomplete without operational artifacts.

Deliver:

• Metric naming and unit guidelines.

• Standard histogram bucket schemas.

• Cardinality budget guidance.

• Migration guide from legacy metrics.

• PromQL examples.

• Recording rules.

• Alert rules.

• Default Grafana dashboards for:

  • SLO and tail latency.
  • Query path.
  • ANN traversal and timeouts.
  • Feeding and persistence.
  • Resource saturation.
  • Replication and bucket movement.
  • Metrics pipeline health.

• Runbooks for metric gaps, cardinality overflow, exporter failure, and telemetry backpressure.

Example p99 query:

  histogram_quantile(
    0.99,
    sum by (le, clusterid, documenttype) (
      rate(content_proton_documentdb_matching_query_latency_seconds_bucket[5m])
    )
  )

Open questions

• What is the initial standard duration bucket schema?
• Should rank-profile histogram emission be enabled by default, configurable, or capped?
• What should the default runtime cardinality limit be for each metric type?
• How should overflow dimensions be represented in legacy and typed APIs?
• Should the first OpenMetrics endpoint be a new versioned endpoint or an extension of the existing endpoint?
• How should cumulative histogram state coexist with existing interval snapshots?
• When should explicit histograms gain OTLP exponential histogram and Prometheus native histogram support?
• Should Container HDR Histogram metrics migrate to the common histogram model in a follow-up RFC?
• Which future RFC should define cross-Container-to-Proton trace propagation and exemplars?