Skip to content

feat(metrics): enhance cache efficiency metrics - #720

Open
JaredTan95 wants to merge 1 commit into
llm-d:mainfrom
JaredTan95:cache-efficiency-metrics
Open

feat(metrics): enhance cache efficiency metrics#720
JaredTan95 wants to merge 1 commit into
llm-d:mainfrom
JaredTan95:cache-efficiency-metrics

Conversation

@JaredTan95

Copy link
Copy Markdown
Member

Part of #617.

Completes Sub-issue 3: Cache Efficiency Metrics Enhancement.

The index previously exposed aggregate admission/eviction counters without backend attribution or explicit cache efficiency gauges. This change adds a backend label to the existing counters and introduces kvcache_index_entries and kvcache_index_hit_rate gauges through the existing metrics registry.

  • Distinguish in_memory, cost_aware_memory, redis, and valkey backends, with unknown for custom index implementations. Metrics logging continues to aggregate admissions and evictions across backend series.
  • Report entry counts for the in-memory and cost-aware-memory indexes using inexpensive size queries. Redis/Valkey gauges are not emitted because these backends do not expose an inexpensive, accurate request-key count.
  • Update the gauges after index operations and record lookup hit metrics synchronously. The hit-rate gauge uses the existing process-wide cumulative lookup_hits_total / lookup_requests_total values; it is not an independently calculated per-backend hit ratio.
  • Test collector registration, gauge updates, and isolation of admission/eviction counter series between backends.

Existing counter metric names are preserved; consumers now receive backend-labeled series and can use sum(...) to retain aggregate views.

Validation:

  • go test ./pkg/kvcache/metrics ./pkg/kvcache/kvblock
  • go test -race ./pkg/kvcache/kvblock
  • git diff --check

@github-actions github-actions Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 10, 2026
@github-actions

Copy link
Copy Markdown

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

Signed-off-by: JaredTan95 <jian.tan@daocloud.io>
@JaredTan95
JaredTan95 force-pushed the cache-efficiency-metrics branch from 2e3d7de to 43ea0c6 Compare September 10, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant