V2 saturation: the compute bound (k2) measures a KV stock, but the constraint it stands for is a rate

[ev-shindin]

Summary

The V2 saturation analyzer models a replica's capacity as min(k1, k2) in KV tokens:

• k1 = TotalKvCapacityTokens × kvCacheThreshold — the memory bound.
• k2 — the compute bound, recorded as tokensInUse at the moment the replica was seen
  queueing, then averaged into a rolling history keyed
  model|accelerator|gpuCount|outputBucket.

k2 therefore measures a KV stock, while the constraint it is supposed to represent is
a rate. On prefill-heavy traffic the two are unrelated: the engine exhausts prompt-token
throughput and begins queueing while KV occupancy is still low. In that regime the analyzer
reports abundant headroom on a replica that is already dropping requests, and the
scale-down path releases replicas it should be holding.

Evidence

Sustained 1000-input/250-output run, 10 replicas × 16 concurrency. The WVA leg was queueing,
dropping requests and cycling replicas at 16.2% average KV utilization — at 16%
occupancy demand/supply reads as abundant headroom, so the engine cannot express what is
happening.

In the comparison run at 100-input/1000-output the same mechanism shows up as its other
face: mid-experiment WVA drains to one replica while load is unchanged, then has to
climb back, and the tail latency for that window is a direct consequence.

Six threshold/window/policy legs were run against the 1000/250 workload — varying
scaleUpThreshold, scaleDownBoundary, the stabilization window and the KEDA scaleDown
policy — and none of them broke the cycling. This is not a tuning problem.

Root cause

internal/engines/analyzers/saturation_v2/analyzer.go L302–L320 (computeK2, Priorities 1
and 2). Three distinct mechanisms, all observed in the runs above:

1. The compute bound is silently discarded. tokensInUse derives from
   kv_cache_usage_perc collected as max_over_time(...[1m]) — a peak. Whenever that peak
   exceeds kvCacheThreshold (0.80), k2 ≥ k1, so min(k1, k2) returns the memory bound
   and the compute signal never binds at all.
2. Supply is over-stated exactly when load is highest, so utilization reads low and
   spareCapacity = totalSupply − totalDemand/scaleDownBoundary grows.
3. Demand deflates faster than supply. The queue term
   QueueLength × (avgInput + avgOutput) dominates demand at peak and collapses to zero the
   moment new replicas absorb the backlog, while k2 stays inflated through the historical
   average (Priority 2). The controller sheds, the backlog returns, and the cycle repeats.

Mechanism 3 is the shed-to-one: after a drain, demand has lost its dominant term while
supply still reflects the peak.

What a fix has to separate

Two questions are conflated today. Suggested direction, and the shape of the change I have
prototyped:

detector:    rates decide WHEN a replica is at its limit
measurement: tokens record WHAT that limit is

A replica is at its limit when it has a backlog at least QueueLengthThreshold deep, or
when its arrival rate has reached the service rate measured while it was backlogged. At
that moment its resident token count is a measurement of the limit — including at 16%
occupancy, which is the whole point. That measurement is stored per workload bucket
(model, accelerator, role, GPU count, request shape) as a running minimum, and every replica
of the bucket reads the same value.

Two constraints any implementation has to respect, both of which cost me a rewrite:

• The value must be identical across replicas of a variant. aggregateByVariant takes
  the MEDIAN of per-replica capacities. A figure that varies with each replica's own load is
  not commensurable across siblings — an idle replica's value blends with a backlogged one's
  and can lift variant capacity enough to turn a scale-up into a scale-down, reintroducing
  shed-to-one by a new route. A bucket ceiling makes the median a no-op.
• The value must not move with the current cycle's load. A capacity recomputed from this
  cycle's arrival rate changes every cycle, which is an oscillation waiting to happen. A
  stored ceiling moves only as a new measurement lowers the running minimum or age relaxes
  it upward — both slow by construction.

Also worth recording: λ and μ do not sit on the same time base. A completion happens one
residence time after the arrival that caused it, so an instantaneous λ against a
completion-derived μ reads as saturation on a replica that is coping during a ramp. λ needs
smoothing over AvgTTFT + AvgOutputTokens × AvgITL.

No new metrics are required. λ (inference_extension_scheduler_attempts_total, EPP),
μ (vllm:request_generation_tokens_count), occupancy, queue depth and KV capacity are all
collected today. Two of them are registered only when the throughput analyzer is enabled,
which a fix must not depend on.

An ITL model (ITL(k) = A·k + B, already fitted by the throughput analyzer) does not
cover this: on prefill-heavy traffic ITL can stay flat while TTFT and the queue explode. It
remains the better model for decode-bound workloads.

What would close this

• A capacity estimate that binds below k1 when the replica is compute-bound at low
  occupancy, rather than being discarded by min(k1, k2).
• No shed while arrival rate is unchanged and only the queue has drained.