[Question] Scale-to-zero for P/D via KEDA + EPP

[vivekk16]

/kind question

What are you trying to do?

KServe's LLMInferenceService with disaggregated P/D behind the EPP/Gateway API Inference Extension currently allows autoscaling of prefill and decode independently with KEDA.

Questions from KServe perspective:

Coordination - If decode and prefill scale independently, can one pool hit zero while the other still has replicas? Is that a valid/expected state, or should they always scale to zero together?

Use case scenarios - Any reference scenario for scale-to-zero P/D showing expected idleReplicaCount/cooldown config and cold-start latency to expect?

[vivekk16]

cc @asm582

[asm582]

As discussed, one solution would be to wrap p and d as a single LWS object with flow control enabled. As a pure metrics solution, one can query running and queued requests for p and d over the desired number of minutes (e.g., 10 minutes). If there is no activity for the desired number of minutes, then LWS could be scaled to zero, leading to coordinated scaling down of p and d.