perf(multimodal): Prewarm Kimi vision encoder before readiness

[qimcis]

Summary

Previously during warmup, only text was was warmed up, in this pr i add multimodal vision warmup as well so that the overhead is not in the first real image request

Test Plan

b300 benchmark with Kimi-K2.5-NVFP4

python3 -m smg_grpc_servicer.tokenspeed \
  --model /models/Kimi-K2.5-NVFP4 \
  --served-model-name kimi-k25-nvfp4 \
  --trust-remote-code \
  --attn-tp-size 8 --moe-tp-size 8 --max-model-len 131072 --max-num-seqs 64 \
  --max-prefill-tokens 32768 --chunked-prefill-size 32768 --gpu-memory-utilization 0.85 \
  --attention-backend tokenspeed_mla --mm-attention-backend fa4 --moe-backend flashinfer_trtllm \
  --sampling-backend flashinfer --quantization nvfp4 --kv-cache-dtype fp8 --disable-kvstore --kvstore-ratio 0.05 \
  --host 127.0.0.1 --port 50051

16 requests at concurrency 4

Run	Wall	Req/s	p50 TTFT	p95 TTFT	p50 Total	Tok/s	Errors
No prewarm	49.06s	0.33	812.8ms	43892.2ms	1446.7ms	20.9	0
Prewarm	33.96s	0.47	897.7ms	28551.8ms	1424.5ms	30.2	0

benchmark times do not include server startup or warmup time, "no prewarm" run still had the normal text/server warmup, but it did not warm up the encoder

-> first image encode without prewarm was ~15010-15025ms
-> the startup prewarm cost was ~10717-10725ms per rank, before ready
-> then, the first real image encode after prewarm: ~39-41ms

[qimcis]

cc @chenht2022

[chenht2022]

Hi, @qimcis. Is it possible to make Warmup more general, including multimodal models other than Kimi such as qwen3.5 image/video, including EPD encoder?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aab19429fc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Skip vision prewarm on decode-only workers

When a VLM server starts with disaggregation_mode="decode", run_event_loop still constructs EventLoop (only the encode role bypasses it), so this unconditional startup path prewarms/captures the vision encoder on decode workers. Decode workers only receive remote KV for extends and run decode steps, so they do not use the vision tower; because this happens before KV-cache sizing, the encoder allocations reduce decode capacity and add startup latency for no benefit. Gate the prewarm to aggregated/prefill roles, leaving EPD encode warmup in encode_loop.

Useful? React with 👍 / 👎.

[qimcis]

Ran the exact Kimi setup on Qwen:
1024x1024 image, conc 4, 16 requests, max_tokens=64

Run	Wall	Req/s	p50 TTFT	p95 TTFT	p50 Total	Tok/s
No prewarm	49.06s	0.33	812.8ms	43892.2ms	1446.7ms	20.9
Prewarm	33.96s	0.47	897.7ms	28551.8ms	1424.5ms	30.2

Results look good, have yet to test with video and EPD yet but will get to that next