Skip to content

perf(multimodal): optimize Qwen vision runtime#609

Draft
yechank-nvidia wants to merge 3 commits into
mainfrom
yechan/qwen-vision-runtime-opt
Draft

perf(multimodal): optimize Qwen vision runtime#609
yechank-nvidia wants to merge 3 commits into
mainfrom
yechan/qwen-vision-runtime-opt

Conversation

@yechank-nvidia

Copy link
Copy Markdown
Collaborator

Summary

  • Reduce Qwen vision CUDA graph replay overhead by reusing prepared grid metadata and updating only active buffer regions.
  • Avoid redundant tensor concatenation when item tensors already share contiguous storage.
  • Reduce encoder output cloning and preserve views when packed outputs are already in request order.
  • Coalesce adjacent vision scatter ranges and avoid unnecessary deepstack allocation/copies when deepstack is disabled.
  • Preserve the existing encoder output contract and chunked-prefill behavior.

Test Plan

  • Verified CUDA graph batch selection, metadata padding, replay buffer updates, output ordering, and cloned-output ownership.
  • Verified image/video embedding assembly and chunked-prefill deepstack behavior.
  • Compared optimized Qwen outputs against the existing path for exact equality.
  • Ran Qwen3.5-4B image and 720p24 10-second video E2E at concurrency 1, 8, and 32.
  • Confirmed E2E performance remained within run variance; measured embedding assembly decreased from approximately 0.21-0.29 ms to 0.08-0.13 ms.
  • pytest -q test/runtime/test_encoder_cudagraph_grid_rows.py test/runtime/test_qwen_vision_runtime_optimizations.py
  • pre-commit run --all-files

Signed-off-by: yechank-nvidia <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank-nvidia <161688079+yechank-nvidia@users.noreply.github.com>
@yechank-nvidia yechank-nvidia requested a review from a team as a code owner July 8, 2026 01:27
@yechank-nvidia yechank-nvidia marked this pull request as draft July 8, 2026 01:33
Signed-off-by: yechank-nvidia <161688079+yechank-nvidia@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant