Skip to content

Router multiget high latency test #2477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
huangminchn wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Router multiget high latency test #2477

huangminchn wants to merge 2 commits into from

Conversation

@huangminchn
Copy link
Contributor

@huangminchn huangminchn commented Feb 12, 2026

Problem Statement

Solution

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.

huangminchn and others added 2 commits February 11, 2026 16:08
@huangminchn @claude
[router] Add pipeline diagnostic metrics to narrow down P99 latency s…
d376ce0 
…pikes

Prior investigation proved that pipeline_latency (HTTP decode to
scatter-gather handler entry) accounts for ~97% of the 1-second P99
multiget_streaming latency spikes. However, pipeline_latency is a
monolithic span with no sub-breakdown, making it impossible to
distinguish between three remaining root cause candidates:
1. EventLoop write contention from slow client I/O
2. Venice handler chain intermittent slowness (throttle/ACL/metadata)
3. EventLoop task queue buildup

This change adds targeted diagnostic metrics to differentiate these
causes:

New latency sub-breakdown (P50/P95/P99):
- pre_handler_latency: decode -> first Venice handler entry
  (captures EventLoop queuing and chunk aggregation time)
- handler_chain_latency: first Venice handler -> last Venice handler
  (captures Venice handler processing time: throttle, ACL, etc.)

New infrastructure gauges:
- eventloop_pending_tasks_avg/max: pending tasks across worker
  EventLoop threads (detects EventLoop task queue buildup)
- unwritable_channel_count: channels with full write buffers
  (detects slow client write backpressure)

Key diagnostic relationships:
- pre_handler_latency spike -> EventLoop starvation
- handler_chain_latency spike -> Venice handler slowness
- eventloop_pending_tasks_max spike -> task queue buildup
- unwritable_channel_count spike -> client write backpressure

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@huangminchn
Even more metrics
72919b3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@huangminchn