Refactor big mutex to reduce contention

[seanmonstar]

This refactor puts the big internal shared mutex on a diet. The goal is to reduce contention, providing performance improvements when h2 is used on multi-threaded runtimes.

Warning

This is a large internal refactor, and it hasn't yet been sufficiently tested in a production deployment. While I tried to keep the behavior exactly the same, it's possible there's subtle changes or bugs.

If you do test this, let me know. I hope to have this tested thoroughly before possibly landing.

Architectural changes

There is now unique ownership of much of the connection state, owned solely by the "connection task".

Most conceptual things were split between an owned handle and shared values such as counts::Counts and counts::Shared. The Store is now uniquely owned by the connection, and the Streams stored in it contain an Arc<stream::Shared> for the parts that need to be shared with a "stream handle".

Stream handle operations now do very little. For receiving, it locks the shared connection recv buffer, pops a frame, and unlocks.

Likewise, when sending data, (or any other status-updating operation), it briefly locks a connection pending_streams queue, inserts itself, and unlocks. All actual changes are written on the stream's pending_ops field. The connection task includes a loop to briefly lock the queue, pop a stream, unlock, and then process the streams pending ops.

So, while this does introduce more locks, they are finer-grained, and should only be held very briefly. Longer "work" is no longer done while holding any lock.

This should result in tasks needing to wait less time for another task that previously was holding the world-lock.

cc #531

[howardjohn]

In my testing now all tests passing and cannot get any deadlocks!

Here is the result of proxying N streams of iperf traffic. "Sharing" is using 1 H2 connnection with N CONNECT streams over it, while "no sharing" is N h2 connections with 1 CONNECT stream each

Test	1 stream	2 stream	4 stream	16 stream
No proxy	157	283	400	122
With patch	30	40	39	31
Without patch	26	30	30	27
With patch, no sharing	32	48	50	36
Without patch, no sharing	24	36	44	31

overall great results!

[seanaye]

Hello!

We benchmarked this PR against our production stack (axum servers + reqwest clients, both on hyper 1.x) and found some regressions for smaller request/responses, some hangs, but improvements for large body streaming.

Setup: [patch.crates-io] pinning head 92e3789 vs merge-base 21211d0, release builds, Intel i7-14700F (28 threads), Linux 7.1.1 nixos 26.11.

~40 ms per-request stall on the server at low per-connection stream concurrency. Measured with h2load (nghttp2, so client side is unaffected) against a plain axum server serving 1 KiB responses:

Shape	base req/s	PR req/s
1 conn, m=1	25,590	24
1 conn, m=8	2,305	221
1 conn, m=64	85,619	57,815
16 conns, m=16	452,819	7,095
4 conns, m=64	341,794	363,428 (+6%)
1 conn, m=64, 64 KiB	33,748	45,931 (+36%)
1 conn, m=8, 1 MiB	3,595	4,143 (+15%)

Every request takes almost exactly ~40 ms, which smells like the Linux delayed-ACK timer: it looks like the connection task loses a wakeup and only makes progress when the peer's TCP stack eventually sends something. High multiplexing (m≥64) masks it (new requests keep the connection task busy), and large bodies mask it (window updates provide wakeups) this would explain why the iperf-over-CONNECT results above look great while request-oriented benchmarks do not.

The same ~41 ms value shows up as rare max-latency outliers on master, so this race may pre-exist and the PR just made it deterministic.

We also found two hangs by cross-pairing h2 versions between client and server:

• PR client (reqwest/hyper) hangs at ≥8 concurrent streams on one connection, even against a healthy master server (serial works fine).
• PR server hangs receiving request bodies (POST echo, 8 KiB bodies) even from a healthy master client.

I think this suggests there is considerable improvement provided the issues with smaller requests are addressed

Yea, I was looking at that benchmark myself at first. But I realized it's not representative of a real load. It sends into the connection as hard as possible without yielding, so the task is imbalanced. When I add in a yield every N requests, the numbers improve. Not as great a master still.

But, if real world testing shows it to be better everywhere else, I'm fine with just killing that benchmark.

I think our tests might suggest something different. h2load -m 1 is fully synchronized request/response with no send pressure at all, and it's the most affected shape in our testing. The workers in our test await each response before sending the next request. The pattern of being worst at low concurrency, fine at high multiplexing or with large flowing bodies points at a missing wakeup rather than task imbalance.

[seanaye]

I re-ran the tests on the new commits, this helps but it doesn't fully resolve the problem. I think the race might depend on CPU idle state. Its possible to get really poor results on the exact same test if you run the test cold (from cpu idle) vs warm.

Results for the same test h2load -n 300 -c 1 -m 1

CPU state	req/s
warm (immediately after 8 s of all-core spin loop)	5,897
cold (after 45 s of idle)	24

This reproduces 100% reliably in both directions, and probably explains why it's hard to reproduce on your end: CI runners and machines running test suites keep their cores warm, so the connection task's thread wins the race.

You can try this out for yourself here. Full disclosure the test harness was written with AI

[brentechols]

I dug into the 1 connection / N streams regression and opened a stacked draft fix in #920.

Problem found

#917 registers the connection task waker at the start of Inner::poll_complete, before draining pending stream operations. That means stream handles can wake the connection task while the connection task is still actively polling the current batch.

For receive-side capacity releases, that changes the batching behavior main had with the old single-waker boundary. Instead of accumulating releases until the active poll drains, #917 can wake and flush smaller batches, emitting smaller and more frequent connection-level WINDOW_UPDATE frames. In the instrumented reproducer, base #917 sent roughly twice as many connection WINDOW_UPDATE frames as main/fixed, with about half the average update size.

Solution

The fix in #920 delays shared.conn_task.register(cx.waker()) until after recv/send pending work has drained, restoring the previous coalescing boundary. The pending paths still register the waker before returning Poll::Pending, so transport wakeups are not missed.

Validation

Focused test suite on the fix branch:

cargo test --offline -p h2-tests --test flow_control
47 passed; 4 ignored

Benchmark configuration:

requests=300
response_size=1048576
stream_window_size=65535
connection_window_size=65535
worker_threads=8
TCP_NODELAY=false
connections=1,2,4,8
streams_per_connection=1,2,4,8

Single-run Windows loopback results:

Conns	Streams	Total conc.	Main req/s	Base #917 req/s	Fixed req/s	Fixed vs base	Fixed vs main
1	1	1	1,090.3	48.2	1,030.4	21.37x	95%
1	2	2	1,010.0	69.3	1,100.3	15.89x	109%
1	4	4	1,041.1	71.7	1,050.7	14.65x	101%
1	8	8	736.2	81.5	753.3	9.25x	102%
2	1	2	682.2	145.9	742.3	5.09x	109%
2	2	4	593.9	282.5	566.4	2.01x	95%
2	4	8	542.6	337.1	510.5	1.51x	94%
2	8	16	532.7	297.3	642.1	2.16x	121%
4	1	4	1,744.8	1,532.3	1,728.7	1.13x	99%
4	2	8	1,922.9	1,485.0	2,190.4	1.48x	114%
4	4	16	1,862.6	1,356.7	1,962.3	1.45x	105%
4	8	32	1,402.9	1,760.3	1,689.1	0.96x	120%
8	1	8	3,152.6	1,960.6	3,144.0	1.60x	100%
8	2	16	3,157.2	3,413.5	2,839.2	0.83x	90%
8	4	32	2,802.2	2,622.1	2,554.9	0.97x	91%
8	8	64	3,296.0	3,151.1	3,334.9	1.06x	101%

The main takeaway: the severe one-connection cliff disappears. Base #917 was only ~48-81 req/s for one connection; fixed #917 is back around ~753-1100 req/s, roughly matching main for those cases.

[seanmonstar]

I think this PR is in much better shape, The things previously found now have some unit test coverage, and have been fixed as best as I can tell.

As for some of you sharing, it seems this patch improves performance in multi-threaded workloads by 15-40%, which is great!