Skip to content

[NEEDS #280] fusedev: use the Linux 7.3 FUSE-over-io_uring buffer pools - #281

Draft
Kranzes wants to merge 3 commits into
cloud-hypervisor:masterfrom
Kranzes:uring-bufpool-up
Draft

Kranzes wants to merge 3 commits into
cloud-hypervisor:masterfrom
Kranzes:uring-bufpool-up

Conversation

@Kranzes

@Kranzes Kranzes commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Linux 7.3 lets a FUSE-over-io_uring server give each queue one pool of payload buffers instead of registering a buffer per ring entry. Each buffer has the connection's maximum payload size, 1 MiB with our INIT reply. With 16 entries per queue on a 24-CPU machine, that is 384 buffers. The kernel lends a pool buffer to an entry only while it serves a request that has a payload or a reply body, and returns it when the entry commits.

The first commit adds the protocol 7.46 uAPI to abi::fuse_uring: the ADD_QUEUE and ADD_BUFPOOL commands, the FUSE_HAS_IO_URING_BUFPOOL INIT flag, the offset field of the entry header and the bufpool field of the command. FsOptions doesn't know the INIT flag, so the server records it from the raw INIT flags, along with the max_write and max_pages of its reply. Server::uring_payload_size() uses those to compute the kernel's buffer size the way fuse_uring_create() does.

The second commit uses them. When the kernel offers pools and UringConfig::bufs_per_queue isn't 0, UringFuseServing maps one region for all queues and hands each queue its slice with ADD_QUEUE and ADD_BUFPOOL before any entry registers. Entries then register without a payload buffer, and each request's Reader and Writer point at the buffer whose offset the kernel reports. A request without a payload and without out arguments gets no buffer, and its offset of 0 looks the same as the first buffer, so the opcode table now records which requests have out arguments and the others get an empty reply body. Kernels before 7.3 don't offer pools at INIT and keep a buffer per entry. bufs_per_queue defaults to 4, the default number of entries. UringConfig gains a field, so code that builds it with a struct literal needs ..Default::default().

With the benchmark daemon's --uring mode, 4 workers and 16 entries per queue on a 24-CPU machine, random 1 MiB direct reads from one fio job per CPU ran at the same speed with and without pools. Over three alternating runs, RSS dropped from 393 MiB to 106 MiB. The reserved size dropped from 1492 MiB to 431 MiB, but most of that comes from the two scratch areas every entry allocates at the size of its own payload buffer, which pooled entries don't need. docs/fuse-uring-performance.md has the setup and the table, and the benchmark daemon takes --uring-bufs N to compare.

This sits on #280, because pooled entries rely on the worker giving its entries back only after the kernel does. The mount test now runs with 0, 1 and 4 buffers per queue, writing and reading 3 MiB files from four threads, so requests wait for buffers. The test suite passes in a user namespace on Linux 7.3-rc7 on both async runtimes, apart from the 14 passthrough tests that need real root and also fail on master. Zero-copy, the other 7.3 feature, needs CAP_SYS_ADMIN and comes in a later PR.

A uring worker freed its ring entries as soon as it stopped. When it
stopped on an error while the connection was still up, such as a failed
command or an interrupted io_uring_enter(), its other entries were still
registered, and the kernel could copy the next request into freed
memory. Closing the ring doesn't help in time, because the kernel
cancels the commands later, from a workqueue.

The worker now counts the entries whose command the kernel hasn't
completed, and waits for those completions before it frees them. Once
the connection is torn down they arrive at once. If they don't arrive
within a second, the worker leaks the entries instead of freeing memory
the kernel may still write to.

A signal that interrupts io_uring_enter() no longer stops the worker,
whether it was waiting or only submitting. If the wait in the drain
fails with another error, the worker logs that error.

The new mount test serves a PassthroughFs over io_uring and checks that
dropping the serving layer still returns well within that second. It
can't reproduce the race itself.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Linux 7.3 lets a FUSE-over-io_uring server give each queue a pool of
payload buffers instead of a buffer per ring entry. abi::fuse_uring gets
the ADD_QUEUE and ADD_BUFPOOL commands, the FUSE_HAS_IO_URING_BUFPOOL
INIT flag, the offset field of the entry header and the bufpool field of
the command.

FsOptions doesn't know the INIT flag, so the server records it from the
raw INIT flags, together with the max_write and max_pages of its reply.
Server::uring_bufpool_supported() reports the flag.
Server::uring_payload_size() returns the size of the kernel's payload
buffers, computed the way fuse_uring_create() does.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
On Linux 7.3 and later, UringFuseServing gives each kernel queue a pool
of UringConfig::bufs_per_queue buffers before any entry registers. The
kernel lends a buffer to an entry while it serves a request with a
payload or a reply body, and reports where the buffer starts. Pooled
entries keep only their header and scratch area. Kernels that don't
offer pools at INIT keep a buffer per entry, as does bufs_per_queue = 0.
It defaults to 4, the default number of entries, and counts as at most
entries_per_queue.

A request without a payload and without out arguments gets no buffer,
and its offset of 0 looks like the first buffer. The opcode table now
records which requests have out arguments, and the others get an empty
reply body, so they can't write into another entry's buffer.

The benchmark daemon takes --uring-bufs N.

The benchmark daemon served a passthrough directory of 24 files of
256 MiB with --uring, 4 workers and 16 entries per queue, on a 24-CPU
machine with Linux 7.3-rc7. fio ran random 1 MiB and 4 KiB direct reads
with libaio at depth 16, one job pinned to each CPU, for 8 seconds,
through a fresh mount each run. The two settings alternated three times:

  buffers per queue  VmSize    VmRSS    1 MiB reads       4 KiB reads
  own per entry      1492 MiB  393 MiB  1518-1728 MiB/s   45.9-46.8k IOPS
  4                  431 MiB   106 MiB  1588-1709 MiB/s   44.5-45.7k IOPS

VmSize also counts each entry's request and reply scratch areas, which
are as large as its own payload buffer, so pooled entries save those
too.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant