Repository navigation
Conversation
A uring worker freed its ring entries as soon as it stopped. When it stopped on an error while the connection was still up, such as a failed command or an interrupted io_uring_enter(), its other entries were still registered, and the kernel could copy the next request into freed memory. Closing the ring doesn't help in time, because the kernel cancels the commands later, from a workqueue. The worker now counts the entries whose command the kernel hasn't completed, and waits for those completions before it frees them. Once the connection is torn down they arrive at once. If they don't arrive within a second, the worker leaks the entries instead of freeing memory the kernel may still write to. A signal that interrupts io_uring_enter() no longer stops the worker, whether it was waiting or only submitting. If the wait in the drain fails with another error, the worker logs that error. The new mount test serves a PassthroughFs over io_uring and checks that dropping the serving layer still returns well within that second. It can't reproduce the race itself. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Linux 7.3 lets a FUSE-over-io_uring server give each queue a pool of payload buffers instead of a buffer per ring entry. abi::fuse_uring gets the ADD_QUEUE and ADD_BUFPOOL commands, the FUSE_HAS_IO_URING_BUFPOOL INIT flag, the offset field of the entry header and the bufpool field of the command. FsOptions doesn't know the INIT flag, so the server records it from the raw INIT flags, together with the max_write and max_pages of its reply. Server::uring_bufpool_supported() reports the flag. Server::uring_payload_size() returns the size of the kernel's payload buffers, computed the way fuse_uring_create() does. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
On Linux 7.3 and later, UringFuseServing gives each kernel queue a pool of UringConfig::bufs_per_queue buffers before any entry registers. The kernel lends a buffer to an entry while it serves a request with a payload or a reply body, and reports where the buffer starts. Pooled entries keep only their header and scratch area. Kernels that don't offer pools at INIT keep a buffer per entry, as does bufs_per_queue = 0. It defaults to 4, the default number of entries, and counts as at most entries_per_queue. A request without a payload and without out arguments gets no buffer, and its offset of 0 looks like the first buffer. The opcode table now records which requests have out arguments, and the others get an empty reply body, so they can't write into another entry's buffer. The benchmark daemon takes --uring-bufs N. The benchmark daemon served a passthrough directory of 24 files of 256 MiB with --uring, 4 workers and 16 entries per queue, on a 24-CPU machine with Linux 7.3-rc7. fio ran random 1 MiB and 4 KiB direct reads with libaio at depth 16, one job pinned to each CPU, for 8 seconds, through a fresh mount each run. The two settings alternated three times: buffers per queue VmSize VmRSS 1 MiB reads 4 KiB reads own per entry 1492 MiB 393 MiB 1518-1728 MiB/s 45.9-46.8k IOPS 4 431 MiB 106 MiB 1588-1709 MiB/s 44.5-45.7k IOPS VmSize also counts each entry's request and reply scratch areas, which are as large as its own payload buffer, so pooled entries save those too. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Kranzes
force-pushed
the
uring-bufpool-up
branch
from
October 28, 2026 15:19
7b67c06 to
daa15c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linux 7.3 lets a FUSE-over-io_uring server give each queue one pool of payload buffers instead of registering a buffer per ring entry. Each buffer has the connection's maximum payload size, 1 MiB with our INIT reply. With 16 entries per queue on a 24-CPU machine, that is 384 buffers. The kernel lends a pool buffer to an entry only while it serves a request that has a payload or a reply body, and returns it when the entry commits.
The first commit adds the protocol 7.46 uAPI to
abi::fuse_uring: theADD_QUEUEandADD_BUFPOOLcommands, theFUSE_HAS_IO_URING_BUFPOOLINIT flag, theoffsetfield of the entry header and thebufpoolfield of the command.FsOptionsdoesn't know the INIT flag, so the server records it from the raw INIT flags, along with themax_writeandmax_pagesof its reply.Server::uring_payload_size()uses those to compute the kernel's buffer size the wayfuse_uring_create()does.The second commit uses them. When the kernel offers pools and
UringConfig::bufs_per_queueisn't 0,UringFuseServingmaps one region for all queues and hands each queue its slice withADD_QUEUEandADD_BUFPOOLbefore any entry registers. Entries then register without a payload buffer, and each request's Reader and Writer point at the buffer whose offset the kernel reports. A request without a payload and without out arguments gets no buffer, and its offset of 0 looks the same as the first buffer, so the opcode table now records which requests have out arguments and the others get an empty reply body. Kernels before 7.3 don't offer pools at INIT and keep a buffer per entry.bufs_per_queuedefaults to 4, the default number of entries.UringConfiggains a field, so code that builds it with a struct literal needs..Default::default().With the benchmark daemon's
--uringmode, 4 workers and 16 entries per queue on a 24-CPU machine, random 1 MiB direct reads from one fio job per CPU ran at the same speed with and without pools. Over three alternating runs, RSS dropped from 393 MiB to 106 MiB. The reserved size dropped from 1492 MiB to 431 MiB, but most of that comes from the two scratch areas every entry allocates at the size of its own payload buffer, which pooled entries don't need.docs/fuse-uring-performance.mdhas the setup and the table, and the benchmark daemon takes--uring-bufs Nto compare.This sits on #280, because pooled entries rely on the worker giving its entries back only after the kernel does. The mount test now runs with 0, 1 and 4 buffers per queue, writing and reading 3 MiB files from four threads, so requests wait for buffers. The test suite passes in a user namespace on Linux 7.3-rc7 on both async runtimes, apart from the 14 passthrough tests that need real root and also fail on master. Zero-copy, the other 7.3 feature, needs
CAP_SYS_ADMINand comes in a later PR.