Repository navigation
Conversation
Kranzes
requested review from
bergwolf,
eryugey,
imeoer,
jiangliu and
liubogithub
as code owners
October 19, 2026 23:34
A uring worker freed its ring entries as soon as it stopped. When it stopped on an error while the connection was still up, such as a failed command or an interrupted io_uring_enter(), its other entries were still registered, and the kernel could copy the next request into freed memory. Closing the ring doesn't help in time, because the kernel cancels the commands later, from a workqueue. The worker now counts the entries whose command the kernel hasn't completed, and waits for those completions before it frees them. Once the connection is torn down they arrive at once. If they don't arrive within a second, the worker leaks the entries instead of freeing memory the kernel may still write to. A signal that interrupts io_uring_enter() no longer stops the worker, whether it was waiting or only submitting. If the wait in the drain fails with another error, the worker logs that error. The new mount test serves a PassthroughFs over io_uring and checks that dropping the serving layer still returns well within that second. It can't reproduce the race itself. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
A uring worker that stopped on an error while the connection was up left its queues without entries. The kernel kept putting requests from those CPUs on them, and they waited forever, as did requests the worker had taken from the same batch but not answered. The rest of the mount kept working, so nothing noticed. The failing worker now ends the connection, as umount -f does, by writing its fusectl abort file. The kernel fails the waiting requests, gives the worker its entries back, and fails the other workers' commands too, so every worker stops and FuseServing::wait() returns, and the application sees that the mount is gone. Only the first worker writes the file. A worker whose commands fail with ENOTCONN, ENODEV or ECANCELED leaves the connection alone, because it has ended already, for example through an unmount from outside, and its minor number may belong to a new connection by then. The worker finds the file from the mount's minor device number in /proc/self/mountinfo, because a stat() of the mountpoint would wait for a request nobody serves yet. It escapes the mountpoint the way mountinfo does. Without fusectl it logs an error that asks for an unmount by hand. The new test mounts over io_uring at a path with a space, ends the connection the same way, and checks that wait() returns. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
A panic in a request handler unwound the uring worker thread, which dropped its ring entries while the kernel still held them and could copy the next request into the freed memory. The worker's queues were left without entries, so requests from those CPUs waited forever, and so did the request that panicked. The worker now catches the panic and stops as it does on an error: it ends the connection through the fusectl abort file, waits for the kernel to give the entries back, and then lets the panic go on. It logs the panic first, because nothing joins the worker thread's panic. A new test serves a file system whose LOOKUP panics, and checks that the lookup fails and FuseServing::wait() returns. Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Kranzes
force-pushed
the
uring-worker-abort
branch
from
October 28, 2026 15:19
e8a1d6b to
09226cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A uring worker that stopped on an error while the connection was up left its queues without entries. The kernel kept putting requests from those CPUs on them, and they waited forever, as did the requests the worker had taken in the same batch but not answered. The rest of the mount kept working, so nothing noticed.
The failing worker now ends the connection the way
umount -fdoes, by writing its fusectlabortfile. The kernel fails the waiting requests, gives the worker its entries back, and fails the other workers' commands. Every worker stops andFuseServing::wait()returns, so the application sees that the mount is gone. Only the first worker writes the file. A worker whose commands fail with ENOTCONN, ENODEV or ECANCELED leaves the connection alone, because it has ended already, for example through an unmount from outside, and its minor number may belong to a new connection by then.The worker finds the file from the mount's minor device number in
/proc/self/mountinfo, because astat()of the mountpoint would wait for a request that nobody serves yet. It escapes the mountpoint the way mountinfo does. Without fusectl it logs an error that asks for an unmount by hand.A second commit covers a request handler that panics. Unwinding used to drop the worker's ring entries while the kernel still held them and could copy the next request into the freed memory. The worker now catches the panic, ends the connection the same way, waits for the kernel to give the entries back, and then lets the panic go on.
One new test mounts over io_uring at a path with a space, ends the connection the same way, and checks that
wait()returns. Another serves a file system whose LOOKUP panics, and checks that the lookup fails instead of waiting and thatwait()returns. This builds on #280, which makes a stopping worker wait for the kernel to give its entries back.On Linux 7.3-rc7 the new tests ran over io_uring with fusectl, and the test suite passes in a user namespace on both async runtimes and with the uring feature, apart from the 14 passthrough tests that need real root and also fail on master.