Skip to content

[NEEDS #270] Make file IO on the tokio runtime block less - #271

Open
Kranzes wants to merge 15 commits into
cloud-hypervisor:masterfrom
Kranzes:async-tokio-io
Open

Kranzes wants to merge 15 commits into
cloud-hypervisor:masterfrom
Kranzes:async-tokio-io

Conversation

@Kranzes

@Kranzes Kranzes commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

NEEDS #270. This branch is stacked on #270, which is stacked on #267. The commits of this PR start at "benchmark: measure vectored IO and the passthrough thread pool".

On the tokio runtime, every async file operation is a blocking preadv() or pwritev() on the runtime thread. That runtime serves macOS, Linux without io_uring, and applications that drive FuseDevTask from their own tokio runtime (#267). This makes those calls fewer, and moves the ones that block onto the blocking thread pool when PassthroughFs::enable_async_thread_pool(true) is set.

  • benchmark: measure vectored IO and the passthrough thread pool. async_io_microbench times vectored writes and reads of 1, 16 and 256 page-sized buffers, and PassthroughFs READ and WRITE with the pool off and on. The daemon gets --thread-pool, and bench_sync_async.sh gets a pool mode.
  • async-io: read and write vectored buffers with one syscall on tokio. The vectored file methods split the buffers into one operation each, which adds no concurrency on tokio. A virtio-fs WRITE of 1 MiB in 256 pages made 256 pwritev() calls. They now make one call per 1024 buffers. io_uring keeps the split. The new RuntimeType::file_io_blocks() tells callers whether file IO blocks the calling thread.
  • passthrough: serve blocking READ and WRITE on the thread pool. With the pool enabled, READ cache misses and WRITEs on tokio, and the O_DIRECT and WRITE_KILL_PRIV requests on any runtime, run the sync handlers on a pool thread. The pool thread works on its own copy of the data and never touches the transport buffer. A task that hasn't started when its request is dropped, for example for a takeover, doesn't run.
  • passthrough: don't make LSEEK wait for READ and WRITE. lseek() takes the handle's read lock instead of its write lock, so an LSEEK no longer waits for a pooled READ or WRITE on the same handle. It also drops the handle's cached readdir cookie, which a seek makes stale.
  • passthrough: look up the handle once for a pooled READ. A READ that misses the page cache passes the handle data it looked up for the inline attempt to the pool. In no_open mode that saves one open() per READ.
  • passthrough: copy O_DIRECT data once on the thread pool. The pool's buffer starts at a page boundary, so a pooled O_DIRECT READ or WRITE does its IO on it directly, without the sync handler's bounce buffer. Its allocation also fails with ENOMEM where the WRITE payload's Vec used to abort.
  • passthrough: undo pooled LOOKUP, OPEN and CREATE that nobody receives. If a request is dropped while a pool thread serves its LOOKUP, OPEN or CREATE, the handle or lookup count it creates never reaches the kernel. The pool task now returns its result in a guard that releases the handle or forgets the lookup if nobody claims it.

Measured on an AMD Ryzen AI 9 HX 370 with Linux 7.2.7, one worker on tokio, backing files on ext4:

before after
256 × 4 KiB vectored write (cached) 144 µs 22.5 µs
256 × 4 KiB vectored read (cached) 67 µs 20 µs
reading a file while another READ waits in a slow FUSE file system below, pool on 2 s 1.6 ms
fio random 4 KiB reads, 4 jobs, cold cache, pool on 12.1K IOPS 34.5K IOPS
fio random 4 KiB writes, cached, pool on 433K IOPS 424K IOPS

The pool costs one copy and a thread handoff per request, about 3.5 µs for 4 KiB. A pooled 4 KiB O_DIRECT READ takes 23.4 µs instead of 24.7 µs with two copies, and a pooled buffered 1 MiB READ 37.8 µs instead of 43.2 µs.

Known limitations, which I left for later:

  • READ cache hits stay inline only on file systems with RWF_NOWAIT. On tmpfs, overlayfs and FUSE every READ goes to the pool.
  • A pool task that already runs when its request is dropped completes. The kernel may send the request again after a takeover, so a WRITE to an O_APPEND file can land twice, and an O_EXCL CREATE fails with EEXIST because the first one's file stays. Stopping the serving task through its state flag avoids that, and the docs say so.
  • A READ or WRITE that flips the handle's O_DIRECT flag holds the write lock for its whole IO, so an LSEEK on that handle still waits for it.

The crate detects the runtime type on first use, and only the
FUSE_BACKEND_RS_ASYNC_RUNTIME environment variable overrides it. A
daemon that drives FuseDevTask from its own tokio runtime has to pick
tokio, because that runtime can't poll tokio-uring files. Setting the
variable from the daemon needs std::env::set_var, which is unsafe once
other threads may read the environment.

Add async_runtime::set_runtime_type() to select the type in code. It
overrides the variable and fails if the type in effect is a different
one. Add async_runtime::runtime_type() to read the type, and make
RuntimeType public for both.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
With the tokio runtime, FuseDevTask reads the nonblocking fuse device
with plain preadv(). While no request is pending, the read fails with
EAGAIN, and poll_handler() yields and reads again right away, so an
idle task keeps a CPU busy.

Register the fd with the runtime's IO driver and wait until it is
readable before each read and after EAGAIN. Wait for errors too,
because the kernel reports a torn down connection with EPOLLERR alone.
A 100ms timer wakes the wait, so an idle task still returns once its
state flag is set. Without the timer, stopping a worker whose unmount
fails would hang. An error that isn't an OS error, such as the runtime
shutting down, ends the task. The io_uring runtime is unchanged.
AsyncFd::register_with_interest() needs tokio 1.53.3, which deprecates
with_interest().

Registering panics outside of a tokio runtime, so a task that another
executor polls keeps retrying reads as before. The docs say that
dropping poll_handler() to stop an idle task is only safe with the
tokio runtime. tokio-uring leaves its read pending in the kernel,
which then copies the next request into the freed buffer.

One test checks that an idle task uses little CPU and returns on
unmount. Before this change it used all of the 500ms it idled. Another
checks that the state flag stops an idle task while the mount stays
up. A third serves a mount without any tokio runtime.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
A daemon can hand its FUSE device fds to a new process, which then
serves the connection without a FUSE_INIT. The new Server can't learn
what the old one negotiated, so it parses requests as if no option
were enabled, such as the FUSE_EXT_GROUPS extension of create
requests.

Add Server::init_state() to read the protocol version and the options
enabled at INIT, and Server::restore_init() to set them on the new
server. Add abi::fuse_abi::UNIQUE_RESEND, the bit the kernel sets in
requests that notify_resend() makes it send again. The server already
replies with the unique id unchanged, and a test now covers that.

Document that a thread that serves requests must not call
notify_inval_entry(). It waits for the parent's lock, which a lookup
in that directory holds until its reply.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Add tests that serve a mount with FuseDevTask on a current-thread
tokio runtime, from a file system that can make a lookup wait. They
cover two cases:

- A waiting lookup doesn't hold up another lookup in the same
  directory, and notify_inval_entry() from another thread works while
  the task serves.
- A second server takes over the connection from one that stopped with
  a lookup in flight. It restores the INIT state, sends
  notify_resend() and answers the lookup that the kernel sends again.

The file system enables FUSE_PARALLEL_DIROPS, because without it the
kernel serializes lookups in a directory.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
The async server logged a read error it didn't know and read again.
Now that a tokio task waits for the device to become readable, a device
that stays readable but fails every read made the task spin at full CPU
and log on every pass. EINVAL for a buffer smaller than the kernel's
requests does that.

Retry ENOENT, which the kernel returns when a request was interrupted
before the read, and stop the task on any other error. The synchronous
server does the same.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
AsyncFileReadWriteVolatile used tokio_uring only for
IoBuf::bytes_total(), so call FileVolatileBuf::cap() instead. The vfs
async handlers used the Linux-only libc::stat64 and
VfsOptions::no_open, so use fuse_abi::stat64 and check no_open on
Linux only.

The umbrella crate now forwards async-io to the transports with
pkg?/feature, so async-io alone no longer pulls in the Linux-only
virtiofs transport.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
async-io failed to build on macOS, because it pulled in io-uring on
every platform and io-uring only builds on Linux. The code that uses
io_uring was already Linux-only.

Make io-uring a Linux-only dependency, like tokio-uring. Other
platforms run async-io on tokio. The macfuse example gets the
AsyncFileSystem impl that async-io requires, and the Makefile builds
and tests async-io on macOS.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
When the io-uring probe failed, set_runtime_type(RuntimeType::Uring)
returned "async runtime type is Tokio, not Uring". The probe only
logged its reason, so an application couldn't tell a low
RLIMIT_MEMLOCK from a seccomp filter.

Keep the probe's error and return it from set_runtime_type(), with its
kind. The fallback warning includes it too. The test blocks
io_uring_setup() with a seccomp filter, and skips where it can't
install one, such as under QEMU.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
async_io_microbench times writes and reads between a file and 1, 16 or
256 page-sized buffers, as a virtio-fs descriptor chain hands them
over. It also times PassthroughFs READ and WRITE of 4 KiB, 128 KiB and
1 MiB with the blocking thread pool off and on. Its READ reply buffer
refuses the inline page cache attempt, so READ measures the cache miss
path.

The daemon gets a --thread-pool flag for the async mode, and
bench_sync_async.sh gets a pool mode that passes it. Both
micro-benchmarks share their PassthroughFs setup in benches/common.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
The tokio runtime serves macOS, Linux without io_uring, and
applications that select it. On it, every async file operation is a
blocking preadv() or pwritev(). Still,
async_read_vectored_at_volatile() and
async_write_vectored_at_volatile() split the buffers into one
operation each, which adds no concurrency. A virtio-fs WRITE of 1 MiB
in 256 pages makes 256 pwritev() calls.

Make one call for all buffers, unless the file goes through io_uring,
which keeps the split. preadv() and pwritev() make one call per 1024
buffers, the most one call takes on Linux and macOS. The new
RuntimeType::file_io_blocks() tells callers such as PassthroughFs
whether file IO blocks the calling thread.

In async_io_microbench on tokio, with the data in the page cache, 256
buffers of 4 KiB take 22.5 µs instead of 144 µs to write and 20 µs
instead of 67 µs to read. 16 buffers take 1.4 µs instead of 5.0 µs to
write and 1.3 µs instead of 4.0 µs to read. With the page cache
dropped, reading 16 buffers takes 549 µs instead of 645 µs.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
On the tokio runtime, PassthroughFs serves READ cache misses and every
WRITE with a blocking preadv() or pwritev() on the runtime thread,
even with enable_async_thread_pool(). A slow backing file then stalls
every other request of the worker. The O_DIRECT and WRITE_KILL_PRIV
requests that it relays to the synchronous handlers block the runtime
thread on any runtime.

With the pool enabled, serve these requests with the synchronous
handlers on a pool thread. The pool thread never touches the transport
buffer, because the request frees it when its future is dropped, for
example for a takeover. A READ reads into a buffer that the pool task
owns, and the runtime thread copies it into the reply. A READ larger
than the reply buffer fails before the task allocates. A WRITE copies
its payload out of the request before the task starts. A task that no
pool thread has started when its request is dropped doesn't run, so
the old copy of a WRITE can't overwrite newer data after a takeover.

Measured with one worker on the tokio runtime and the backing files on
ext4:
- With a READ waiting in a FUSE file system below, reading another
  file through the mount takes 1.6 ms instead of 2 s.
- fio random 4 KiB reads with 4 jobs and a cold page cache reach 34.5K
  IOPS instead of 12.1K.
- Cached fio random 4 KiB writes reach 424K IOPS instead of 433K. The
  copy and the thread handoff add about 3.5 µs to a 4 KiB READ miss or
  WRITE.

READ cache hits stay inline on file systems with RWF_NOWAIT. On
others, such as tmpfs, overlayfs and FUSE, every READ counts as a
miss. io_uring keeps its native path for buffered READ and WRITE.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
lseek() took the handle's write lock, which READ and WRITE hold as
readers for the length of their IO. With the thread pool, an LSEEK on
a handle with a pooled READ or WRITE in flight blocked the runtime
thread until the IO ended.

Take the read lock instead. READ and WRITE use positioned IO, so the
file position doesn't matter to them. do_readdir() still seeks and
reads a directory under the write lock, so an lseek() can't move the
position in between. lseek() now also drops the handle's cached
readdir cookie, which a seek makes stale. The kernel sends LSEEK only
for regular files, so both only guard against other clients.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
async_read() looks up the handle data for its inline RWF_NOWAIT
attempt. On a miss it ran read() on the pool, which looked the handle
up again. In no_open mode a lookup opens the file, so a pooled READ
opened it twice.

Move the body of read() into do_read(), which takes the handle data,
and pass the data that async_read() already has to the pool. An
O_DIRECT READ now looks up its handle in async_read() too, so in
no_open mode it opens the file on the runtime thread, as a buffered
READ already did.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
A pooled O_DIRECT READ read into the synchronous handler's page-aligned
bounce buffer, copied that into the pool task's buffer, and the runtime
thread then copied it into the reply. A pooled O_DIRECT WRITE took the
same two copies the other way.

Start the pool task's buffer at a page boundary, and let do_read() and
the new do_write() do the direct IO on it, without the bounce buffer.
Like read_direct() and write_direct(), they read until EOF and write
the whole payload. The buffer is a Vec that is one page larger than
needed. Allocating it with posix_memalign() instead made a pooled
1 MiB READ four times slower. If the allocation fails, the request gets
ENOMEM, where the WRITE payload's Vec used to abort.

In async_io_microbench on tokio, with O_DIRECT cases added locally, a
pooled O_DIRECT READ of 4 KiB takes 23.4 instead of 24.7 µs and a
WRITE 20.2 instead of 22.1 µs. Device IO dominates the larger sizes.
A buffered 1 MiB READ on the pool takes 37.8 instead of 43.2 µs.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
If a request future is dropped while a pool thread serves its LOOKUP,
OPEN or CREATE, for example for a takeover, the pool thread still
creates a lookup count or a handle that the kernel never learns of.
OPEN leaks the handle and its fd. LOOKUP leaks a lookup count, which
keeps the inode and its O_PATH fd until the file system is dropped.
CREATE leaks both.

The pool task now returns its result in a guard that undoes it if
nobody claims it. OPEN releases the handle, LOOKUP forgets the lookup
count, and CREATE does both. The JoinHandle hands the result back, so
the undo runs on the pool thread if the request is gone when the task
ends, and otherwise on the thread that drops the request.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant