fusedev: take uring payload buffers from kernel buffer pools

On Linux 7.3 and later, UringFuseServing gives each kernel queue a pool
of UringConfig::bufs_per_queue buffers before any entry registers. The
kernel lends a buffer to an entry while it serves a request with a
payload or a reply body, and reports where the buffer starts. Pooled
entries keep only their header and scratch area. Kernels that don't
offer pools at INIT keep a buffer per entry, as does bufs_per_queue = 0.
It defaults to 4, the default number of entries, and counts as at most
entries_per_queue.

A request without a payload and without out arguments gets no buffer,
and its offset of 0 looks like the first buffer. The opcode table now
records which requests have out arguments, and the others get an empty
reply body, so they can't write into another entry's buffer.

The benchmark daemon takes --uring-bufs N.

The benchmark daemon served a passthrough directory of 24 files of
256 MiB with --uring, 4 workers and 16 entries per queue, on a 24-CPU
machine with Linux 7.3-rc7. fio ran random 1 MiB and 4 KiB direct reads
with libaio at depth 16, one job pinned to each CPU, for 8 seconds,
through a fresh mount each run. The two settings alternated three times:

  buffers per queue  VmSize    VmRSS    1 MiB reads       4 KiB reads
  own per entry      1492 MiB  393 MiB  1518-1728 MiB/s   45.9-46.8k IOPS
  4                  431 MiB   106 MiB  1588-1709 MiB/s   44.5-45.7k IOPS

VmSize also counts each entry's request and reply scratch areas, which
are as large as its own payload buffer, so pooled entries save those
too.

Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>

Author: Kranzes

CHANGELOG.md

@@ -29,6 +29,12 @@
   FUSE-over-io_uring transport (kernel 6.14+). Both join the `FuseServing` interface, so a daemon
   can pick its transport at runtime behind `Box<dyn FuseServing>` and fall back to the classic
   transport on `Error::UringNotSupported`.
+- On Linux 7.3 and later, `UringFuseServing` gives each kernel queue a buffer pool of
+  `UringConfig::bufs_per_queue` payload buffers. The kernel lends them to the queue's ring entries
+  while they serve a request, so with fewer buffers than entries a mount needs less memory.
+  `bufs_per_queue` defaults to 4, the default number of entries, and counts as at most
+  `entries_per_queue`. 0 keeps a buffer per entry, as on older kernels. `UringConfig` has a new
+  field, so struct literals need `..Default::default()`.
 - `abi::fuse_uring` has the commands, INIT flag and fields of protocol 7.46 for buffer pools.
   `Server::uring_bufpool_supported()` tells whether the kernel offered them at INIT, and
   `Server::uring_payload_size()` returns the size of the kernel's payload buffers.

crates/fuse-backend-fusedev/Cargo.toml

@@ -31,7 +31,7 @@ log = "0.4.6"
 mio = { version = "0.8", features = ["os-poll", "os-ext"] }
 nix = "0.24"
 tokio = { version = "1", optional = true }
-vm-memory = { workspace = true }
+vm-memory = { workspace = true, features = ["backend-mmap"] }
 vmm-sys-util = { workspace = true }
 
 [target.'cfg(target_os = "macos")'.dependencies]