[NEEDS #279, #281] fusedev: serve Linux 7.3 io_uring zero-copy reads and writes

CHANGELOG.md

@@ -37,6 +37,26 @@
 - [269](https://github.com/cloud-hypervisor/fuse-backend-rs/pull/269): `FuseSession::detach()` lazily unmounts a Linux session through `&self`,
   so the owner of a serving layer can start the teardown while another thread waits in
   `FuseServing::wait()`.
+- On Linux 7.3 and later, `UringFuseServing` gives each kernel queue a buffer pool of
+  `UringConfig::bufs_per_queue` payload buffers. The kernel lends them to the queue's ring entries
+  while they serve a request, so with fewer buffers than entries a mount needs less memory.
+  `bufs_per_queue` defaults to 4, the default number of entries, and counts as at most
+  `entries_per_queue`. 0 keeps a buffer per entry, as on older kernels. `UringConfig` has a new
+  field, so struct literals need `..Default::default()`.
+- `abi::fuse_uring` has the commands, INIT flag and fields of protocol 7.46 for buffer pools.
+  `Server::uring_bufpool_supported()` tells whether the kernel offered them at INIT, and
+  `Server::uring_payload_size()` returns the size of the kernel's payload buffers.
+- `OpenOptions::IO_URING_ZERO_COPY` and the zero-copy flags of protocol 7.46 in
+  `abi::fuse_uring` let a file system ask for FUSE-over-io_uring zero-copy on an open.
+  `Reader::with_zero_copy()` attaches request data that only the transport can move, through a
+  `ZeroCopySource`, and `Reader::read_to_at()` hands it over. `FileReadWriteVolatile::raw_fd()`
+  gives the transport the file descriptor to move it to. Other reads get a copy through memory.
+- `passthrough::Config::uring_zero_copy` asks for FUSE-over-io_uring zero-copy on the files that
+  `PassthroughFs` opens, except files it opens with O_DIRECT.
+- With `UringConfig::zero_copy` on Linux 7.3 and later, `UringFuseServing` serves the reads and
+  writes of opens that ask for it with zero-copy. It moves the data between the client's pages
+  and the file with io_uring fixed-buffer operations. It needs buffer pools and CAP_SYS_ADMIN,
+  and serves without it otherwise. `UringConfig` has a new field.
 
 ### Changed
 - Bump the declared minimum supported Rust version (MSRV) from 1.80 to 1.86.
@@ -49,6 +69,7 @@
 - The FUSE-over-io_uring transport answers opcodes it doesn't know with ENOSYS instead of EIO.
   Kernels that send `FUSE_STATX` only fall back to `FUSE_GETATTR` after ENOSYS, so `statx()`
   calls that ask for more than the basic fields, like the birth time `stat(1)` shows, failed.
+- `FuseUringEntInOut::padding` is now `offset`, and `FuseUringCmdReq` has a new `bufpool` field.
 - [254](https://github.com/cloud-hypervisor/fuse-backend-rs/pull/254): `fuse-backend-rs` is now a thin facade re-exporting the sub-crates. Every
   historical `fuse_backend_rs::{abi, api, buffer, common, transport, passthrough,
   overlayfs}` import path and cargo feature name (`fusedev`, `virtiofs`,
@@ -88,6 +109,13 @@
 - READDIR and READDIRPLUS answer ENOMEM when the reply can't hold the reply header plus the
   requested size. The check left out the header, so a writer short by up to 16 bytes started the
   listing and failed partway.
+- A `UringFuseServing` worker that stops frees its ring entries only after the kernel has given
+  them all back, and leaks them if that takes more than a second. A worker that stopped on an
+  error while the connection was up freed entries the kernel could still copy requests into. A
+  signal that interrupts the worker's `io_uring_enter()` calls no longer stops it.
+  signal that interrupts the worker's wait no longer stops it.
+- `UringWriter::write_from_at()` fails when the data doesn't fit into the reply body. It counted
+  free space in the header slot too, and could read file data past the end of the body.
 - [278](https://github.com/cloud-hypervisor/fuse-backend-rs/pull/278): Treat `EOPNOTSUPP`/`ENOTSUP` and `ENOSYS` from `getxattr` as "not opaque"
   in `Layer::is_opaque()`, so `OverlayFs::import()` no longer aborts when a layer's
   backing filesystem has no xattr support at all (tmpfs answers `EOPNOTSUPP` to

crates/fuse-backend-core/src/abi/fuse_abi_linux.rs

@@ -84,6 +84,10 @@ const FOPEN_CACHE_DIR: u32 = 8;
 /// the file is stream-like (no file position at all)
 const FOPEN_STREAM: u32 = 16;
 
+/// Serve reads and writes of this open file with FUSE-over-io_uring
+/// zero-copy, when the queue that serves them has it (Linux 7.3).
+const FOPEN_IO_URING_ZERO_COPY: u32 = 1 << 8;
+
 bitflags! {
     /// Options controlling the behavior of files opened by the server in response
     /// to an open or create request.
@@ -98,6 +102,15 @@ bitflags! {
         const CACHE_DIR = FOPEN_CACHE_DIR;
         /// the file is stream-like (no file position at all)
         const STREAM = FOPEN_STREAM;
+        /// Serve reads and writes of this open file with FUSE-over-io_uring
+        /// zero-copy, when the queue that serves them has it (Linux 7.3).
+        /// Other transports and queues copy the data as before. The data of
+        /// a zero-copy request stays in the client's pages, so the file
+        /// system must move it with `ZeroCopyWriter::write_from()` and
+        /// `ZeroCopyReader::read_to()`, into or out of a file that has a
+        /// descriptor (`FileReadWriteVolatile::raw_fd()`). Plain reads and
+        /// writes of it fail.
+        const IO_URING_ZERO_COPY = FOPEN_IO_URING_ZERO_COPY;
     }
 }
 

crates/fuse-backend-core/src/abi/fuse_uring.rs

@@ -1,12 +1,10 @@
 // Copyright (C) 2026 Alibaba Cloud. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
-//! Linux FUSE-over-io_uring ABI definitions, protocol version 7.42 (kernel 6.14+).
+//! Linux FUSE-over-io_uring ABI definitions, protocol version 7.42 (kernel 6.14+),
+//! and the buffer pools of protocol 7.46 (kernel 7.3).
 //!
-//! Experimental: the kernel interface is still evolving (reduced queue counts,
-//! bufpool/zero-copy extensions are being developed upstream). Only the minimal
-//! 7.42 interface is defined here; in-flight extensions are intentionally left
-//! out.
+//! Experimental: the kernel interface is still evolving.
 
 #![allow(missing_docs)]
 
@@ -15,6 +13,9 @@ use vm_memory::ByteValued;
 /// INIT flag: client supports serving FUSE requests over io_uring (protocol 7.42).
 pub const FUSE_OVER_IO_URING: u64 = 1_u64 << 41;
 
+/// INIT flag the kernel sets when it supports io_uring buffer pools (protocol 7.46).
+pub const FUSE_HAS_IO_URING_BUFPOOL: u64 = 1_u64 << 43;
+
 /// Size of the fuse_in_header/fuse_out_header slot inside `FuseUringReqHeader`.
 pub const FUSE_URING_IN_OUT_HEADER_SZ: usize = 128;
 
@@ -34,6 +35,22 @@ pub const FUSE_URING_IOV_PAYLOAD: usize = 1;
 pub const FUSE_IO_URING_CMD_INVALID: u32 = 0;
 pub const FUSE_IO_URING_CMD_REGISTER: u32 = 1;
 pub const FUSE_IO_URING_CMD_COMMIT_AND_FETCH: u32 = 2;
+/// Create a queue (protocol 7.46). Buffer pools need it.
+pub const FUSE_IO_URING_CMD_ADD_QUEUE: u32 = 3;
+/// Give a queue a buffer pool (protocol 7.46). The queue must exist and must
+/// not have registered entries with their own payload buffers yet.
+pub const FUSE_IO_URING_CMD_ADD_BUFPOOL: u32 = 4;
+
+/// `FuseUringCmdReq::flags` of ADD_QUEUE: serve the queue's reads and writes
+/// with zero-copy (protocol 7.46). The kernel then registers the client's
+/// pages of such a request in the server's io_uring buffer table, at the
+/// index the entry gave at REGISTER, and the server moves the data with
+/// fixed-buffer operations. Needs a buffer pool and CAP_SYS_ADMIN.
+pub const FUSE_URING_ZERO_COPY: u64 = 1;
+
+/// `FuseUringEntInOut::flags` of a request whose pages the kernel
+/// registered for zero-copy instead of copying them (protocol 7.46).
+pub const FUSE_URING_ENT_ZERO_COPY: u64 = 1;
 
 /// Shared in/out descriptor of one ring entry, embedded in `FuseUringReqHeader`.
 #[repr(C)]
@@ -44,7 +61,9 @@ pub struct FuseUringEntInOut {
     pub commit_id: u64,
     /// Size of the payload in bytes (request: set by kernel, reply: set by daemon).
     pub payload_sz: u32,
-    pub padding: u32,
+    /// Offset of the request's payload buffer in the queue's buffer pool, if
+    /// the queue has one (protocol 7.46). Before that it was padding.
+    pub offset: u32,
     pub reserved: u64,
 }
 
@@ -72,8 +91,9 @@ impl Default for FuseUringReqHeader {
 /// Command payload of an uring command SQE, placed in the 80-byte command
 /// area of an SQE128.
 ///
-/// Layout mirrors `struct fuse_uring_cmd_req`. For REGISTER and
-/// COMMIT_AND_FETCH only `commit_id` and `qid` are used.
+/// Layout mirrors `struct fuse_uring_cmd_req`. REGISTER and
+/// COMMIT_AND_FETCH use `commit_id` and `qid`, ADD_QUEUE uses `qid`, and
+/// ADD_BUFPOOL uses `qid` and `bufpool`.
 #[repr(C)]
 #[derive(Clone, Copy, Debug, Default)]
 pub struct FuseUringCmdReq {
@@ -83,13 +103,43 @@ pub struct FuseUringCmdReq {
     /// Index of the per-CPU queue the command is for.
     pub qid: u16,
     pub padding: [u8; 6],
+    /// The buffer pool of an ADD_BUFPOOL command (protocol 7.46).
+    pub bufpool: FuseUringBufpool,
+}
+
+/// Buffer pool of a queue, `fuse_uring_cmd_req.bufpool`. The kernel cuts it
+/// into buffers of the connection's maximum payload size.
+#[repr(C)]
+#[derive(Clone, Copy, Debug, Default)]
+pub struct FuseUringBufpool {
+    /// Address of the pool in the daemon's memory.
+    pub uaddr: u64,
+    /// Length of the pool in bytes.
+    pub len: u32,
+    pub reserved: u32,
+}
+
+impl FuseUringCmdReq {
+    /// Set `ent_zero_copy_buf_index` of a REGISTER command for a queue with
+    /// zero-copy: the index in the server's io_uring buffer table where the
+    /// kernel registers the client's pages of the entry's requests. The
+    /// kernel puts it in a union with `bufpool`, at the same offset.
+    pub fn set_ent_zero_copy_buf_index(&mut self, index: u16) {
+        let mut uaddr = [0u8; 8];
+        uaddr[..2].copy_from_slice(&index.to_ne_bytes());
+        self.bufpool = FuseUringBufpool {
+            uaddr: u64::from_ne_bytes(uaddr),
+            ..Default::default()
+        };
+    }
 }
 
-// SAFETY: all three structs are `repr(C)` plain data types without padding
+// SAFETY: all four structs are `repr(C)` plain data types without padding
 // holes or pointer members, matching their kernel counterparts.
 unsafe impl ByteValued for FuseUringEntInOut {}
 unsafe impl ByteValued for FuseUringReqHeader {}
 unsafe impl ByteValued for FuseUringCmdReq {}
+unsafe impl ByteValued for FuseUringBufpool {}
 
 #[cfg(test)]
 mod tests {
@@ -99,7 +149,9 @@ mod tests {
     fn test_layouts() {
         assert_eq!(std::mem::size_of::<FuseUringEntInOut>(), 32);
         assert_eq!(std::mem::size_of::<FuseUringReqHeader>(), 288);
-        assert_eq!(std::mem::size_of::<FuseUringCmdReq>(), 24);
+        assert_eq!(std::mem::size_of::<FuseUringCmdReq>(), 40);
+        assert_eq!(std::mem::offset_of!(FuseUringCmdReq, bufpool), 24);
+        assert_eq!(std::mem::offset_of!(FuseUringEntInOut, offset), 20);
         assert_eq!(
             std::mem::offset_of!(FuseUringReqHeader, ring_ent_in_out),
             FUSE_URING_IN_OUT_HEADER_SZ + FUSE_URING_OP_IN_OUT_SZ

crates/fuse-backend-core/src/api/server/mod.rs

@@ -21,6 +21,8 @@ use std::io::{self, Read};
 use std::marker::PhantomData;
 use std::mem::size_of;
 use std::sync::atomic::AtomicU64;
+#[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+use std::sync::atomic::{AtomicBool, AtomicU32};
 
 use crate::abi::fuse_abi::*;
 use crate::api::filesystem::{Context, FileSystem, ZeroCopyReader, ZeroCopyWriter};
@@ -65,6 +67,15 @@ pub struct Server<F: FileSystem + Sync> {
     /// Capability flags actually enabled by the INIT exchange.
     #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
     negotiated_init_flags: AtomicU64,
+    /// Whether the kernel offered io_uring buffer pools in INIT.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    uring_bufpool: AtomicBool,
+    /// `max_write` of the INIT reply.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    init_max_write: AtomicU32,
+    /// `max_pages` of the INIT reply, 0 if it doesn't set `FUSE_MAX_PAGES`.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    init_max_pages: AtomicU32,
 }
 
 impl<F: FileSystem + Sync> Server<F> {
@@ -78,6 +89,12 @@ impl<F: FileSystem + Sync> Server<F> {
             extra_init_flags: AtomicU64::new(0),
             #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
             negotiated_init_flags: AtomicU64::new(0),
+            #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+            uring_bufpool: AtomicBool::new(false),
+            #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+            init_max_write: AtomicU32::new(0),
+            #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+            init_max_pages: AtomicU32::new(0),
         }
     }
 
@@ -104,6 +121,51 @@ impl<F: FileSystem + Sync> Server<F> {
         self.negotiated_init_flags.load(Ordering::Acquire) & FsOptions::OVER_IO_URING.bits() != 0
     }
 
+    /// Report whether the kernel offered buffer pools for FUSE-over-io_uring
+    /// in INIT (`FUSE_HAS_IO_URING_BUFPOOL`, Linux 7.3). Returns false before
+    /// INIT completes.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    pub fn uring_bufpool_supported(&self) -> bool {
+        self.uring_bufpool.load(Ordering::Acquire)
+    }
+
+    /// The size of the payload buffer the kernel needs for one
+    /// FUSE-over-io_uring request on this connection, and the size of each
+    /// buffer it cuts from a buffer pool. It follows `fuse_uring_create()`:
+    /// the largest of `FUSE_MIN_READ_BUFFER`, `max_write` and `max_pages`
+    /// pages. The kernel can lower `max_pages` to its `max_pages_limit`
+    /// sysctl, but the INIT reply sets `max_write` to at least `max_pages`
+    /// pages, so that doesn't change the size. Meaningful only after INIT.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    pub fn uring_payload_size(&self) -> usize {
+        // The defaults of process_init_reply() and fuse_conn_init().
+        const MIN_MAX_WRITE: usize = 4096;
+        const DEFAULT_MAX_PAGES: usize = 32;
+        let max_write = self.init_max_write.load(Ordering::Acquire) as usize;
+        let max_pages = match self.init_max_pages.load(Ordering::Acquire) as usize {
+            0 => DEFAULT_MAX_PAGES,
+            n => n,
+        };
+        (MIN_READ_BUFFER as usize)
+            .max(max_write.max(MIN_MAX_WRITE))
+            .max(max_pages * crate::buffer::pagesize())
+    }
+
+    /// Remember what FUSE-over-io_uring needs from INIT: whether the kernel
+    /// offered buffer pools in `kernel_flags`, and the `max_write` and
+    /// `max_pages` of the reply.
+    #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+    fn record_uring_init(&self, kernel_flags: u64, max_write: u32, max_pages: u16) {
+        use crate::abi::fuse_uring::FUSE_HAS_IO_URING_BUFPOOL;
+        self.uring_bufpool.store(
+            kernel_flags & FUSE_HAS_IO_URING_BUFPOOL != 0,
+            Ordering::Release,
+        );
+        self.init_max_write.store(max_write, Ordering::Release);
+        self.init_max_pages
+            .store(u32::from(max_pages), Ordering::Release);
+    }
+
     /// OR the extra INIT flags requested through `set_uring()` into the
     /// enabled capability set and remember the negotiation outcome.
     #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]

crates/fuse-backend-core/src/api/server/sync_io.rs

@@ -977,6 +977,8 @@ impl<F: FileSystem + Sync> Server<F> {
                     out.max_pages = MAX_REQ_PAGES;
                     out.max_write = MAX_REQ_PAGES as u32 * pagesize() as u32; // 1MB
                 }
+                #[cfg(all(target_os = "linux", feature = "fusedev-uring"))]
+                self.record_uring_init(flags_u64, out.max_write, out.max_pages);
                 self.vers.store(
                     encode_version(version.major, version.minor),
                     Ordering::Release,
@@ -1726,10 +1728,10 @@ mod tests {
     use super::super::test_util::{MockFS, TestWriter};
     use super::*;
 
-    fn prepare_srvcontext<'a>(
+    fn prepare_srvcontext<'a, F: FileSystem>(
         read_buf: &'a mut [u8],
         write_buf: &'a mut [u8],
-    ) -> SrvContext<'a, MockFS, (), TestWriter<'a>> {
+    ) -> SrvContext<'a, F, (), TestWriter<'a>> {
         let reader = Reader::<()>::from_slice(read_buf);
         let writer = TestWriter::new(write_buf);
         let in_header = InHeader::default();
@@ -2019,6 +2021,77 @@ mod tests {
         );
     }
 
+    #[cfg(feature = "fusedev-uring")]
+    #[test]
+    fn test_server_init_uring_bufpool() {
+        use crate::abi::fuse_uring::FUSE_HAS_IO_URING_BUFPOOL;
+
+        /// A file system that wants everything the kernel offers.
+        struct WantAll;
+        impl FileSystem for WantAll {
+            type Inode = u64;
+            type Handle = u64;
+            fn init(&self, capable: FsOptions) -> io::Result<FsOptions> {
+                Ok(capable)
+            }
+        }
+
+        // Run INIT with `flags` and `flags2` from the kernel, and return the
+        // server and the reply.
+        let init = |flags: FsOptions, flags2: u64| {
+            let mut read_buf = [0u8; size_of::<InitIn>() + size_of::<InitIn2>()];
+            let init_in = InitIn {
+                major: KERNEL_VERSION,
+                minor: KERNEL_MINOR_VERSION,
+                max_readahead: 0,
+                flags: (flags | FsOptions::INIT_EXT).bits() as u32,
+            };
+            read_buf[..size_of::<InitIn>()].copy_from_slice(init_in.as_slice());
+            let init_in2 = InitIn2 {
+                flags2: ((FsOptions::OVER_IO_URING.bits() | flags2) >> 32) as u32,
+                unused: [0; 11],
+            };
+            read_buf[size_of::<InitIn>()..].copy_from_slice(init_in2.as_slice());
+            let server = Server::new(WantAll);
+            server.set_uring(true);
+            let mut write_buf = [0u8; 4096];
+            let res = {
+                let ctx = prepare_srvcontext(&mut read_buf, &mut write_buf);
+                server.init(ctx, |_| {}).unwrap()
+            };
+            let mut out = InitOut::default();
+            out.as_mut_slice()
+                .copy_from_slice(&write_buf[size_of::<OutHeader>()..res]);
+            (server, out)
+        };
+
+        // The kernel's flag for buffer pools isn't an FsOptions flag, but
+        // the server still sees it.
+        let (server, _) = init(FsOptions::empty(), FUSE_HAS_IO_URING_BUFPOOL);
+        assert!(server.uring_bufpool_supported());
+        let (server, out) = init(FsOptions::empty(), 0);
+        assert!(!server.uring_bufpool_supported());
+        // Without FUSE_MAX_PAGES the kernel keeps 32 pages per request.
+        assert_eq!(out.max_pages, 0);
+        assert_eq!(
+            server.uring_payload_size(),
+            (MIN_READ_BUFFER as usize)
+                .max(out.max_write as usize)
+                .max(32 * pagesize())
+        );
+
+        // With FUSE_MAX_PAGES the kernel takes max_pages up to its limit.
+        let (server, out) = init(FsOptions::MAX_PAGES | FsOptions::BIG_WRITES, 0);
+        assert_eq!(out.max_pages, MAX_REQ_PAGES);
+        assert_eq!(
+            server.uring_payload_size(),
+            MAX_REQ_PAGES as usize * pagesize()
+        );
+        // The kernel's max_pages_limit can't change the size, because the
+        // reply's max_write covers max_pages pages.
+        assert!(out.max_write as usize >= MAX_REQ_PAGES as usize * pagesize());
+    }
+
     #[cfg(feature = "fusedev-uring")]
     #[test]
     fn test_server_init_uring_negotiation() {