Skip to content

[Bug] get_all_entries reads full payload of every file at startup, causing OOM on large stores #569

@dylansomana

Description

@dylansomana

Describe the bug

Hi! We were trying to use the filesystem backend as a local on-disk cache for camera frame data — frames published as zenoh PUTs at ~30 fps, with a 48 h retention janitor pruning old files. Things worked fine for a couple of days while the cache was small, then zenohd started OOMing on every restart.

I dug into the source to figure out what was going on, and I think I found the culprit. When storage_manager starts a storage, it calls get_all_entries() to seed its latest_updates map. get_all_entries walks every file and calls read_file() on each — but read_file does a full read_to_end() of the payload, even though get_all_entries only needs the timestamp and discards the payload and encoding. You end up storing the entire cache - for every payload - in memory all at once and zenohd crashes.

get_all_entries only needs the timestamp and it's already available without reading the payload via data_info_mgr.get_encoding_and_timestamp(). Switching get_all_entries to that
path should fix the issue. Happy to put in a PR if y'all agree with the approach.

To reproduce

  1. Configure zenohd with storage_manager and a filesystem-backend storage, e.g.:
{
  "plugins": {
    "storage_manager": {
      "volumes": { "fs": {} },
      "storages": {
        "cache": {
          "key_expr": "demo/cache/**",
        }
      }
    }
  }
}
  1. Publish enough data under demo/cache/** to exceed the host's available memory.
  2. Restart zenohd.

Expected: zenohd starts up and serves queries against demo/cache/** from disk.

Actual: zenohd hangs after starting the storage manager and the process is OOM-killed.

System info

  • Platform: NixOS 25.11 on aarch64
  • zenohd: 1.9.0
  • Store size at time of crash: ~130 GB, ~1.3M files
  • Available host memory: 64 GB

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions