Skip to content

RocksDB L5+L6 compaction may require more than ~192 GiB free disk space due to high read/write amplification #5164

@eval-exec

Description

@eval-exec

Bug Report

Current Behavior

A ckb v0.205.0 testnet node repeatedly hit RocksDB compaction errors during a large L5 + L6 compaction.

Relevant logs:

2026/04/03-12:11:49.105337 51 [ERROR] [file/sst_file_manager_impl.cc:191] free space [196756488192 bytes] is less than needed headroom [206219919386 bytes]
2026/04/03-12:11:49.105408 51 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3159] Waiting after background compaction error: Compaction too large: , Accumulated
background error counts: 50193

[JOB 15] Compacting 1@5 + 3144@6 files to L6, score 0.41

2026/04/03-14:02:40.814464 51 [ERROR] [file/sst_file_manager_impl.cc:191] free space [196766220288 bytes] is less than needed headroom [206244698514 bytes]
2026/04/03-14:02:40.814552 51 [db/db_impl/db_impl_compaction_flush.cc:51] Cancelled compaction because not enough room
2026/04/03-14:02:40.814558 51 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3159] Waiting after background compaction error: Compaction too large: , Accumulated
background error counts: 508

This appears to be a compaction for column family 2 (COLUMN_BLOCK_BODY). During this period, the node kept generating new .sst files and disk usage grew very quickly.

From the logs, RocksDB needed about 206244698514 bytes of free space, which is about 192 GiB, while available free space was about 196766220288 bytes, about 183 GiB.

Expected Behavior

Environment

  • CKB version: v0.205.0
  • Chain: testnet
  • Operating system: ubuntu
  • Arch: x86_64
  • Installation: [GitHub Release, Built from source]

Metadata

Metadata

Assignees

Labels

t:bugType: This doesn't seem right.

Type

No type

Projects

Status

🏗 In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions