I/O imbalance across bookies

[Jayer23]

Description

We observed an abnormal write pattern in our Pulsar cluster:

The nodes exposed in bookie metrics have little difference in read and write io. Producer has compressed write enabled.

In fact, monitoring at the node level found that io was greatly magnified, and the io differences between nodes were large. Some nodes had io less than 50MB/s, while some nodes had io greater than 450MB/s.

Journal disks on some nodes show very high write throughput (e.g., 400+ MB/s).

Environment

• Pulsar version: 3.0.7
• Storage config:
  • Journal: 3×SSD
  • Ledger：9×HDD
• Replication: 3 replicas, 2 quorum writes

Expected Behavior

Write operations should be balanced between bookies