kvrocks unresponsive during scheduled compaction

[svenwltr]

Hello!

We are currently having issues with compaction, because KVrocks as a whole gets unresponsive during compaction. All clients using KVrocks report connection and read timeouts during the compaction. Our usage pattern might matter in this case, because we update most of the keys during the day.

We are looking for ways to tune the compaction. It wouldn't matter much, when the latency is elevated for a longer time, but KVrocks should say responsive.

Any ideas how to achieve this?

Metrics

Here are some metrics from tonight:

The CPU and memory usage gets reported by Kubernetes, while the others are from the kvrocks-exporter. There is a gap in the exporter metrics during the compaction, and the exporter is full of level=error msg="Couldn't connect to KVrocks instance" messages during that time.

Logs

The compaction starts with this message:

compaction_checker.cc:137] [compaction checker] Going to compact the key in file: /kvrocks_data/db/777334.sst, delete ratio: 0.297214

And after the compaction is done, we get a lot of broken pipes:

Going to remove the client: 172.20.198.110:41252, while encounter error: Broken pipe, SSL Error: error:00000000:lib(0)::reason(0)

Full logs during compaction are available here: https://gist.github.com/svenwltr/626604257576a7b687d202aade2a27df

Configuration

Our current configuration is rather small and looks like this:

  --rocksdb.write_buffer_size 64
  --rocksdb.wal_size_limit_mb 500
  --rocksdb.block_cache_size 250
  --rocksdb.max_background_jobs 1
  --rocksdb.max_subcompactions 1

We lowered rocksdb.max_background_jobs and rocksdb.max_subcompactions to 1, but this actually made it worse, since the compaction took longer and was unresponsive nonetheless. We were hoping that while taking longer, KVrocks would stay responsive.

[PragmaTwice]

Maybe you can try to set a suitable compaction-checker-cron and compaction-cron?

Not an expert of compaction, to see if @git-hulk have any idea.

[svenwltr]

@git-hulk The disk metrics look relaxed. Reading 5.4MB/s and writing 7.4MB/s during that time while the limit is 125MB/s.

Also during the day the read throughput goes up to 11MB/s and write to 18MB/s without any issues.

[git-hulk]

@svenwltr Thanks for your input. How much CPU quota is allocated for the Kvrocks pod? It seems to use 80% at its peak but down after that, but it's strange if disk/CPU usage were good, but the instance was unable to connect. It should be throttled by CPU or Disk bandwidth. Kubernetes can monitor the CPU throttle metric but not sure if you have collected that.

[svenwltr]

@git-hulk The pod has a request of one core and a limit of 4 cores. The node where the pod is running on is underutilized and does not use more than 50% of available cores.

I took a look at the container_cpu_cfs_throttled_seconds_total metric from cadvisor and it is 0 for the whole time. So not throttling happening all the time.

I was also confused, that there is not obvious bottleneck on this. Also the burst of Broken pipe errors after the compaction is confusing, because it looks like kvrocks just stops handling any network traffic during compaction.

[git-hulk]

@svenwltr Thank you. And yes, from your provided logs, I found that connections were reset after compaction, which means all operations stalled but no writes stalled during compaction. It makes no sense if CPU usage is around 50%. I forgot to ask which Kvrocks version you're using? it might relate to #2596 if you're using 2.10.0

[svenwltr]

@git-hulk We are actually at 2.10.0. Sorry, I missed that there is another release. I will upgrade to 2.10.1 and report back!

Thank you!

[git-hulk]

@svenwltr Thanks for your information. Can ping me back if you have any issues after upgrading to 2.10.1

[svenwltr]

There were no issues tonight. Thank you very much!