Tune Compaction and Backup

[ltagliamonte-dd]

Hello folks! I'm in a pinch with kvrocks compaction/backup tuning....

I'm working on a data pipeline where I process a bunch of parquet files into kvrocks, every N files I create a checkpoint on S3 by calling compaction (COMPACT) on kvrocks and then backup (BGSAVE), and uploading the backup folder on S3.

Very often during the compaction and backup the memory consumption goes really high and my k8s pod with 40GB of mem get's oomkilled.
Is there a way to keep at bay the mem usage during compaction and backup?
What are the knobs to tune? I don't see any knob for backups.

This is my first time using kvrocks and I'm not really familiar with all the config params.

[git-hulk]

@ltagliamonte-dd Could you provide rocksdb.* configurations in kvrocks.conf?

[ltagliamonte]

Thanks for the answers! Since I'm calling compaction at every run of my software I disabled them passing the flag to the cmd line.
My point about smarter compaction of the SST is that they all get reprocessed all the time isn't it?

[git-hulk]

My point about smarter compaction of the SST is that they all get reprocessed all the time isn't it?

Oh, yes. That's right.

[git-hulk]

@ltagliamonte-dd Wanna know if everything is going well after reducing the number of compaction threads.

[ltagliamonte-dd]

Hello @git-hulk well yes and no, mem usage is contained however compaction take up to 60m to complete.
I'd like to find other params i can tune to reduce the memory footprint, like reduce the number in memory cache (i'm just writing in this kvrocks instance) and accelerate the compactions.

[git-hulk]

@ltagliamonte-dd The default value of memory cache is 4GiB and it only be used while having the read requests. So guess it would not help if the memory is consumed by the compaction. Another possible approach is to try to reduce target_file_size_base and increase a few compaction threads