Memory leak

[damjankuznar]

Describe the bug
We are running blutgang in a k8s custer pod with 1GB memory and ephemeral storage limit and we are observing that blutgang is periodically being killed and restarted due to exceeding memory and/or ephemeral storage. We have sled cache capacity configured to 60MB.

Image of k8s pod memory metric over the last day - you can see how memory increases over time to 1GB and then pod gets restarted and memory is reset.

Example of kubectl describe pod for blutgang

Status:           Failed
Reason:           Evicted
Message:          Pod ephemeral local storage usage exceeds the total limit of containers 1Gi.

Listing the files in cache folder we can see that it's exceeding 60MB limit that was set in the config (full config posted below):

root@json-rpc-proxy-7b4bd8db79-v7wfx:/app/blutgang-cache# ls -lah
total 238M
drwxr-xr-x 3 root root 4.0K Aug 16 18:02 .
drwxr-xr-x 1 root root 4.0K Aug 16 18:02 ..
drwxr-xr-x 2 root root  36K Aug 16 22:17 blobs
-rw-r--r-- 1 root root   62 Aug 16 18:02 conf
-rw-r--r-- 1 root root 237M Aug 16 22:17 db

To Reproduce
Steps to reproduce the behavior:

1. Run blutgang
2. Observe memory and sled storage usage

Expected behavior

Specs:

• Blutgang version: 0.3.6

Blutgang options:

• RPCs used: local Erigon nodes

[blutgang]
do_clear = false
ma_length = 100
sort_on_startup = false
health_check = true
header_check = true
supress_rpc_check = true
address = "0.0.0.0:3000"
ttl = 500
max_retries = 32
expected_block_time = 13000
health_check_ttl = 200

[admin]
enabled = false
address = "127.0.0.1:5715"
readonly = true
jwt = false
key = ""

[sled]
db_path = "./blutgang-cache"
mode = "HighThroughput"
print_profile = false
cache_capacity = 60000000
compression = false
flush_every_ms = 240

[makemake-kbo]

Known issue with sled 0.x

Blutgang will replace the current version with a more up to date one that addresses this.