perf(loki.write): Pre-allocate buffers used for batch encoding

[kalleep]

Brief description of Pull Request

By pre-allocating buffers for requests we can reduce total amount of memory allocated by loki.write

Pull Request Details

Before this change we allocated two buffers every time we encoded a batch:

• One temporary buffer to how the proto serialized data of push.PushRequest
• One buffer that snappy wrote to and we actually sent with the request.

Because a shard will only ever send one batch at a time we can allocate two buffers that we re-suse for each request. My first attempt to fix this was to fix the size check in a batch. We only check size of line and size of structured metadata when appending to a batch.

Ideally we should count the serialized bytes it would take to add an entry to a batch and labels. I made two attempts to do this. They where both correct but it consumed a lot of CPU to do this. If this had worked out we could always pre-allocate these two buffers and always re-used them and batch_size would actually be respected. Currently we always go over the configured value a lot.

Because it's really expensive to keep a correct count I reverted back to the old behavior and changed the API a bit. We now call request on batch to get the request we have built and number of entries it contains. We then adjust the buffers after the largest batch we have seen and reuse them.

Issue(s) fixed by this Pull Request

Notes to the Reviewer

I did some other changes to batch:

• Move the check to see if we can add to a batch into add itself and handle errors at call sites instead.

PR Checklist

• Documentation added
• Tests updated
• Config converters updated