Skip to content
This repository was archived by the owner on Apr 2, 2024. It is now read-only.

Conversation

@andremedeiros
Copy link

Problem

Heka's KafkaInput plugin writes the offset for every Kafka message. When it's a low volume topic, this doesn't cause any issues, but with high volume ones, disk IO becomes a limiting factor to how quickly Heka can process messages.

Solution

Instead of writing the offset on every message to disk, this PR mmaps it. A quick benchmark shows the advantages of using this method:

BenchmarkFile-8                  10000         119687 ns/op
BenchmarkMmap-8             2000000000           0.64 ns/op

No new tests were written, and the ones that verify checkpoint correctness still pass.

r. @fbogsany @rafrombrc

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant