Metric kafka_log_log_size

[arnitolog]

I see that multiple brokers report log size for the same partition, which makes sense since we have leader and follower replicas. And this multiplies the number of collected metrics by the replication factor value. For example, if the topic has a replication factor of 3, we will have 3 values for every partition:

It becomes even more complicated when we have several thousand topics.

And if we want to monitor topic/partition size, we need only one value, most likely from the leader.
So the question is, is it possible to add a new label broker_type=leader|follower to this metric, so we can easily filter out all follower metrics in ingestion time, and keep only data from the leader?

[mimaison]

The metrics-reporter just passes Kafka metric names to the Prometheus client. By design we don't want to alter the names and labels of metrics. Anyway the reporter does not know which partitions each broker leads so it's not something that we can do.

This behavior is not metrics-reporter specific, you get the same metrics if you use JMX exporter to expose your metrics.

[arnitolog]

thanks @mimaison. I just thought that if this project is designed specifically for Kafka, it might make life a bit easier for Kafka administrators and provide more info in certain use cases

[mimaison]

To clarify, it's designed specifically for Kafka in the sense that it implements the MetricsReporter API that Kafka exposes to collect metrics. Implementations of this interface just get the metrics without any details about the state of the cluster. One advantage is that the plugin is notified as metrics are created/deleted which allows optimizing the metric collection logic.