Is your feature request related to a problem? Please describe.
The content.proton.search_protocol.query.latency metric provides only mean and max measurements. This means that we can't determine the tail performance of proton queries. Tail latency is very useful for understanding and debugging performance issues, and for modelling cluster performance. At the moment it's only possible to measure tail latency at the container with query_latency, so we're missing a lot of information.
Describe the solution you'd like
Extend the content.proton.search_protocol.query.latency metric to include p95/p99, similar to the query_latency container metric.
Describe alternatives you've considered
We can derive this data from traces, but would need to run a high volume of tracing to get useful metrics, which would add overhead.
Additional context
Is your feature request related to a problem? Please describe.
The
content.proton.search_protocol.query.latencymetric provides only mean and max measurements. This means that we can't determine the tail performance of proton queries. Tail latency is very useful for understanding and debugging performance issues, and for modelling cluster performance. At the moment it's only possible to measure tail latency at the container withquery_latency, so we're missing a lot of information.Describe the solution you'd like
Extend the
content.proton.search_protocol.query.latencymetric to include p95/p99, similar to thequery_latencycontainer metric.Describe alternatives you've considered
We can derive this data from traces, but would need to run a high volume of tracing to get useful metrics, which would add overhead.
Additional context