Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Prometheus Pushgateway Support #175

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

marcoschulte
Copy link

@marcoschulte marcoschulte commented Feb 14, 2025

Hi everyone,

This PR adds support for sending k6 metrics to a Prometheus Push Gateway.

Why?

I encountered this need while working with OpenShift's User Workload Monitoring, which provides a Prometheus instance that cannot be configured, meaning the remote write endpoint could not be enabled.

There is an existing extension for Pushgateway support: xk6-output-prometheus-pushgateway. However, it has some limitations:

  • Metrics (except gauges) report incorrect values since the extension does not track counters, histograms,... internally.
  • The available Grafana dashboards do not work with this extension due to different metric names.

Why integrate this into xk6-output-prometheus-remote?

To ensure compatibility with existing dashboards and correct metric tracking, I integrated Pushgateway support into the remote-write extension. This allows us to reuse the existing logic for metric state tracking and naming conventions.

Features of the Pushgateway support:

  • Uses the same metric names and label conventions → 100% compatible with existing Grafana dashboards
  • Supports all configuration parameters (except Sigv4), including trendstats, additional headers, bearer token, basic auth
  • Supports native histograms or histograms as gauges

Notes on Native Histograms:

Currently, the Pushgateway accepts native histograms but is not able to export them in OpenMetrics/Text format via /metrics. It only exports an "+Inf" bucket. This could be addressed by either:

  1. Using a "classic" histogram internally when sending data to the Pushgateway.
  2. Waiting until the Pushgateway supports native histogram export.

Usage

K6_PROMETHEUS_RW_USE_PUSHGATEWAY=true \
K6_PROMETHEUS_RW_PUSHGATEWAY_JOB=my-job \
K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9091 \
k6 run -o experimental-prometheus-rw --tag testid=my-test samples/simple.js

Looking forward to your feedback!

@marcoschulte marcoschulte requested a review from a team as a code owner February 14, 2025 14:04
@marcoschulte marcoschulte requested review from oleiade and olegbespalov and removed request for a team February 14, 2025 14:04
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codebien
Copy link
Contributor

Hi @marcoschulte,
thanks for your contribution. 🙇

Unfortunately, at the moment, we don't have the capacity to maintain this part, so I would encourage you to try to contribute directly to https://github.com/martymarron/xk6-output-prometheus-pushgateway or to open a new extension for it.

A better solution on the long-term might be for you to pass-through OpenTelemetry. Did you consider it?

@marcoschulte
Copy link
Author

Hi @codebien,

thanks for the OpenTelemetry suggestion. However, if I understand correctly, that would still rely on Prometheus' RemoteWrite capability, which unfortunately I cannot use.

I don’t think it would be reasonable to contribute to the existing Pushgateway extension or to create a new one.

This PR reuses 95% of the existing functionality, only making a last-second turn to send metrics to a Pushgateway instead of the RemoteWrite endpoint. Creating a new extension would essentially mean cloning this repository and making a small modification. Similarly, contributing to the Pushgateway extension would require rewriting it to function exactly like this extension already does. In both cases, we would end up maintaining duplicate code, whether due to potential changes in the k6 extensions API or to ensure compatibility with the conventions assumed by existing Grafana dashboards.

That being said, if you believe k6 supporting Pushgateways would be valuable, I think this extension is the right place for it. Most of the LOC introduced in this PR are tests, the actual production code is minimal. However, I understand that you need to balance added value against maintenance effort.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants