Skip to content

fix(Grafana): Metric _pod_requests already include _daemon_requests#8965

Open
mkilchhofer wants to merge 1 commit intoaws:mainfrom
swisspost:bugfix/grafana_dashboards--daemon_requests
Open

fix(Grafana): Metric _pod_requests already include _daemon_requests#8965
mkilchhofer wants to merge 1 commit intoaws:mainfrom
swisspost:bugfix/grafana_dashboards--daemon_requests

Conversation

@mkilchhofer
Copy link

@mkilchhofer mkilchhofer commented Feb 17, 2026

Fixes #7407

Description

The dashboards provided on the website (https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/#monitoring-with-grafana-optional) calculates the used capacity on the Karpenter-managed nodes.
For this, they currently use this forumla:

karpenter_nodes_total_daemon_requests + karpenter_nodes_total_pod_requests
--------------------------------------------------------------------------
        karpenter_nodes_allocatable

This formula is wrong as the metric karpenter_nodes_total_pod_requests already include the requests of the DaemonSet pods according to the metrics documentation:

### `karpenter_nodes_total_pod_requests`
Node total pod requests are the resources requested by pods bound to nodes, including the DaemonSet pods.
- Stability Level: BETA

Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/reference/metrics.md

How was this change tested?

Importing the old and the new dashboard into our Grafana Cloud instance and compared the results:

Old:
image

New:
image

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mkilchhofer mkilchhofer changed the title fix(Grafana): Metric .._pod_requests already include daemon_requests fix(Grafana): Metric _pod_requests already include _daemon_requests Feb 17, 2026
@mkilchhofer
Copy link
Author

mkilchhofer commented Feb 17, 2026

I am not sure how far this change goes back, I can only confirm that both 1.9 and 1.8 were affected since we at @swisspost used both versions and both had this behavior.

So I adapted the dashboards for

  • /docs/.. (I assume the current version)
  • /preview/.. (to also include it in further releases)
  • /v1.8/..
  • /v1.9/..

There are other dashboard Jsons with the old formula where I am not sure if metrics once were handled different

  • /v1.0/..
  • /v1.7/..

@mkilchhofer mkilchhofer marked this pull request as ready for review February 17, 2026 11:33
@mkilchhofer mkilchhofer requested a review from a team as a code owner February 17, 2026 11:33
Signed-off-by: Marco Maurer <marco.maurer.1@post.ch>
@mkilchhofer mkilchhofer force-pushed the bugfix/grafana_dashboards--daemon_requests branch from eafdcfb to 3451e22 Compare February 17, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Grafana] CPU Utilization can be misleading, CPU request utilization could be better

1 participant