[enhancement] Filter out decommissioned worker nodes in Kubernetes / Views / Nodes dashboard

[Donnerdiebel]

Describe the enhancement you'd like

First of all thanks for your work on these dashboards! This is very much appreciated!

We have multiple k8s clusters running on AWS EKS with Karpenter. As a result, worker nodes get regularly replaced.
In the Kubernetes / Views / Nodes dashboard, the node variable gets filled by a Label values query for the kube_node_info metric, which will return all labels including already replaced worker nodes, regardless of the selected time range.
As a consequence, the node drop down box also contains nodes which have already been decommissioned.

Changing the variable query type to "query result" with a prometheus query like 'kube_node_info{cluster="$cluster"}' and a corresponding regex filter '/node="(?[^"]+)/' would take care of only listing nodes which actually have metrics in the selected time range.

If this change would have any chance of merging, I would provide a merge request.

Once again thanks for your work on this and all the best,
Björn

Additional context

No response

[dotdc]

Hi @Donnerdiebel,

Thank you for the kind words! I understand your use case, but I have some doubts about the proposed solution, have you tried it? There’s a small "hack" between node and instance labels to allow querying nodes by name instead of IP.

Feel free to open a PR, I would gladly accept a patch that works in both cases without impacting performance.

[Donnerdiebel]

Hi @dotdc
Thanks for your swift reply ;-)
I tested the above changes within our environment without any issues. As we are using victoriametrics as TSDB I will need to setup a prometheus test environment to check for any compatibility issues.
As soon as this is done and if this also shows no further issues I would open a PR and we could discuss the changes further.

Thanks again and take care,
Björn