Metrics not visible for all Windows containers on AKS on upgrade to latest helm chart version of prometheus-windows-exporter

[sivarama-p-raju]

Current Behavior

First and foremost, I would like to apologize that it took so long to report this. We initially thought our use of the prometheus windows exporter would be temporary, but now it looks like we will need it permanently as Sysdig Monitor that we generally use has no plan to add support for Windows anytime soon.

Helm chart: prometheus-windows-exporter
Helm Repo: https://prometheus-community.github.io/helm-charts

We have been using the Prometheus Windows Exporter to export metrics for Windows containers so that we could scrape them and make them available to a datasource on Grafana where we have some dashboards showing some of the below:

• CPU and Memory utilization in comparison to the requests and limits:

Here is an example of how this dashboard looks like:

metrics used in combination with other kube-state metrics:

windows_container_memory_usage_private_working_set_bytes
windows_container_cpu_usage_seconds_total

• Network bytes sent and recieved

Metrics used:

windows_container_network_transmit_bytes_total
windows_container_network_receive_bytes_total

• Disk read and write

metrics used:

windows_container_storage_write_size_bytes_total
windows_container_storage_read_size_bytes_total

We notice that the dashboards have worked great until the chart version 0.7.1. We have observed problems with any updated chart versions after 0.7.1 including the latest version 0.12.6.

The problems we observe is that as soon as we update the helm chart and the pods are rolled, the dashboard becomes a little empty. We see that the metrics for not all windows containers is visible. As soon as we rollback to 0.7.1, they come back and everything is normal.

Example, this is how it looks before helm chart upgrade, after upgrade and after a rollback:

As we can see, metrics for most (not all though) containers disappear after upgrade which are back on a fallback.

Expected Behavior

We should be able to upgrade to latest versions of the chart and still have the same functionality.

Steps To Reproduce

1. Environment: AKS with Windows node image version `AKSWindows-2022-containerd-20348.5020.260415`

2. Kubernetes Version: 1.34.6

3. Here is how our custom values looks like:


config: |-
  collectors:
    enabled: '[defaults],container,tcp'

global:
  imagePullSecrets:
  - name: <custom>
  imageRegistry: "<custom>/docker"

serviceAccount:
  create: false

rbac:
  create: false

resources:
  limits:
    memory: 256Mi
  requests:
    cpu: 20m
    memory: 96Mi

nodeSelector:
  kubernetes.io/os: windows

tolerations:
- key: workload
  operator: Equal
  value: win
  effect: NoSchedule
- key: node.kubernetes.io/unschedulable
  operator: Exists
  effect: NoSchedule
- key: node.kubernetes.io/network-unavailable
  operator: Exists
  effect: NoSchedule
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute


4. Helm chart details:

- Chart Name: `prometheus-windows-exporter`
- Helm repository: `https://prometheus-community.github.io/helm-charts`
- Working version: `0.7.1`
- Non-working versions: any versions after `0.7.1` and including the latest version `0.12.6`

Environment

• windows_exporter Version: 0.12.6
• Windows Server Version: AKS with Windows node image version AKSWindows-2022-containerd-20348.5020.260415

windows_exporter logs

1) Logs when running version `0.7.1`

- `Container: configure-firewall`


Name                          : {09b8bb48-0484-4ee6-a8b7-bebe43a40734}
DisplayName                   : windows-exporter
Description                   : 
DisplayGroup                  : 
Group                         : 
Enabled                       : True
Profile                       : Any
Platform                      : {}
Direction                     : Inbound
Action                        : Allow
EdgeTraversalPolicy           : Block
LooseSourceMapping            : False
LocalOnlyMapping              : False
Owner                         : 
PrimaryStatus                 : OK
Status                        : The rule was parsed successfully from the store. (65536)
EnforcementStatus             : NotApplicable
PolicyStoreSource             : PersistentStore
PolicyStoreSourceType         : Local
RemoteDynamicKeywordAddresses : {}
PolicyAppId                   : 


- `Container: windows-exporter`


ts=2026-05-07T09:24:48.509Z level=info caller=config.go:77 msg="Loading configuration file: C:\\hpc/config.yml"
ts=2026-05-07T09:24:48.582Z level=warn caller=os.go:113 msg="The os collect holds a number of deprecated metrics and will be removed mid 2025. See https://github.com/prometheus-community/windows_exporter/pull/1596 for more information."
ts=2026-05-07T09:24:48.585Z level=warn caller=cs.go:65 msg="The cs collector is deprecated and will be removed in a future release. Logical processors has been moved to cpu_info collector. Physical memory has been moved to memory collector. Hostname has been moved to os collector."
ts=2026-05-07T09:24:48.715Z level=info caller=exporter.go:309 msg="Running as NT AUTHORITY\\SYSTEM"
ts=2026-05-07T09:24:48.715Z level=info caller=exporter.go:218 msg="Enabled collectors: cs, memory, service, system, container, cpu, logical_disk, physical_disk, net, os"
ts=2026-05-07T09:24:48.716Z level=info caller=exporter.go:241 msg="Starting windows_exporter" version=0.29.2 branch=HEAD revision=622813343f930c121c8b1c061d6dd6f02d96015f goversion=go1.23.3 builddate=20241122-12:23:59 maxprocs=8
ts=2026-05-07T09:24:48.719Z level=info caller=tls_config.go:347 msg="Listening on" address=[::]:9182
ts=2026-05-07T09:24:48.719Z level=info caller=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9182


2) Logs when running version `0.12.6`

- `Container: configure-firewall`


Name                          : {5befbd34-ff70-4df7-b014-dbf61e9bd51d}
DisplayName                   : windows-exporter
Description                   : 
DisplayGroup                  : 
Group                         : 
Enabled                       : True
Profile                       : Any
Platform                      : {}
Direction                     : Inbound
Action                        : Allow
EdgeTraversalPolicy           : Block
LooseSourceMapping            : False
LocalOnlyMapping              : False
Owner                         : 
PrimaryStatus                 : OK
Status                        : The rule was parsed successfully from the store. (65536)
EnforcementStatus             : NotApplicable
PolicyStoreSource             : PersistentStore
PolicyStoreSourceType         : Local
RemoteDynamicKeywordAddresses : {}
PolicyAppId                   : 


- `Container: windows-exporter`


time=2026-05-07T10:17:04.251Z level=INFO source=main.go:153 msg="using configuration file: C:\\hpc/config.yml"
time=2026-05-07T10:17:04.678Z level=INFO source=net.go:288 msg="nic/addresses collector is in an experimental state! The configuration and metrics may change in future. Please report any issues." collector=net
time=2026-05-07T10:17:04.684Z level=INFO source=main.go:277 msg="Running as WORKGROUP\\AKSWINDEV000002$"
time=2026-05-07T10:17:04.684Z level=INFO source=main.go:190 msg="Enabled collectors: cpu, memory, logical_disk, physical_disk, net, os, service, system, container, tcp"
time=2026-05-07T10:17:04.685Z level=INFO source=main.go:208 msg="starting windows_exporter in 435.0836ms" version=0.31.6 branch=HEAD revision=20762588b37dd454e9e9a64e4efc3b469e44ce0a goversion=go1.25.0 builddate=20260401-14:42:35 maxprocs=8
time=2026-05-07T10:17:04.687Z level=INFO source=tls_config.go:354 msg="Listening on" address=[::]:9182
time=2026-05-07T10:17:04.687Z level=INFO source=tls_config.go:357 msg="TLS is disabled." http2=false address=[::]:9182

Anything else?

I am sorry if I am missing any crucial information needed for troubleshooting this. Please let me know if I can provide any more information.

[jkroepke]

The question that I have, are the metrics gone as well, if you are not combine them with kube-state-metrics, e.g. just query the raw data.

Since Microsoft refused a testing subscription, we are not longer to able to test the exporter against AKS.