ServiceMonitor TLS serverName hardcoded to wva-system, silently breaks metric scraping in any other namespace

[biranofer]

Summary

The wva-controller-manager-metrics-monitor ServiceMonitor's TLS serverName is hardcoded to the default namespace (wva-system) via a kustomize patch, rather than being derived from the actual deploy namespace. When WVA is deployed to any other namespace, TLS verification during Prometheus/Thanos scraping fails silently, and all wva_* metrics (including wva_desired_replicas) never make it into Prometheus/Thanos for that deployment.

Scope

OpenShift only. The base ServiceMonitor (config/base/monitoring/servicemonitor.yaml, used by the plain Kubernetes overlay) sets tlsConfig.insecureSkipVerify: true with no serverName at all — self-signed certs, no hostname check, nothing to mismatch. The openshift kustomize component (config/components/openshift/kustomization.yaml) patches that same ServiceMonitor to instead use OpenShift's service-CA-issued certs with strict verification (insecureSkipVerify: false + CA configmap + the hardcoded serverName). Only deployments using the OpenShift overlay are affected; plain Kubernetes/kind/minikube deployments never load this component and never hit the mismatch.

Where

config/components/openshift/kustomization.yaml:

    - path: /spec/endpoints/0/tlsConfig/serverName
      value: wva-controller-manager-metrics-service.wva-system.svc

This value is only correct because the default overlay's namespace is wva-system (config/overlays/*/openshift/kustomization.yaml). Any deployment where the namespace is overridden (benchmark runs, multi-tenant clusters, per-user namespaces) keeps this stale literal.

Impact

• Prometheus/Thanos TLS-verifies the scrape target's certificate SAN against serverName. The actual cert (issued by OpenShift's service-CA for the Service) has a SAN scoped to the real namespace, e.g. wva-controller-manager-metrics-service.biran.svc. Mismatch → TLS handshake fails → scrape target down.
• Failure is silent: the ServiceMonitor object itself looks healthy; there's no user-facing error. Downstream consumers (KEDA Prometheus triggers, prometheus-adapter-backed HPAs reading wva_desired_replicas, dashboards) simply see "no data," indistinguishable from a metric that was never emitted.
• Concretely: in a benchmark run in namespace biran, WVA's controller internally computed a target of 4 replicas, but the KEDA-driven HPA never scaled past 1, because wva_desired_replicas never reached Prometheus.

Reproduction

1. Deploy WVA with the OpenShift overlay into a namespace other than wva-system.
2. Query Thanos/Prometheus for wva_desired_replicas{namespace="<your-namespace>"} — returns empty.
3. Inspect the ServiceMonitor: oc get servicemonitor <name> -n <namespace> -o yaml — tlsConfig.serverName says ...wva-system.svc, not ...<your-namespace>.svc.

Workaround

Live-patch the ServiceMonitor after standup:

oc patch servicemonitor wva-controller-manager-metrics-monitor -n <namespace> --type=json \
  -p='[{"op":"replace","path":"/spec/endpoints/0/tlsConfig/serverName","value":"wva-controller-manager-metrics-service.<namespace>.svc"}]'

Doesn't survive a fresh re-render/standup, and isn't discoverable without already knowing about this bug.

Suggested fix

Derive serverName from the actual target namespace at render time (Kustomize namespace substitution/vars) instead of patching in a literal, so it's correct regardless of which namespace WVA is deployed into.