Settings not applied when deployed via Helm

[rgcosma]

Cluster Setup

• Kubernetes version: 1.27
• Type of Cluster (RKE1, RKE2, k3s, EKS, GKE, AKS, OKE, etc): RKE2
• Installation option (Helm Chart / Custom Installation):
  • If Helm Chart, did you supply any non-default values? If so, what values did you supply: prometheus.prometheusSpec.retention

Describe the bug
When deploying using Helm from https://github.com/rancher/charts/tree/dev-v2.9/charts/prometheus-federator/104.0.0-rc1%2Bup0.4.1 and setting any value - like the one mentioned - two problems occur:

1. while the setting is applied, the change is not visible in the Rancher web interface, which always shows the default from the bundled values in the federator binary; so if retention is 10d it will always show that, but checking the statefulset parameters I can see the container args say --storage.tsdb.retention.time=100d which is correct
2. Any change done via the web interface (Monitoring->Project Monitors->Edit Config) is ignored, even if it shows as saved, and I can also see the new setting in the ProjectHelmChart CRD in the namespace

To Reproduce

1. Deploy via Helm, changing any value from default
2. Check web interface - new value is not shown
3. Check deployment/statefulset - value is actually applied
4. Change value in web interface - no error, value is stored, but nothing actually changes

Result
I cannot configure parameters for individual project monitoring, only once global for the entire cluster

Expected Result
I can change parameters for each project monitor

[rgcosma]

nm, that is not the correct way of editing project monitoring parameters, but via an object like

apiVersion: helm.cattle.io/v1alpha1
kind: ProjectHelmChart
spec:
  values:
    alertmanager:
      ...
    federate:
      ...
    grafana:
      ...
    prometheus:
      ...

the documentation is confusing as heck I would say