Hello! I apologize in advance if this is more of a question for the Istio community, but since this is a rather niche problem directly related to PLP, I figure I would ask this community because there is a chance someone here has some insight or experience with this.
I am trying to leverage prom-label-proxy to restrict my Grafana users to querying their namespace only, while allowing them to query Prometheus and edit/create Grafana dashboards. However, I'm having trouble with getting this to work under Istio.
For security compliance, we must use istio-proxy sidecars to facilitate STRICT TLS/mTLS for all cluster traffic. I can only get prom-label-proxy working if I circumvent security and set mTLS mode to PERMISSIVE. When using STRICT mode, the Grafana -> prom-label-proxy -> Prometheus traffic returns upstream connect error or disconnect/reset before headers. reset reason: connection termination.
All NetworkPolicies present allow this traffic to occur. Same with AuthorizationPolicies. Independently, Grafana can reach both prom-label-proxy and Prometheus with curl, prom-label-proxy can reach and query Prometheus with wget. But when, from Grafana, PLP is queried, which goes upstream to Prometheus, I'm ultimately met with the upstream connect error.
I can share config if needed, but nothing is necessarily misconfigured-- everything works correctly when istio is not there to facilitate STRICT mTLS.
If anyone has any experience with working with PLP and Istio I would greatly appreciate your insight. Thank you so much.
Hello! I apologize in advance if this is more of a question for the Istio community, but since this is a rather niche problem directly related to PLP, I figure I would ask this community because there is a chance someone here has some insight or experience with this.
I am trying to leverage
prom-label-proxyto restrict my Grafana users to querying their namespace only, while allowing them to query Prometheus and edit/create Grafana dashboards. However, I'm having trouble with getting this to work under Istio.For security compliance, we must use
istio-proxysidecars to facilitateSTRICTTLS/mTLS for all cluster traffic. I can only getprom-label-proxyworking if I circumvent security and set mTLS mode toPERMISSIVE. When usingSTRICTmode, the Grafana ->prom-label-proxy-> Prometheus traffic returnsupstream connect error or disconnect/reset before headers. reset reason: connection termination.All
NetworkPoliciespresent allow this traffic to occur. Same withAuthorizationPolicies. Independently, Grafana can reach bothprom-label-proxyand Prometheus withcurl,prom-label-proxycan reach and query Prometheus withwget. But when, from Grafana, PLP is queried, which goes upstream to Prometheus, I'm ultimately met with the upstream connect error.I can share config if needed, but nothing is necessarily misconfigured-- everything works correctly when istio is not there to facilitate
STRICTmTLS.If anyone has any experience with working with PLP and Istio I would greatly appreciate your insight. Thank you so much.