Remove status metrics for deleted resources #10597

ryanrolds · 2025-02-04T16:35:10Z

Description

The status metrics for resources are left around after the resource is deleted. If the resource was invalid, the metric would continue to report invalid causing false alarms that required restarting the controller to clear.

Sleeping for 1 second was required to avoid a race condition that resulted in the new metrics getting caught up in the clearing/unregistering. I don't like adding the sleep, but that only other option is upgrading the metrics package we are using to a newer supported one (which doesn't have the race condition, the other package use a mutex).

When we upgrade to a new package we can remove this. The E2E test should have long-term value. To address concerns about the clearing causing the metrics to flicker, it's been made opt-in.

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works

solo-changelog-bot · 2025-02-05T17:10:23Z

Issues linked to changelog:
kgateway-dev#6938

ryanrolds · 2025-02-05T17:23:04Z

/kick-ci

ryanrolds · 2025-02-05T17:31:13Z

/kick-ci

github-actions · 2025-02-05T17:46:23Z

Visit the preview URL for this PR (updated for commit 9e9877a):

https://gloo-edge--pr10597-rolds-resource-metri-eq3r3xf3.web.app

_{(expires Thu, 20 Feb 2025 21:37:11 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

ryanrolds · 2025-02-05T19:35:02Z

/kick-ci

ryanrolds · 2025-02-06T16:55:26Z

/kick

nfuden

I dont quite understand the rationale for clearing all statuses rather than doing something a bit more targeted and am concerned it could cause some flicker in metrics.

If an extra example can be added to prove otherwise then I am onboard

pkg/utils/statsutils/metrics/metrics.go

projects/gateway/pkg/syncer/translator_syncer.go

Co-authored-by: Nathan Fudenberg <[email protected]>

ryanrolds · 2025-02-07T17:14:45Z

I dont quite understand the rationale for clearing all statuses rather than doing something a bit more targeted and am concerned it could cause some flicker in metrics.

The API provided by the metrics packages just don't provide the level of access needed to filter/remove specific label combinations from the metrics/views. I wish it wasn't that way. The same issue exists in the package we would like to move to, but it won't require a sleep reducing the data race we are both concerned about.

ryanrolds · 2025-02-12T22:59:11Z

/kick-ci

…lo-io/gloo into rolds/resource_metrics_deletion

nfuden · 2025-02-13T16:06:38Z

docs/content/reference/values.txt

@@ -515,6 +515,7 @@
 |gloo.headerSecretRefNsMatchesUs|bool||Set to true to require that secrets sent in headers via headerSecretRefs come from the same namespace as the destination upstream. Default: false|
 |gloo.podDisruptionBudget.minAvailable|string||Corresponds directly with the _minAvailable_ field in the [PodDisruptionBudgetSpec](https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/pod-disruption-budget-v1/#PodDisruptionBudgetSpec). This value is mutually exclusive with _maxUnavailable_.|
 |gloo.podDisruptionBudget.maxUnavailable|string||Corresponds directly with the _maxUnavailable_ field in the [PodDisruptionBudgetSpec](https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/pod-disruption-budget-v1/#PodDisruptionBudgetSpec). This value is mutually exclusive with _minAvailable_.|
+|gloo.clearStatusMetrics|bool||Set to true to clear status metrics for deleted resources. Default is false.|


nit: I think we want to also call out the danger in this too just like the changelog does.
wdyt?

nfuden · 2025-02-13T16:10:26Z

test/kubernetes/e2e/tests/manifests/edge-gateway-test-helm.yaml

+settings:
+  kubeResourceOverride:
+    spec:
+      observabilityOptions:
+        configStatusMetricLabels:
+          Gateway.v1.gateway.solo.io:
+            labelToPath:
+              name: '{.metadata.name}'
+              namespace: '{.metadata.namespace}'
+          Upstream.v1.gloo.solo.io:
+            labelToPath:
+              name: '{.metadata.name}'
+              namespace: '{.metadata.namespace}'
+          VirtualService.v1.gateway.solo.io:
+            labelToPath:
+              name: '{.metadata.name}'
+              namespace: '{.metadata.namespace}'


Why is this added?

The tests are run twice, once with the clearing and once without. This change ensures that non-clearing case is writing resource status metrics (it doesn't by default). tl;dr: The tests need it.

nfuden · 2025-02-13T16:11:13Z

pkg/utils/statsutils/metrics/metrics.go

 	"go.opencensus.io/tag"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/client-go/util/jsonpath"
 )

+const (
+	ClearStatusMetricsEnvVar = "GLOO_CLEAR_STATUS_METRICS"


Do we want this to be prefixed with GLOO given upcoming naming changes or do we think thats in fact a boon?

Im currently leaning towards this being the right naming but figure i might as well make the though persisted so others can chime in

I feel it can go in like this and when the time comes we can rename all env vars prefixed by GLOO_ to a different one (by then we should have consensus on a prefix)

Is there an alternative? It looks like the convention is to prefix env vars with GLOO_. I would assume the naming change would result in all the env vars needing their prefix changed.

.github/workflows/pr-kubernetes-tests.yaml

davidjumani · 2025-02-13T16:15:23Z

pkg/utils/statsutils/metrics/metrics.go

 	"go.opencensus.io/tag"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/client-go/util/jsonpath"
 )

+const (
+	ClearStatusMetricsEnvVar = "GLOO_CLEAR_STATUS_METRICS"


I feel it can go in like this and when the time comes we can rename all env vars prefixed by GLOO_ to a different one (by then we should have consensus on a prefix)

pkg/utils/statsutils/metrics/metrics.go

test/kubernetes/e2e/features/metrics/edge.go

davidjumani · 2025-02-13T17:09:39Z

test/kubernetes/e2e/features/metrics/edge.go

+	s.Assertions.NoError(err, "can delete nginx server upstream and VS")
+
+	// Vary based on if .Values.gloo.deployment.clearStatusMetrics is set to true
+	if s.TestInstallation.Metadata.ValuesManifestFile == e2e.ManifestPath("clear-status-metrics.yaml") {


Could you point out where is this run with different manifest files?

Done.

On my list of things to do is adjust the values validation done in the testutils to return the merged Helm values loaded into the structs found in ./install/helm/gloo/generate/values.go. That would allow conditions, like above, to vary on the specific field we are driving helm with.

I'm not comfortable with checking the manifest file since the OSS tests run in enterprise which could lead to incorrect results. Would it be better to check the env var of the deployment ? cc @nfuden

How about I make a PR with the addition to the testutils that I mentioned. After it's merged, I updated this PR to use it?

The end result will be the test checking the field at

gloo/install/helm/gloo/generate/values.go

Line 316 in b0ce83b

ClearStatusMetrics *bool `json:"clearStatusMetrics,omitempty" desc:"Set to true to clear status metrics for deleted resources. This may cause metric scraping to infrequently not see status metrics. Default is false."`

Per our huddle, I've updated the test to check the Gloo deployment for the env var and use that to determine which assertions to make.

ryanrolds · 2025-02-13T21:20:41Z

I had to make one more change to address flakey test. Because the "view" is shared hard to know if they should exist or not at the start of the test.

ryanrolds added 3 commits February 3, 2025 08:35

Partial work trying to remove deleted resource metrics

280a48f

Workign clear using a sleep

b06b297

Merge branch 'main' into rolds/resource_metrics_deletion

32fb3a3

github-actions bot added the keep pr updated label Feb 4, 2025

ryanrolds added 2 commits February 5, 2025 09:04

Working E2E test for metrics clearing

e989884

Adding changelog

458437f

ryanrolds changed the title ~~Clear resource metrics to remove metrics for deleted resources~~ Remove status metrics for deleted resources Feb 5, 2025

Cleanup

05e632c

ryanrolds marked this pull request as ready for review February 5, 2025 17:22

ryanrolds added 2 commits February 5, 2025 09:28

Fixed typo

27a846f

Minor change

a26072c

Ran generate

9de8524

nfuden requested changes Feb 7, 2025

View reviewed changes

ryanrolds and others added 2 commits February 7, 2025 09:12

Update pkg/utils/statsutils/metrics/metrics.go

dad0d38

Co-authored-by: Nathan Fudenberg <[email protected]>

Update pkg/utils/statsutils/metrics/metrics.go

79164e8

Co-authored-by: Nathan Fudenberg <[email protected]>

changelog-bot and others added 8 commits February 10, 2025 15:41

Adding changelog file to new location

7b2a039

Deleting changelog file from old location

42dc57b

Merge branch 'main' into rolds/resource_metrics_deletion

af43566

Sleep for a moment in the e2e tests

d69d642

Adding changelog file to new location

8cdf050

Deleting changelog file from old location

538a1e4

Added Helm value to control the clear status metrics behavior

7f37c1c

Merge branch 'main' into rolds/resource_metrics_deletion

fb4d7c0

ryanrolds added 3 commits February 12, 2025 15:45

Add test to a runner and fix metrics bleed when not clearing

acaa5cc

Merge branch 'rolds/resource_metrics_deletion' of ssh://github.com/so…

0ef77cf

…lo-io/gloo into rolds/resource_metrics_deletion

Updated Helm values

21d4ba4

ryanrolds requested a review from a team as a code owner February 13, 2025 00:16

ryanrolds requested a review from nfuden February 13, 2025 00:18

Fixing unit test

d3b57a6

nfuden approved these changes Feb 13, 2025

View reviewed changes

ryanrolds added 2 commits February 13, 2025 08:24

Added note to Helm value about possible issues when scraping

5927c32

Fixing a goof

78a387a

davidjumani reviewed Feb 13, 2025

View reviewed changes

Adressing feedback

b0ce83b

ryanrolds requested a review from davidjumani February 13, 2025 18:21

Fetch deployment and check for env vars

39eb4a5

davidjumani approved these changes Feb 13, 2025

View reviewed changes

ryanrolds added 2 commits February 13, 2025 13:16

Fixing flakey test

4484e51

Fixing flakey test

5173831

ryanrolds added 2 commits February 13, 2025 13:24

Minor adjustment

e9122fc

Minor adjustment

9e9877a

nfuden approved these changes Feb 13, 2025

View reviewed changes

davidjumani approved these changes Feb 13, 2025

View reviewed changes

soloio-bulldozer bot merged commit 0719073 into main Feb 13, 2025
20 checks passed

soloio-bulldozer bot deleted the rolds/resource_metrics_deletion branch February 13, 2025 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove status metrics for deleted resources #10597

Remove status metrics for deleted resources #10597

ryanrolds commented Feb 4, 2025 •

edited

Loading

solo-changelog-bot bot commented Feb 5, 2025

ryanrolds commented Feb 5, 2025

ryanrolds commented Feb 5, 2025

github-actions bot commented Feb 5, 2025 •

edited

Loading

ryanrolds commented Feb 5, 2025

ryanrolds commented Feb 6, 2025

nfuden left a comment

ryanrolds commented Feb 7, 2025

ryanrolds commented Feb 12, 2025

nfuden Feb 13, 2025

ryanrolds Feb 13, 2025

ryanrolds Feb 13, 2025

nfuden Feb 13, 2025

ryanrolds Feb 13, 2025

nfuden Feb 13, 2025

davidjumani Feb 13, 2025

ryanrolds Feb 13, 2025

davidjumani Feb 13, 2025

davidjumani Feb 13, 2025

ryanrolds Feb 13, 2025

davidjumani Feb 13, 2025

ryanrolds Feb 13, 2025

ryanrolds Feb 13, 2025 •

edited

Loading

ryanrolds Feb 13, 2025

ryanrolds commented Feb 13, 2025

Remove status metrics for deleted resources #10597

Remove status metrics for deleted resources #10597

Conversation

ryanrolds commented Feb 4, 2025 • edited Loading

Description

Checklist:

solo-changelog-bot bot commented Feb 5, 2025

ryanrolds commented Feb 5, 2025

ryanrolds commented Feb 5, 2025

github-actions bot commented Feb 5, 2025 • edited Loading

ryanrolds commented Feb 5, 2025

ryanrolds commented Feb 6, 2025

nfuden left a comment

Choose a reason for hiding this comment

ryanrolds commented Feb 7, 2025

ryanrolds commented Feb 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanrolds Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanrolds commented Feb 13, 2025

ryanrolds commented Feb 4, 2025 •

edited

Loading

github-actions bot commented Feb 5, 2025 •

edited

Loading

ryanrolds Feb 13, 2025 •

edited

Loading