Check deleted metrics form the test ns

[mykysha]

What type of PR is this?

/kind failing-test
/kind flake

What this PR does / why we need it:

Strictly check the deleted metrics from the test environment. This way the test will target only the metrics from the current test run, and ignore the metrics that might not have been cleared previously.

Which issue(s) this PR fixes:

Part of #4139

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	c68fdf9
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/67a4926ee373cb0008a6b23e

[mszadkow]

looks good

[mykysha]

Have removed improvements to the log verbosity from here, will create another PR just for that

[mimowo]

IIUC this PR also aims to improve the error messages which is great!

Please post example failure messages when the tests fail by inducing a failure due to invalid asserts.

Please also describe what is the actual root cause for the flakes.

EDIT: oh I see you moved the error message improvement out of the PR - this is even better to understand what is the purpose of every change. Thanks!

[mbobrovskyi]

/retest

Created an issue #4160.

[mbobrovskyi]

/lgtm
Thanks!

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 4b0718a932500915e69ee2bfe093df0dc0b3d021

[mimowo]

Strictly check the deleted metrics from the test environment. This way the test will target only the metrics from the current test run, and ignore the metrics that might not have been cleared previously.

Indeed, after de-coupling the change it becomes quite easy to see the culprit.

/approve

Still, please follow up with the logging improvements.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mimowo, mykysha

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment