Skip to content

e2e tests leave resources behind after running #4712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mahmoudhossam opened this issue Mar 28, 2025 · 5 comments
Open

e2e tests leave resources behind after running #4712

mahmoudhossam opened this issue Mar 28, 2025 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@mahmoudhossam
Copy link

mahmoudhossam commented Mar 28, 2025

What broke? What's expected?

Running e2e tests leaves cert-manager related resources behind, causing failures in subsequent runs.

Namely, the resources leases.coordination.k8s.io/cert-manager-cainjector-leader-election and leases.coordination.k8s.io/cert-manager-controller in the kube-system namespace.

Reproducing this issue

Run make test-e2e then kubectl get -n kube-system leases.coordination.k8s.io

KubeBuilder (CLI) Version

Version: cmd.version{KubeBuilderVersion:"4.5.2", KubernetesVendor:"1.32.1", GitCommit:"7c707052daa2e8bd51f47548c02710b1f1f7a77e", BuildDate:"2025-03-26T19:24:40Z", GoOs:"darwin", GoArch:"arm64"}

PROJECT version

3

Plugin versions

layout:
- go.kubebuilder.io/v4

Other versions

Kubectl:

Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.2

Go 1.23.0
sigs.k8s.io/controller-runtime v0.20.0

Extra Labels

No response

@mahmoudhossam mahmoudhossam added the kind/bug Categorizes issue or PR as related to a bug. label Mar 28, 2025
@camilamacedo86
Copy link
Member

Thanks for opening this issue!

Just to better understand the context, could you please clarify a few things?

  • What E2E tests are you running?

    • Are you referring to the Kubebuilder E2E tests used to maintain the project itself?
    • Or are you running E2E tests scaffolded in a project generated by Kubebuilder?
  • How are you executing the tests?

    • Can you please share the exact steps or commands you're using to reproduce the scenario?

This information will help us investigate and assist you more effectively. Thanks again!

@camilamacedo86 camilamacedo86 added the triage/needs-information Indicates an issue needs more information in order to work on it. label Mar 31, 2025
@mahmoudhossam
Copy link
Author

Thanks for opening this issue!

Just to better understand the context, could you please clarify a few things?

* What E2E tests are you running?
  
  * Are you referring to the Kubebuilder E2E tests used to maintain the project itself?
  * Or are you running E2E tests scaffolded in a project generated by Kubebuilder?

* How are you executing the tests?
  
  * Can you please share the exact steps or commands you're using to reproduce the scenario?

This information will help us investigate and assist you more effectively. Thanks again!

I'm running the E2E tests scaffolded in a project generated by Kubebuilder.

To be more specific: the problem is with this code.

This way of uninstalling cert-manager doesn't take into account the resources that could be generated by cert-manager as part of its runtime.

Maybe using helm for (un)installing cert-manager will fix this?

@camilamacedo86
Copy link
Member

camilamacedo86 commented Mar 31, 2025

Hi @mahmoudhossam,

Thank you for your response!

I'm running the E2E tests scaffolded in a project generated by Kubebuilder.

Given that, your proposal in #4718 doesn’t actually address your scenario.
The file you've modified won't affect the projects scaffolded by Kubebuilder at all — it won’t change anything in the generated code.

In this case, what we can do is:

  1. Review the documentation of both projects (Prometheus and CertManager).
  2. See how we can improve the uninstall methods scaffolded by default.
  3. Update the template that generates the utils file:
    utils.go
  4. Then, run make generate to ensure the samples and tutorial code are properly updated.
  5. We also will need to update our own utils ( used in the kubebuilder itself e2e tests, here: https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/utils

Let me know your thoughts — please feel free to propose an improvement in for the uninstall methods if they are required.

@mahmoudhossam
Copy link
Author

Thanks for the speedy response, @camilamacedo86!

I'm not using the tutorial code, I just pulled the first instance of the code I'm seeing in the Kubebuilder codebase.

The codebase I'm working on (unfortunately private) works with existing kubernetes types so make generate is currently unused since there are no CRDs in use.

As for the test utils code, I can test with using helm to install/uninstall cert-manager and report back.

@camilamacedo86
Copy link
Member

camilamacedo86 commented Mar 31, 2025

using helm to install/uninstall cert-manager

We should not use Helm to do that since Helm is not required for the projects.
If we want to/or we need to improve the process to uninstall, then we need to strengthen the funcs

https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugins/golang/v4/scaffolds/internal/templates/test/utils/utils.go#L98-L106

https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugins/golang/v4/scaffolds/internal/templates/test/utils/utils.go#L134-L141

And the same changes applied to the funcs used by Kubebuilder tests persei at: https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/utils

Ideally, this is defined in the documentation of those solutions.
How do the maintainers of Prometheus and CertManager suggest uninstalling their solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
2 participants