Skip to content

fix: make sure machine pools are deleted before gke cluster #1458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

salasberryfin
Copy link
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

During cluster deletion, a GCPManagedCluster may be deleted before its associated GCPManagedMachinePool/s which actually needs the cluster resource to exist to delete successfully:

E0321 10:41:55.730846       1 controller.go:324] "Reconciler error" err="GCPManagedCluster.infrastructure.cluster.x-k8s.io \"cluster-gke\" not found" controller="gcpmanagedmachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="GCPManagedMachinePool" GCPManagedMachinePool="default/cluster-gke-mp-0" namespace="default" name="cluster-gke-mp-0" reconcileID="5e38a1b8-a006-4c22-973b-a7ebe1ca8c06"
E0321 10:42:05.971755       1 gcpmanagedmachinepool_controller.go:270] "Failed to retrieve GCPManagedCluster from the API Server" err="GCPManagedCluster.infrastructure.cluster.x-k8s.io \"cluster-gke\" not found" controller="gcpmanagedmachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="GCPManagedMachinePool" GCPManagedMachinePool="default/cluster-gke-mp-0" namespace="default" name="cluster-gke-mp-0" reconcileID="becd4efd-9abf-4dda-a3fb-ec1e1cfd6d95"

This happens intermittently and it's hard to reproduce, but there needs to be logic that waits for the machine pools to be removed before continuing with deleting the cluster.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

This is similar to this CAPA issue.

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

Release note:

Wait for machine pools to be deleted before deleting gke cluster

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 25, 2025
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 25, 2025
Copy link

netlify bot commented Mar 25, 2025

Deploy Preview for kubernetes-sigs-cluster-api-gcp ready!

Name Link
🔨 Latest commit 59a24be
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-cluster-api-gcp/deploys/68078003a1438400078cb6fa
😎 Deploy Preview https://deploy-preview-1458--kubernetes-sigs-cluster-api-gcp.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@salasberryfin salasberryfin force-pushed the gke-remove-machinepools-before-cluster branch from 2aefcfa to 7abfdae Compare March 25, 2025 08:01
@salasberryfin salasberryfin moved this to PR to be reviewed in CAPI / Turtles Mar 25, 2025
Copy link
Contributor

@alexander-demicev alexander-demicev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 25, 2025
@alexander-demicev alexander-demicev moved this from PR to be reviewed to Blocked in CAPI / Turtles Mar 25, 2025
Copy link
Member

@furkatgofurov7 furkatgofurov7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@salasberryfin
Copy link
Contributor Author

salasberryfin commented Apr 3, 2025

cc: @cpanato @damdo @richardcase

@salasberryfin salasberryfin moved this from Blocked to Team Backlog in CAPI / Turtles Apr 22, 2025
@salasberryfin salasberryfin force-pushed the gke-remove-machinepools-before-cluster branch from 7abfdae to 59a24be Compare April 22, 2025 11:39
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Apr 22, 2025
@salasberryfin salasberryfin moved this from Team Backlog to In Progress (8 max) in CAPI / Turtles Apr 22, 2025
@salasberryfin salasberryfin moved this from In Progress (8 max) to Blocked in CAPI / Turtles Apr 22, 2025
@richardcase
Copy link
Member

This is great, thanks @salasberryfin

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexander-demicev, richardcase, salasberryfin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [richardcase,salasberryfin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@richardcase
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2025
@k8s-ci-robot k8s-ci-robot merged commit 79cc2c3 into kubernetes-sigs:main Apr 23, 2025
17 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.1.0 milestone Apr 23, 2025
@github-project-automation github-project-automation bot moved this from Blocked to Done in CAPI / Turtles Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants