Skip to content

Three additional pod-wait loops ignore ctx cancellation (follow-on to #1055) #1060

Description

@CryptoJones

Summary

PR #1057 fixed the bootstrap-token pod-wait loop (#1055) but the same fixed-iteration / time.Sleep / ctx-blind pattern is present at three other call sites in the controller:

  • internal/controller/humiocluster_pods.go:702waitForNewPods waits up to 10s for new pods to appear, sleeping 1s/iteration with no ctx check
  • internal/controller/humiocluster_controller.go:2488checkEvictionStatusForPod loops up to 10 times calling getClusterNodesStatus with no ctx check between iterations
  • internal/controller/humiocluster_controller.go:2577updateEvictionStatus sleeps 1s/iteration up to 10s without honoring ctx

Why this matters

Same blast radius as #1055: an operator drain, leader-election handoff, or pod eviction during these waits is held hostage by the loop. SIGTERM can wait up to 10s per call. With multiple waits stacked across a reconcile pass, the operator can blow past the kubelet's terminationGracePeriodSeconds and die ungracefully.

Proposed fix

Same pattern as PR #1057: replace each with wait.PollUntilContextTimeout(ctx, time.Second, ...). PR forthcoming.

Out of scope (filing separately if confirmed)

isEvictedNodeAlive at humiocluster_controller.go:2458 has the same for i := 0; i < waitForPodTimeoutSeconds; i++ outer loop with no sleep, iterating over a nodesStatus parameter that's never refreshed. That looks like either dead code or a broken-refresh logic bug rather than a ctx-cancellation issue. Happy to file separately if a maintainer confirms.

Proudly Made in Nebraska. Go Big Red! 🌽 https://xkcd.com/2347/

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions