Avoid blocking on completion signals when the waiter has left - #681
Avoid blocking on completion signals when the waiter has left#681kevindharmawan wants to merge 2 commits into
Conversation
Signed-off-by: Kevin <61072789+kevindharmawan@users.noreply.github.com>
Signed-off-by: Kevin <61072789+kevindharmawan@users.noreply.github.com>
dd062a5 to
6bd2924
Compare
| if d.delivered != nil { | ||
| close(d.delivered) | ||
| } | ||
| if c.stopped() { |
There was a problem hiding this comment.
Behavior change worth confirming: previously decide blocked on the unbuffered deliverChan send when the waiter had left (view aborted), so the post-delivery bookkeeping below (incrementCurrentDecisionsInView, checkIfRotate/changeView, MaybePruneRevokedRequests, acquireLeaderToken) never ran in the aborted case. It now always runs once delivery happened. This unblocks the run loop (the real fix) and is defensible because the decision was in fact delivered, but it means a rotation changeView and leader-token acquisition can now fire for a view that is being aborted, racing the in-progress view change. changeView guards against going backwards (latestView > newViewNumber), which mitigates it, but please confirm the extra increment/rotate cannot momentarily double-count or start a spurious rotation view before the pending abort/view-change is processed.
| stopView bool | ||
| } | ||
|
|
||
| type inFlightAttempt struct { |
There was a problem hiding this comment.
Dead state: id, view, and sequence on inFlightAttempt (and ViewChanger.inFlightAttemptSeq that feeds id) are written but never read anywhere — attempt identity is established purely by pointer equality (v.inFlightAttempt == attempt in the cleanup defer). Only decideCh, syncCh, and viewRef are actually used. Consider dropping the unused fields and the inFlightAttemptSeq counter to reduce clutter.
|
|
||
| // Decide delivers to the application and informs the view changer after delivery. | ||
| // It is kept for compatibility; in-flight views use per-attempt callbacks. | ||
| func (v *ViewChanger) Decide(proposal types.Proposal, signatures []types.Signature, requests []types.RequestInfo) { |
There was a problem hiding this comment.
These public Decide/Sync methods (and currentInFlightAttempt) are now dead in production. In-flight views are wired with the per-attempt callbacks (Decider: callbacks, Sync: callbacks), and nothing else wires ViewChanger as a Decider/Synchronizer (the heartbeat handler is c.controller, and consensus.go never passes the ViewChanger as a Decider/Sync). Only the new tests call them. Beyond being dead code, they are a latent foot-gun: when invoked with a nil attempt they still call Application.Deliver/Synchronizer.Sync without stopping any view or signaling. Consider removing them rather than keeping them as a compatibility shim.
| return | ||
| default: | ||
| return | ||
| case <-v.stopChan: |
There was a problem hiding this comment.
Dead branch: because this select has a default, it never blocks, so the case <-v.stopChan: branch can never be the distinguishing choice — and all three branches just return anyway. The stopChan case is unreachable/pointless here (same in syncInFlight, which correctly omits it). Simplify to a plain non-blocking send:
select {
case attempt.decideCh <- struct{}{}:
default:
}|
🤖 This PR was reviewed by Claude. See the inline comments above for detailed findings. |
The controller and the view changer both used unbuffered channels to report completion, where the receiver can legitimately stop waiting and leave the sender blocked forever.
Controller:
Decidestops waiting ondeliverChanwhen the view aborts, butdecidewas still blocked sending on it, so the run loop stalled until shutdown. Now uses a per-decisiondeliveredchannel closed after delivery. In the aborted casedecideno longer returns early, so the decision-count increment, rotation check, pruning and leader-token acquisition now run, since the decision was in fact delivered.View changer: once
commitInFlightProposalreturned via timeout or stop, nothing readinFlightDecideChanorinFlightSyncChan. A lateDecideorSyncthen blocked the in-flight view's run goroutine, which blockedAbort()since it waits on that goroutine, which blocked the view changer in its own cleanup. No later attempt could start to drain the channel.Decidecould at least escape onstopChan, whileSynchad no escape at all. Now each attempt owns aninFlightAttemptholding its own buffereddecideChandsyncCh, cleared when the attempt ends, with non-blocking sends. The in-flight view'sDeciderandSyncare callbacks bound to that attempt at creation, so a lateDecideorSynccan only signal its own attempt and cannot disturb a later one.Adds regression tests for all three completion paths.