Skip to content

Avoid blocking on completion signals when the waiter has left - #681

Open
kevindharmawan wants to merge 2 commits into
hyperledger-labs:mainfrom
SeeMoReLab:fix-completion-channel-blocking
Open

Avoid blocking on completion signals when the waiter has left#681
kevindharmawan wants to merge 2 commits into
hyperledger-labs:mainfrom
SeeMoReLab:fix-completion-channel-blocking

Conversation

@kevindharmawan

@kevindharmawan kevindharmawan commented Aug 9, 2026

Copy link
Copy Markdown

The controller and the view changer both used unbuffered channels to report completion, where the receiver can legitimately stop waiting and leave the sender blocked forever.

Controller: Decide stops waiting on deliverChan when the view aborts, but decide was still blocked sending on it, so the run loop stalled until shutdown. Now uses a per-decision delivered channel closed after delivery. In the aborted case decide no longer returns early, so the decision-count increment, rotation check, pruning and leader-token acquisition now run, since the decision was in fact delivered.

View changer: once commitInFlightProposal returned via timeout or stop, nothing read inFlightDecideChan or inFlightSyncChan. A late Decide or Sync then blocked the in-flight view's run goroutine, which blocked Abort() since it waits on that goroutine, which blocked the view changer in its own cleanup. No later attempt could start to drain the channel. Decide could at least escape on stopChan, while Sync had no escape at all. Now each attempt owns an inFlightAttempt holding its own buffered decideCh and syncCh, cleared when the attempt ends, with non-blocking sends. The in-flight view's Decider and Sync are callbacks bound to that attempt at creation, so a late Decide or Sync can only signal its own attempt and cannot disturb a later one.

Adds regression tests for all three completion paths.

Signed-off-by: Kevin <61072789+kevindharmawan@users.noreply.github.com>
Signed-off-by: Kevin <61072789+kevindharmawan@users.noreply.github.com>
@kevindharmawan
kevindharmawan force-pushed the fix-completion-channel-blocking branch from dd062a5 to 6bd2924 Compare August 13, 2026 04:52
if d.delivered != nil {
close(d.delivered)
}
if c.stopped() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behavior change worth confirming: previously decide blocked on the unbuffered deliverChan send when the waiter had left (view aborted), so the post-delivery bookkeeping below (incrementCurrentDecisionsInView, checkIfRotate/changeView, MaybePruneRevokedRequests, acquireLeaderToken) never ran in the aborted case. It now always runs once delivery happened. This unblocks the run loop (the real fix) and is defensible because the decision was in fact delivered, but it means a rotation changeView and leader-token acquisition can now fire for a view that is being aborted, racing the in-progress view change. changeView guards against going backwards (latestView > newViewNumber), which mitigates it, but please confirm the extra increment/rotate cannot momentarily double-count or start a spurious rotation view before the pending abort/view-change is processed.

stopView bool
}

type inFlightAttempt struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead state: id, view, and sequence on inFlightAttempt (and ViewChanger.inFlightAttemptSeq that feeds id) are written but never read anywhere — attempt identity is established purely by pointer equality (v.inFlightAttempt == attempt in the cleanup defer). Only decideCh, syncCh, and viewRef are actually used. Consider dropping the unused fields and the inFlightAttemptSeq counter to reduce clutter.


// Decide delivers to the application and informs the view changer after delivery.
// It is kept for compatibility; in-flight views use per-attempt callbacks.
func (v *ViewChanger) Decide(proposal types.Proposal, signatures []types.Signature, requests []types.RequestInfo) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These public Decide/Sync methods (and currentInFlightAttempt) are now dead in production. In-flight views are wired with the per-attempt callbacks (Decider: callbacks, Sync: callbacks), and nothing else wires ViewChanger as a Decider/Synchronizer (the heartbeat handler is c.controller, and consensus.go never passes the ViewChanger as a Decider/Sync). Only the new tests call them. Beyond being dead code, they are a latent foot-gun: when invoked with a nil attempt they still call Application.Deliver/Synchronizer.Sync without stopping any view or signaling. Consider removing them rather than keeping them as a compatibility shim.

return
default:
return
case <-v.stopChan:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead branch: because this select has a default, it never blocks, so the case <-v.stopChan: branch can never be the distinguishing choice — and all three branches just return anyway. The stopChan case is unreachable/pointless here (same in syncInFlight, which correctly omits it). Simplify to a plain non-blocking send:

	select {
	case attempt.decideCh <- struct{}{}:
	default:
	}

@HagarMeir

Copy link
Copy Markdown
Contributor

🤖 This PR was reviewed by Claude. See the inline comments above for detailed findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants