Implement trapdoor - atomic notifier

[peterargue]

Trapdoor was originally implemented by @AlexHentschel (PR).

Most of our workqueues currently use the engine.Notifier to synchronize producers and consumers. This works well in most cases, but has a subtle limitation that can cause inefficient use of workers in some cases. See comments in the notifier.go below for details. tl;dr when multiple workers are waiting and then multiple producers signal new work is available, some of the signals may be dropped, leaving a subset of the workers to handle all the work while others remain idle.

Trapdoor solves this by using a sync.Cond to ensure that one signal is emitted for each call to Activate(). This means that if there are waiting workers, one worker will be woken up for each call to Activate()

The current implementation uses a different API then engine.Notify. I think it might be useful for them both to implement the same interface, allowing for developers to choose the implementation that's needed for their use case. Notify may drop signals, but has no locks and is thus light weight. Trapdoor ensures no signals are dropped, but requires a mutex.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!