fix: reduce lock contention in distributor channels (apache#10026)

* fix: lock contention in distributor channels

Reduce lock contention in distributor channels via:

- use atomic counters instead of "counter behind mutex" where
  appropriate
- use less state
- only lock when needed
- move "wake" operation out of lock scopes (they are eventual operations
  anyways and many wake operations results in "futex wake" operations --
  i.e. a syscall -- which you should avoid while holding the lock)

* refactor: add more docs and tests for distributor channels

---------

Co-authored-by: Andrew Lamb <[email protected]>