Move replica banning to its own task

[drdrsh]

This PR moves banning logic to be owned an async task.

The async task will receives events from various sources that want to request a replica be banned.

This design allows for agents other than clients to ban replicas. This should allows us to introduce manual instance banning or banning based on periodic health checks.

We also replace the RWLock around banlists and use ArcSwap instead. This should make Banlist reads more or less lock-free

[drdrsh]

I moved the check for "All replicas are banned" to be in ban to avoid making the super hot is_banned method slower. Under normal operations, we rarely ever call ban but we call is_banned all the time.

[levkk]

Let's make this send instead of try_send. We really don't want to ever lose this message.

[levkk]

We might want to create some kind of structure uniquely identifying a "pool", if we don't have one already. Passing around strings is fine, but they are arbitrarily large and error-prone.

[levkk]

Thinking out loud:

This channel can get saturated during incidents. Imagine we have 30k clients that together produce 50k QPS. If a replica goes down, all of a sudden we'll get 50k banning events which will quickly saturate this channel. The async worker task may not even get scheduled to ban the replica.

With an RW lock, one of the clients will ban, and all the others will read the banlist and see that it's banned immediately. RW locks are probably more expensive than ArcSwap (something we should actually validate), but they are more effective than channels I think (something we should investigate).

I think the middle ground can be a Mutex for banning, and an ArcSwap for reading the list, so only one task gets to ban at a time instead of a thundering herd (pseudo-code):

fn ban() {
  if (is_banned()) return;

  let guard = ban_mutex.lock();

  let new_ban_list = ...;

  arc_swap.set(new_ban_list);
}

fn is_banned() {
   let ban_list = (*arc_swap);
   ban_list.contains_key(address);
}

The thesis that this architecture allows for other agents than clients to ban seems shaky. Another agent, e.g. an admin command, can easily use the pool.ban and pool.unban methods, for example, and it would require very little changes to the code and no changes to the arch.