Live upgrade

[libertyeagle]

1. suspend user requests & stop polling wcs ...
2. flush queues / or save queue states
3. save engine local/shared states (e.g., memfd, conn id, etc.)
4. engine rebuild
5. memfd, conn resources need to be rebuilt

[libertyeagle]

https://libertyeagle.notion.site/Live-Upgrade-1137020489df416695e52944e95bf2db

[libertyeagle]

In our initial version, we will not support live upgrade to shared resources between a group of engines, e.g., add new Commands between mrpc and RpcAdapter engines.

In 2.0 version, this can be implemented in some way similar to the following:

let local_states: HashMap<EngineId, Vec<Box<dyn Any + Send + Sync>>> = HashMap::new();
let shared_states: HashMap<String, Box<dyn Any + Send + Sync>> = HashMap::new();

// engine_groups is a group of inter-connected engines correspond to the same user thread
// e.g., mrpc engine and RpcAdapter
for engine in engine_group.topology_order() {
  engine.dump(&mut local_state[engine_id], &mut shared_states);
}

for engine in engine_group.topology_order() {
  // for shared states like Sender<mrpc::Command>, Receiver<mrpc;:Command>
  // if mrpc::Command type has not changed, then mrpc and RpcAdapter engines directly re-attaches the Sender and Receiver from `shared_states`
  // if the types has changed and the queue needs to upgrade, first we ensure the old queue is flushed before dumping engine states
  // then mrpc engine creates Sender and Receiver, it attaches Sender to itself
  // and update the Receiver in `shared_states`
  // so when RpcAdapter is recreated, it can extract the Receiver from `shared_states`
  reloadable_module.upgrade_from_old_engine(&mut local_state[engine_id], &mut shared_states);
}

Live upgrade to the data path queues (i.e, Vertex, TxIQueue, ... in current implementation) can also be implemented in similar manners.

[libertyeagle]

After the deadline, we can even try to support dynamic adding new modules with dynamiclly changing dependencies between engines.

[libertyeagle]

Currently, handle_new_client's API is different for each engine. RpcAdapter has create_engine instead of handle_new_client. We may get rid of handle_new_client and use create_engine instead. RpcAdapter's create_engine currently looks like this:

    pub fn create_engine(
        runtime_manager: &RuntimeManager,
        n: Node,
        mode: SchedulingMode,
        client_pid: Pid,
        cmd_rx: tokio::sync::mpsc::UnboundedReceiver<ipc::mrpc::cmd::Command>,
        cmd_tx: tokio::sync::mpsc::UnboundedSender<ipc::mrpc::cmd::Completion>,
    ) -> Result<RpcAdapterEngine>

To support flexiable live upgrade, we may change the API to the following:

    pub fn handle_new_client(
        &mut self,
        n: Node,
        mode: SchedulingMode,
        client_pid: Pid,
        shared_resources: HashMap<String, Box<dyn Any + Send + Sync>>
    ) -> Result<(Box<dyn Engine>, HashMap<EngineType, HashMap<String, Box<dyn Any + Send + Sync>>>

Here shared_resources is some resources created by other engines among an engine group. (e.g., cmd_rx, cmd_tx can all be created by mrpc engine, and it put them to shared_resources for RpcAdapter to take). Here in our mrpc case, the engine group is mRPC + RpcAdapter + (potentially) Salloc. Note that we can let mrpc engine to create the command queues, instead of let koala do it. It is fine as all backend modules are trusted. RpcAdapter crate should also depend on mrpc crate.

The return is an engine (through a trait object), and shared resources created by this engine for downstream engines to use (we create engines in topological order). EngineType might be a String, in this way we may support dynamically adding new engines with dynamically specified engine to engine dependencies to our system.

In the future, we may also implement Node here in some way similar to shared_resources. The engines in an engine group are responsible for creating the inter-engine queues. Then koala only serves as a controller and a runtime. Differnet types of engines can serve as hot-plug modules.

[crazyboycjr]

My intention was to replace the following two lines with a mailbox-like data structure. This way, the communication among various engines and the communication between koala Control to other engines are no longer specialized (e.g., in the case here we dedicate two queues for MrpcEngine and RpcAdapter). This way, every engine can deliver control messages/signals to other engines as long as the destination engine can process that control signal.

- cmd_rx: tokio::sync::mpsc::UnboundedReceiver<ipc::mrpc::cmd::Command>,
- cmd_tx: tokio::sync::mpsc::UnboundedSender<ipc::mrpc::cmd::Completion>,
+ bus: Bus,

[crazyboycjr]

Can we just pass a mutable reference of HashMap as shared_resources?

[libertyeagle]

It can be. I just don't want an engine be able to access other engine's shared resource. Koala control will be in charge of delivering the resources to corrsponding engine. I think either way is fine, as creating engines is not on the fast path.

[crazyboycjr]

It took people 30s to stare at the interface and understand its return value. The following may look better.

    pub fn handle_new_client(
        &mut self,
        n: Node,
        mode: SchedulingMode,
        client_pid: Pid,
        shared_resources: &mut SharedResource,
    ) -> Result<Box<dyn Engine>, Error>,

[libertyeagle]

agree. We stick with this interface. we can leave a higher abstraction of bus to future implementation.

[libertyeagle]

To have a higher level abstraction of message bus, we may not avoid dynamic dispatch.
Some creates for reference:
https://docs.rs/messagebus/0.9.13/messagebus/trait.Message.html
https://docs.rs/bastion/latest/bastion/message/struct.Msg.html#