Add support for removing nodes from the metadata store cluster #2531
Conversation
This allows users to manually add nodes to the metadata cluster.
Removing nodes from the metadata store cluster is implemented by introducing a transitional MetadataServerState::Leaving which tells the leader of the metadata store cluster to remove the node which is Leaving. Only after the configuration change has been committed and applied, the state will be set to Outsider. If a node missed the configuration change but sees its state in the NodesConfiguration switching to Outsider, it will stop Raft.
By setting the default of MetadataServerState to Member, newly added Restate nodes will try to join the metadata store cluster automatically.
| // we use the NodesConfiguration as the authoritative signal for leaving the cluster | ||
| let should_leave = my_node_config.metadata_server_config.metadata_server_state | ||
| == MetadataServerState::Outsider; |
There was a problem hiding this comment.
I realized that this won't reliably work in the case where the user removed the node and then shortly after added it back. If the node missed the actual configuration change and then reaches this point, it looks at the MetadataServerStatus and thinks that everything is ok. The problem is that w/o sending a join request, the current leader won't add this node back to the configuration.
There was a problem hiding this comment.
One idea for solving this problem could be to also introduce a transitional state for joining a cluster (similar to Leaving). That way, adding a node won't immediately set the state to Member and would tell a node which missed the configuration change to step down and rejoin.
|
Since this is parked. I'll remove myself as a reviewer until we want to discuss this design again. |
Removing nodes from the metadata store cluster is implemented by introducing
a transitional MetadataServerState::Leaving which tells the leader of the
metadata store cluster to remove the node which is Leaving. Only after the
configuration change has been committed and applied, the state will be set
to Outsider. If a node missed the configuration change but sees its state
in the NodesConfiguration switching to Outsider, it will stop Raft.
This PR sets the default
MetadataServerStatetoMemberso that newly added nodes will automatically try to join the existing metadata store cluster. Moreover, it adds arestatectl meta nodes add --id <NODE-ID>andrestatectl meta nodes remove --id <NODE-ID>command to add/remove nodes manually.cc @pcholakov