Create Position around Management Cluster #8210
Conversation
k8s-ci-robot
added
the
cncf-cla: no
k8s-ci-robot
added
sig/multicluster
|
Welcome @corentone! |
|
Hi @corentone. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/M
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
Not all multi-cluster management designs have a monolith. I do not think this SIG should take a position that requires or recommends a monolith. There are multiple roles involved in multi-cluster management. I think that it may be more helpful to identify those roles. A system with a monolith can be described as having one cluster that plays all those roles.
For example, in https://github.com/kubestellar/kubestellar/ we identify distinct roles and allow flexibility in what plays which role. We start by identifying a concept that is less than a "cluster". We define a "space" to be the fragment of cluster behavior that is only concerned with generic API machinery stuff. A space can store and serve kube API objects and subjects them to the general-purpose controllers (the ones that apply to all kinds of API objects, not the controllers involved specifically with containerized workload (Pod, Service, ...) ). KubeStellar defines the following roles.
- an "inventory space" holds the inventory of managed clusters. That is, API objects that refer to or describe the managed clusters. In OCM these are
ManagedClusterobjects. - a "workload description space (WDS)" holds both (a) objects that describe the workload (both desired state from users and reported state to users) and (b) control objects that control the multi-cluster management behavior.
- a "workload execution cluster (WEC)" is a managed cluster, where the workload is run or executed.
- the current KubeStellar implementation uses OCM and has a "transport space" that holds the workload interface objects of OCM (e.g.,
ManifestWork). The current implementation of KubeStellar requires that one space play both "inventory space" and "transport space" roles, calling this joint role "inventory and transport space (ITS)". - the current implementation of KubeStellar uses https://github.com/kubestellar/kubeflex as a space manager. The KubeFlex API objects reside in, and the KubeFlex central controller runs in, a role called the "KubeFlex hosting cluster".
- KubeFlex considers itself to manage "ControlPlanes", and supports a few different types. One is "vcluster", which uses https://www.vcluster.com/ to create virtual slices of its hosting cluster. Another is "k8s", which is essentially hosted Kubernetes kube-apiserver, its storage, and general-purpose controllers (the essence of a "space"). KubeFlex can also have a ControlPlane that represents its hosting cluster. We are working on enabling KubeFlex to adopt an arbitrary existing cluster as a ControlPlane.
One configuration that KubeStellar supports is one real cluster playing the roles of WDS, ITS, and KubeFlex hosting cluster.
In OCM, where the workload description is wrapped and the workload execution cluster holds the unwrapped objects, could one cluster play both WDS and WEC roles?
There was a problem hiding this comment.
Thank you for those details. I agree with your statement of not being a monolith.
I'm only trying to define the nuance of what a management cluster is; I don't think I force everything to be in the same or a monolith. Maybe my wording is wrong? I could rename the title to be "Management Clusters" to make it clearer.... but I'm just trying to get a wide definition of what such a cluster is (by opposition to a workload cluster).
There was a problem hiding this comment.
I wonder what is Kubestellar recommended way to host WDS/ITS/Kubeflex?
There was a problem hiding this comment.
Q: If the Inventory Space and the Workload Description Space are truly separate, how can we define an API that allows to schedule a workload on the clusters in the inventory?
There was a problem hiding this comment.
@MikeSpreitzer I think those are questions for you. Let me know if I can help.
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better |
There was a problem hiding this comment.
I am not convinced of this "SHOULD". I am not convinced that this sort of statement belongs in a definition.
There was a problem hiding this comment.
I wonder if you are not convinced that the "hub" cluster and a "spoke" cluster should not be the same or just "should" does not belong to a "definition"? I am curious about the reason if it's the former.
There was a problem hiding this comment.
@MikeSpreitzer could you clarify your statement please? If you mean that the management cluster relationship to workload clusters should be different than what the definition says, could you clarify your position (they MUST isolated, the definition shouldn't talk about their relationship or the SHOULD is too strong and it should be "Management cluster MAY also be a workload cluster")
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better | ||
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters |
There was a problem hiding this comment.
This statement supposes multiplicity only in the form of potentially competing equals; it omits the possibility of clusters fulfilling distinct roles.
There was a problem hiding this comment.
I tried to clarify in the next section that it could be multiple roles:
to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster)
is the wording not strong enough? I don't mean to close that door and thought the current wording was enough and not emitting a direction, just requiring that the admin oversees potential overlap between different management clusters. If there is no overlap, they are fine to co-exist as separate clusters.
There was a problem hiding this comment.
@MikeSpreitzer Did that answer your comment (e.g. can I resolve). I'm not sure I fully understood it and tried my best to answer it. Please let me know if I missed it.
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
This makes sense in general, but I am not convinced that there are no use cases for combining roles in one cluster.
There was a problem hiding this comment.
the first paragraph encourages different clusters for different roles actually.
This second paragraph is just about being part of the workload clusters or not.
Let me try to think of introducing the notion of "role" or something like that, as a subdivision of the broad Management Cluster.
There was a problem hiding this comment.
I wonder what's the definition of "workload"? I usually associate them with applications but not controllers so it's okey to me to run controllers in the "central" cluster that requires leader-election.
There was a problem hiding this comment.
it is definitely okay to run a controller in the management cluster. A controller doesn't have to be a "Workload" and be considered part of the "control-plane".
I think its the persona that matters. If it is a platform admin -owned controller, performing management tasks, I wouldn't consider it a workload. Workload to me is an application serving actual business-logic purpose.
All in all, running those management controllers is the reason why I want to define management clusters and not just a management API. (we had discussed internally giving simply an API with machinery... but then very quickly you want to bring a controller to act on this API and look where to run it)
There was a problem hiding this comment.
@ryanzhang-oss should I define workload more? I think a sentence in this doc may be enough.
Workload works great when opposing management, for hub maybe we need to say spoke? or is workload still good?
There was a problem hiding this comment.
AFAIK, workload in k8s basically means things like deployment/daemonset/statefulset, so it can be anything. I am not sure if there is a way to say the deployment contains a controller vs a real application.
k8s-ci-robot
added
cncf-cla: yes
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
Thank you for those details. I agree with your statement of not being a monolith.
I'm only trying to define the nuance of what a management cluster is; I don't think I force everything to be in the same or a monolith. Maybe my wording is wrong? I could rename the title to be "Management Clusters" to make it clearer.... but I'm just trying to get a wide definition of what such a cluster is (by opposition to a workload cluster).
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better | ||
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters |
There was a problem hiding this comment.
I tried to clarify in the next section that it could be multiple roles:
to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster)
is the wording not strong enough? I don't mean to close that door and thought the current wording was enough and not emitting a direction, just requiring that the admin oversees potential overlap between different management clusters. If there is no overlap, they are fine to co-exist as separate clusters.
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
the first paragraph encourages different clusters for different roles actually.
This second paragraph is just about being part of the workload clusters or not.
Let me try to think of introducing the notion of "role" or something like that, as a subdivision of the broad Management Cluster.
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
I wonder what is Kubestellar recommended way to host WDS/ITS/Kubeflex?
|
|
||
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative |
There was a problem hiding this comment.
I wonder what does "visibility" mean here? There can be workload clusters that do not have public IP.
There was a problem hiding this comment.
Agreed, visibility is vague. I think it is somewhat the ability to query more data about the cluster. I'm thinking it would be the ability to "READ" via kubectl (thought it may not be everything). But it should be able to potentially see the Cluster object in GKE or other platforms (wherever the provisioning/lifecycle happens).
Maybe this is too vague, I originally had this in to mandate some kind of visibility on the cluster so that the controller could do something (if the controller has only access to the clusterprofile, I dont know what it can really do).
But we may be able to remove it and it wouldn't affect the definition much.
There was a problem hiding this comment.
I don't think that MUST is correct then.
several existing multi-cluster systems work without the hub being able to read the apiserver of the member.
The reason is that having 1 system (the hub) able to read ALL members is a different security model than having each individual member able to access the hub.
There was a problem hiding this comment.
@serbrech good point; the must is likely too strong then. Would a "should" be okay? Or should it be a "may"? Now, even if we reverse the security model (e.g. work api model where the cluster reaches out the central cluster), there is still power from the hub over the workload cluster given it.
Here is what I could change:
| them). It MUST have visibility over the available clusters and MAY have administrative | |
| them). It MUST have the ClusterProfiles written on it MAY have visibility over the available clusters and have administrative |
(I could also drop the clusterprofile part as obvious)
There was a problem hiding this comment.
Thank you for revisiting. I'm aligned with MAY.
Specific features could require this visibility, but not the general use case of a hub cluster.
There was a problem hiding this comment.
In my systems, the workload clusters pull from the management cluster. Controllers run on the workload clusters so MAY have administrative privileges works well in my world.
Visible is interesting for my world; my management clusters are authoritative for the data and the workload clusters reconcile against it. But, there MAY be two management clusters from which a workload cluster pulls (i.e. a dev and prod source-of-truth); each management cluster authoritative for particular data.
Lastly, could visibility be decomposed into data storage about a workload cluster and mechanism for transport/reconciliation between the hub/workload cluster?
There was a problem hiding this comment.
I've removed visibility to be "access to api, metrics or workloads". So it could be that it doesn't access any of those or a subset of them.
@cbaenziger I'm not sure what you mean about data storage for visibility, apiserver/etcd?
Visible is interesting for my world; my management clusters are authoritative for the data and the workload clusters reconcile against it. But, there MAY be two management clusters from which a workload cluster pulls (i.e. a dev and prod source-of-truth); each management cluster authoritative for particular data.
we should talk more about this example. There could be multiple multiple management clusters for a given cluster, and the definition allows for it.
Now putting my infra hat (not PR writer), given the current isolation of kubernetes between workloads, I would prefer to mix Business Units with a similar risk) instead of different environments (prod vs dev).
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better |
There was a problem hiding this comment.
I wonder if you are not convinced that the "hub" cluster and a "spoke" cluster should not be the same or just "should" does not belong to a "definition"? I am curious about the reason if it's the former.
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
I wonder what's the definition of "workload"? I usually associate them with applications but not controllers so it's okey to me to run controllers in the "central" cluster that requires leader-election.
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better |
There was a problem hiding this comment.
@MikeSpreitzer could you clarify your statement please? If you mean that the management cluster relationship to workload clusters should be different than what the definition says, could you clarify your position (they MUST isolated, the definition shouldn't talk about their relationship or the SHOULD is too strong and it should be "Management cluster MAY also be a workload cluster")
|
|
||
| A (multicluster) management cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have visibility over the available clusters and MAY have administrative |
There was a problem hiding this comment.
@serbrech good point; the must is likely too strong then. Would a "should" be okay? Or should it be a "may"? Now, even if we reverse the security model (e.g. work api model where the cluster reaches out the central cluster), there is still power from the hub over the workload cluster given it.
Here is what I could change:
| them). It MUST have visibility over the available clusters and MAY have administrative | |
| them). It MUST have the ClusterProfiles written on it MAY have visibility over the available clusters and have administrative |
(I could also drop the clusterprofile part as obvious)
| them). It MUST have visibility over the available clusters and MAY have administrative | ||
| privileges over them. It SHOULD not be part of workload clusters to provide a better | ||
| security isolation, especially when it has any administrative privileges over them. | ||
| There MAY be multiple management clusters overseeing the same set of Workload Clusters |
There was a problem hiding this comment.
@MikeSpreitzer Did that answer your comment (e.g. can I resolve). I'm not sure I fully understood it and tried my best to answer it. Please let me know if I missed it.
| config-delivery management cluster), to allow for migrations (from old management cluster to new | ||
| management cluster) and likely more. | ||
| * Management cluster also being part of the workload-running Fleet: We do recommend that the | ||
| management cluster(s) be isolated from the running Workload Fleet for security and management |
There was a problem hiding this comment.
@ryanzhang-oss should I define workload more? I think a sentence in this doc may be enough.
Workload works great when opposing management, for hub maybe we need to say spoke? or is workload still good?
| @@ -0,0 +1,46 @@ | |||
| # Management Cluster - SIG Multicluster Position Statement | |||
There was a problem hiding this comment.
@MikeSpreitzer I think those are questions for you. Let me know if I can help.
|
Couple of questions on this: Would be good to state more explicitly the goal for this position doc -- is it:
|
| There MAY be multiple management clusters overseeing the same set of Workload Clusters | ||
| and it is left to the administrator to guarantee that they don't compete in their | ||
| management tasks. There SHOULD be a single clusterset managed by a management cluster. | ||
| Management clusters can be used for both control-plane or data-plane features. |
There was a problem hiding this comment.
I think it would be helpful to say what is meant here by "control-plane" and "data-plane".
In the call on Tuesday I heard something like the data plane is about the workload and the control plane is about controlling the workload's propagation from hub to execution clusters. (I specifically say "execution" clusters to contrast with where the authoritative description of the workload lives in the hub, which might be in a cluster or something like it. In KubeStellar we keep the hub-side authoritative description of the workload un-wrapped in what we call a "space", which something that has (at least) a cluster's API machinery.)
There was a problem hiding this comment.
I tried to clarify by differentiating "Business applications" with "Infrastructure".
I steered away from execution because people think about infra pods which are being run and get confused on where they would go. I think kubestellar "execution cluster" is equal to what I'd call "workload cluster".
I think your "space" would live on a hub cluster, are you suggesting we should separate the definition of hub and space cluster? (separating infra and app-infra)
Side notes that I may add to the definition/doc if that helps and the current wording doesn't cover it:
There are two types of Workloads: Business/Product/application vs Infrastructure/Platform.
There are two personas and the configs they manage: Platform/Infra Team and Product/Application/Business teams.
I consider workload clusters as running (e.g. pods serving traffic or achieving functionality) business applications. Workload clusters may also run infrastructure applications that assist them in serving.
Hub clusters run infrastructure applications and may hold infrastructure/definitions for business applications.
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
…ster-position-statement.md * added link to clusterset doc * added a context sentence of workload vs hub * removed "visibility" to make about api, metrics or workload access. * Added a rationale on infra vs app controllers. There may be hub clusters that run infra multicluster, but there may also be some that run application infra things. That means the personas accessing them (and their permission) may be different.
|
@bowei I missed this last edit; will add more. replying here and will edit the doc again. The goal for this doc is a definition. |
|
Naming Poll is now closed; Hub won with 38 votes, ahead of the 30 votes for Management and 1 for Command. I've renamed it. |
Co-authored-by: Ryan Zhang <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: corentone, ryanzhang-oss The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
@ryanzhang-oss: changing LGTM is restricted to collaborators In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| available. There has been a variety of examples of which we can quote ArgoCD, MultiKueue | ||
| or any of the Federation effort (Karmada, KubeAdmiral), all of them not-naming the "location" |
There was a problem hiding this comment.
Shouldn't Open Cluster Management be included in this list?
|
|
||
| A (multicluster) hub cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate | ||
| them). It MUST have the ClusterProfiles written on it MAY have access to the api, metrics or |
There was a problem hiding this comment.
Is this definition intended to cover what Open Cluster Management does? OCM has its own representation of a workload (execution) cluster, called ManagedCluster.
| ## Definition | ||
|
|
||
| A (multicluster) hub cluster is a Kubernetes cluster that acts as a | ||
| control-plane for other Kubernetes clusters (named Workload Clusters to differentiate |
There was a problem hiding this comment.
Could we maybe say something like "workload execution cluster", to differentiate it from central cluster(s) that hold the central description of the desired workload and centrally reported state of the workload?
No description provided.