Skip to content

Commit 20de7b7

Browse files
ArvindthiruArvindthiru
authored
docs: Eviction & Placement Disruption budget (#1007)
1 parent a7f9b82 commit 20de7b7

File tree

6 files changed

+266
-5
lines changed

6 files changed

+266
-5
lines changed

‎docs/concepts/EvictionAndDisruptionBudget/README.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Eviction & Placement Disruption Budget
2+
3+
This document explains the concept of `Eviction` and `Placement Disruption Budget` in the context of the fleet.
4+
5+
## Overview
6+
7+
`Eviction` provides a way to force remove resources from a target cluster once the resources have already been propagated from the hub cluster by a `Placement` object.
8+
`Eviction` is considered as an voluntary disruption triggered by the user. `Eviction` alone doesn't guarantee that resources won't be propagated to target cluster again by the scheduler.
9+
The users need to use [taints](../../howtos/taint-toleration.md) in conjunction with `Eviction` to prevent the scheduler from picking the target cluster again.
10+
11+
The `Placement Disruption Budget` object protects against voluntary disruptions.
12+
13+
The only voluntary disruption that can occur in the fleet is the eviction of resources from a target cluster which can be achieved by creating the `ClusterResourcePlacementEviction` object.
14+
15+
Some cases of involuntary disruptions in the context of fleet,
16+
- The removal of resources from a member cluster by the scheduler due to scheduling policy changes.
17+
- Users manually deleting workload resources running on a member cluster.
18+
- Users manually deleting the `ClusterResourceBinding` object which is an internal resource the represents the placement of resources on a member cluster.
19+
- Workloads failing to run properly on a member cluster due to misconfiguration or cluster related issues.
20+
21+
For all the cases of involuntary disruptions described above, the `Placement Disruption Budget` object does not protect against them.
22+
23+
## ClusterResourcePlacementEviction
24+
25+
An eviction object is used to remove resources from a member cluster once the resources have already been propagated from the hub cluster.
26+
27+
The eviction object is only reconciled once after which it reaches a terminal state. Below is the list of terminal states for `ClusterResourcePlacementEviction`,
28+
- `ClusterResourcePlacementEviction` is valid and it's executed successfully.
29+
- `ClusterResourcePlacementEviction` is invalid.
30+
- `ClusterResourcePlacementEviction` is valid but it's not executed.
31+
32+
To successfully evict resources from a cluster, the user needs to specify:
33+
34+
- The name of the `ClusterResourcePlacement` object which propagated resources to the target cluster.
35+
- The name of the target cluster from which we need to evict resources.
36+
37+
When specifying the `ClusterResourcePlacement` object in the eviction's spec, the user needs to consider the following cases:
38+
39+
- For `PickFixed` CRP, eviction is not allowed; it is recommended that one directly edit the list of target clusters on the CRP object.
40+
- For `PickAll` & `PickN` CRPs, eviction is allowed because the users cannot deterministically pick or unpick a cluster based on the placement strategy; it's up to the scheduler.
41+
42+
> **Note:** After an eviction is executed, there is no guarantee that the cluster won't be picked again by the scheduler to propagate resources for a `ClusterResourcePlacement` resource.
43+
> The user needs to specify a [taint](../../howtos/taint-toleration.md) on the cluster to prevent the scheduler from picking the cluster again. This is especially true for `PickAll ClusterResourcePlacement` because
44+
> the scheduler will try to propagate resources to all the clusters in the fleet.
45+
46+
## ClusterResourcePlacementDisruptionBudget
47+
48+
The `ClusterResourcePlacementDisruptionBudget` is used to protect resources propagated by a `ClusterResourcePlacement` to a target cluster from voluntary disruption, i.e., `ClusterResourcePlacementEviction`.
49+
50+
> **Note:** When specifying a `ClusterResourcePlacementDisruptionBudget`, the name should be the same as the `ClusterResourcePlacement` that it's trying to protect.
51+
52+
Users are allowed to specify one of two fields in the `ClusterResourcePlacementDisruptionBudget` spec since they are mutually exclusive:
53+
54+
- MaxUnavailable - specifies the maximum number of clusters in which a placement can be unavailable due to any form of disruptions.
55+
- MinAvailable - specifies the minimum number of clusters in which placements are available despite any form of disruptions.
56+
57+
for both `MaxUnavailable` and `MinAvailable`, the user can specify the number of clusters as an integer or as a percentage of the total number of clusters in the fleet.
58+
59+
> **Note:** For both MaxUnavailable and MinAvailable, involuntary disruptions are not subject to the disruption budget but will still count against it.
60+
61+
When specifying a disruption budget for a particular `ClusterResourcePlacement`, the user needs to consider the following cases:
62+
63+
| CRP type | `MinAvailable` DB with an integer | `MinAvailable` DB with a percentage | `MaxUnavailable` DB with an integer | `MaxUnavailable` DB with a percentage |
64+
|--------------|-----------------------------------|-------------------------------------|-------------------------------------|---------------------------------------|
65+
| `PickFixed` | ❌ | ❌ | ❌ | ❌ |
66+
| `PickAll` | ✅ | ❌ | ❌ | ❌ |
67+
| `PickN` | ✅ | ✅ | ✅ | ✅ |
68+
69+
> **Note:** We don't allow eviction for `PickFixed` CRP and hence specifying a `ClusterResourcePlacementDisruptionBudget` for `PickFixed` CRP does nothing.
70+
> And for `PickAll` CRP, the user can only specify `MinAvailable` because total number of clusters selected by a `PickAll` CRP is non-deterministic.
71+
> If the user creates an invalid `ClusterResourcePlacementDisruptionBudget` object, when an eviction is created, the eviction won't be successfully executed.

‎docs/concepts/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,6 @@ Allow slightly different manifests depends on the cluster it lands.
2727

2828
## [PropertyProvider](PropertyProviderAndClusterProperties/README.md)
2929
More ways to select the clusters based on its property.
30+
31+
## [Eviction & Placement Disruption Budget](EvictionAndDisruptionBudget/README.md)
32+
Understand how placements can be evicted and protected from voluntary disruptions in fleet.

‎docs/howtos/README.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,4 +56,10 @@ domains:
5656

5757
This how-to guide explains the specifics of the Fleet `ResourceOverride` API, including its
5858
resource selectors, policy, and more. `ResourceOverride` is a Fleet API that allows you to
59-
modify or override specific attributes across namespaced resources.
59+
modify or override specific attributes across namespaced resources.
60+
61+
* [Using the Fleet `ClusterResourcePlacementEviction` and `ClusterResourcePlacementDisruptionBudget` APIs](eviction-placement-disruption-budget.md)
62+
63+
This how-to guide explains the specifics of the Fleet `ClusterResourcePlacementEviction` and
64+
`ClusterResourcePlacementDisruptionBudget` APIs, including how to evict resources from a
65+
cluster and protect resources from voluntary disruption.

‎docs/howtos/eviction-placement-disruption-budget.md

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Using ClusterResourcePlacementEviction and ClusterResourcePlacementDisruptionBudget
2+
3+
This how-to guide discusses how to create `ClusterResourcePlacementEviction` objects and `ClusterResourcePlacementDisruptionBudget` objects to evict resources from member clusters and protect resources on member clusters from voluntary disruption, respectively.
4+
5+
## Evicting Resources from Member Clusters using ClusterResourcePlacementEviction
6+
7+
The `ClusterResourcePlacementEviction` object is used to remove resources from a member cluster once the resources have already been propagated from the hub cluster.
8+
9+
To successfully evict resources from a cluster, the user needs to specify:
10+
- The name of the `ClusterResourcePlacement` object which propagated resources to the target cluster.
11+
- The name of the target cluster from which we need to evict resources.
12+
13+
In this example, we will create a `ClusterResourcePlacement` object with PickAll placement policy to propagate resources to an existing `MemberCluster`, add a taint to the member cluster
14+
resource and then create a `ClusterResourcePlacementEviction` object to evict resources from the `MemberCluster`.
15+
16+
We will first create a namespace that we will propagate to the member cluster.
17+
18+
```
19+
kubectl create ns test-ns
20+
```
21+
22+
Then we will apply a `ClusterResourcePlacement` with the following spec:
23+
24+
```yaml
25+
spec:
26+
resourceSelectors:
27+
- group: ""
28+
kind: Namespace
29+
version: v1
30+
name: test-ns
31+
policy:
32+
placementType: PickAll
33+
```
34+
35+
The `CRP` status after applying should look something like this:
36+
37+
```yaml
38+
kubectl get crp test-crp
39+
NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE
40+
test-crp 2 True 2 True 2 5m49s
41+
```
42+
43+
let's now add a taint to the member cluster to ensure this cluster is not picked again by the scheduler once we evict resources from it.
44+
45+
Modify the cluster object to add a taint:
46+
47+
```yaml
48+
spec:
49+
heartbeatPeriodSeconds: 60
50+
identity:
51+
kind: ServiceAccount
52+
name: fleet-member-agent-cluster-1
53+
namespace: fleet-system
54+
taints:
55+
- effect: NoSchedule
56+
key: test-key
57+
value: test-value
58+
```
59+
60+
Now we will create a `ClusterResourcePlacementEviction` object to evict resources from the member cluster:
61+
62+
```yaml
63+
apiVersion: placement.kubernetes-fleet.io/v1beta1
64+
kind: ClusterResourcePlacementEviction
65+
metadata:
66+
name: test-eviction
67+
spec:
68+
placementName: test-crp
69+
clusterName: kind-cluster-1
70+
```
71+
72+
the eviction object should look like this, if the eviction was successful:
73+
74+
```yaml
75+
kubectl get crpe test-eviction
76+
NAME VALID EXECUTED
77+
test-eviction True True
78+
```
79+
80+
since the eviction is successful, the resources should be removed from the cluster, let's take a look at the `CRP` object status to verify:
81+
82+
```yaml
83+
kubectl get crp test-crp
84+
NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE
85+
test-crp 2 True 2 15m
86+
```
87+
88+
from the object we can clearly tell that the resources were evicted since the `AVAILABLE` column is empty. If the user needs more information `ClusterResourcePlacement` object's status can be checked.
89+
90+
## Protecting resources from voluntary disruptions using ClusterResourcePlacementDisruptionBudget
91+
92+
In this example, we will create a `ClusterResourcePlacement` object with PickN placement policy to propagate resources to an existing MemberCluster,
93+
then create a `ClusterResourcePlacementDisruptionBudget` object to protect resources on the MemberCluster from voluntary disruption and
94+
then try to evict resources from the MemberCluster using `ClusterResourcePlacementEviction`.
95+
96+
We will first create a namespace that we will propagate to the member cluster.
97+
98+
```
99+
kubectl create ns test-ns
100+
```
101+
102+
Then we will apply a `ClusterResourcePlacement` with the following spec:
103+
104+
```yaml
105+
spec:
106+
resourceSelectors:
107+
- group: ""
108+
kind: Namespace
109+
version: v1
110+
name: test-ns
111+
policy:
112+
placementType: PickN
113+
numberOfClusters: 1
114+
```
115+
116+
The `CRP` object after applying should look something like this:
117+
118+
```yaml
119+
kubectl get crp test-crp
120+
NAME GEN SCHEDULED SCHEDULED-GEN AVAILABLE AVAILABLE-GEN AGE
121+
test-crp 2 True 2 True 2 8s
122+
```
123+
124+
Now we will create a `ClusterResourcePlacementDisruptionBudget` object to protect resources on the member cluster from voluntary disruption:
125+
126+
```yaml
127+
apiVersion: placement.kubernetes-fleet.io/v1beta1
128+
kind: ClusterResourcePlacementDisruptionBudget
129+
metadata:
130+
name: test-crp
131+
spec:
132+
minAvailable: 1
133+
```
134+
135+
> **Note:** An eviction object is only reconciled once, after which it reaches a terminal state, if the user desires to create/apply the same eviction object again they need to delete the existing eviction object and re-create the object for the eviction to occur again.
136+
137+
Now we will create a `ClusterResourcePlacementEviction` object to evict resources from the member cluster:
138+
139+
```yaml
140+
apiVersion: placement.kubernetes-fleet.io/v1beta1
141+
kind: ClusterResourcePlacementEviction
142+
metadata:
143+
name: test-eviction
144+
spec:
145+
placementName: test-crp
146+
clusterName: kind-cluster-1
147+
```
148+
149+
> **Note:** The eviction controller will try to get the corresponding `ClusterResourcePlacementDisruptionBudget` object when a `ClusterResourcePlacementEviction` object is reconciled to check if the specified MaxAvailable or MinAvailable allows the eviction to be executed.
150+
151+
let's take a look at the eviction object to see if the eviction was executed,
152+
153+
```yaml
154+
kubectl get crpe test-eviction
155+
NAME VALID EXECUTED
156+
test-eviction True False
157+
```
158+
159+
from the eviction object we can see the eviction was not executed.
160+
161+
let's take a look at the `ClusterResourcePlacementEviction` object status to verify why the eviction was not executed:
162+
163+
```yaml
164+
status:
165+
conditions:
166+
- lastTransitionTime: "2025-01-21T15:52:29Z"
167+
message: Eviction is valid
168+
observedGeneration: 1
169+
reason: ClusterResourcePlacementEvictionValid
170+
status: "True"
171+
type: Valid
172+
- lastTransitionTime: "2025-01-21T15:52:29Z"
173+
message: 'Eviction is blocked by specified ClusterResourcePlacementDisruptionBudget,
174+
availablePlacements: 1, totalPlacements: 1'
175+
observedGeneration: 1
176+
reason: ClusterResourcePlacementEvictionNotExecuted
177+
status: "False"
178+
type: Executed
179+
```
180+
181+
the eviction status clearly mentions that the eviction was blocked by the specified `ClusterResourcePlacementDisruptionBudget`.

‎examples/eviction/clusterpdb.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
apiVersion: placement.kubernetes-fleet.io/v1alpha1
1+
apiVersion: placement.kubernetes-fleet.io/v1beta1
22
kind: ClusterResourcePlacementDisruptionBudget
33
metadata:
44
name: test-crp
55
spec:
6-
maxUnavailable: 1
6+
minAvailable: 1

‎examples/eviction/eviction.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
apiVersion: placement.kubernetes-fleet.io/v1alpha1
1+
apiVersion: placement.kubernetes-fleet.io/v1beta1
22
kind: ClusterResourcePlacementEviction
33
metadata:
44
name: test-eviction
55
spec:
66
placementName: test-crp
7-
clusterName: cluster-1
7+
clusterName: kind-cluster-1

0 commit comments

Comments
 (0)