Skip to content

Commit 4a7c8b6

Browse files
raihankhantamalsaha
authored andcommitted
Update
Signed-off-by: raihankhan <[email protected]>
1 parent d140f25 commit 4a7c8b6

File tree

11 files changed

+1701
-77
lines changed

11 files changed

+1701
-77
lines changed

docs/guides/elasticsearch/recommendation/_index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ menu:
88
weight: 60
99
menu_name: docs_{{ .version }}
1010
---
11+

docs/guides/elasticsearch/recommendation/overview.md

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ section_menu_id: guides
1313

1414
> New to KubeDB? Please start [here](/docs/README.md).
1515
16-
# Recommendation for KubeDB managed Elasticsearch
16+
# Recommendation for KubeDB managed Elasticsearch and Opensearch
1717

1818
Databases on Kubernetes in production grade infrastructure often need to go through several administrative operations depending on specific resource requirements. Such operations include vertical scaling (cpu, memory) and storage expansion. Autoscaling support for KubeDB managed databases takes care of it. However, databases also need to go through some maintenance operations in order to ensure security, enhance performance, getting bug fixes and new features etc. Such operations mostly require organization's manual intervention. Even if these operations are automated, they need to be done in surveillance. KubeDB simplifies this by generating K8s Native Recommendations.
1919

@@ -30,16 +30,13 @@ KubeDB provisioner watches user provided database custom resource spec and creat
3030
KubeDB provides Three types of recommendation for Elasticsearch and Opensearch.
3131

3232
1. [Version Update Recommendation](/docs/guides/elasticsearch/recommendation/version-update-recommendation.md)
33-
2. TLS Certificate Rotation Recommendation
34-
3. Authentication Secret Rotation Recommendation
33+
2. [TLS Certificate Rotation Recommendation](/docs/guides/elasticsearch/recommendation/rotate-tls-recommendation.md)
34+
3. [Authentication Secret Rotation Recommendation](/docs/guides/elasticsearch/recommendation/rotate-auth-recommendation.md)
3535

3636
The next page describes these recommendations, how to approve/reject them, their generation mechanism and usability.
3737

3838
## Next Steps
3939

40-
- Learn how to monitor Elasticsearch database with KubeDB using [builtin-Prometheus](/docs/guides/elasticsearch/monitoring/using-builtin-prometheus.md) and using [Prometheus operator](/docs/guides/elasticsearch/monitoring/using-prometheus-operator.md).
41-
- Learn how to monitor PostgreSQL database with KubeDB using [builtin-Prometheus](/docs/guides/postgres/monitoring/using-builtin-prometheus.md) and using [Prometheus operator](/docs/guides/postgres/monitoring/using-prometheus-operator.md).
42-
- Learn how to monitor MySQL database with KubeDB using [builtin-Prometheus](/docs/guides/mysql/monitoring/builtin-prometheus/index.md) and using [Prometheus operator](/docs/guides/mysql/monitoring/prometheus-operator/index.md).
43-
- Learn how to monitor MongoDB database with KubeDB using [builtin-Prometheus](/docs/guides/mongodb/monitoring/using-builtin-prometheus.md) and using [Prometheus operator](/docs/guides/mongodb/monitoring/using-prometheus-operator.md).
44-
- Learn how to monitor Redis server with KubeDB using [builtin-Prometheus](/docs/guides/redis/monitoring/using-builtin-prometheus.md) and using [Prometheus operator](/docs/guides/redis/monitoring/using-prometheus-operator.md).
45-
- Learn how to monitor Memcached server with KubeDB using [builtin-Prometheus](/docs/guides/memcached/monitoring/using-builtin-prometheus.md) and using [Prometheus operator](/docs/guides/memcached/monitoring/using-prometheus-operator.md).
40+
- Learn about Elasticsearch [Version Update Recommendation](/docs/guides/elasticsearch/recommendation/version-update-recommendation.md).
41+
- Learn about Elasticsearch [TLS Certificate Rotation Recommendation](/docs/guides/elasticsearch/recommendation/rotate-tls-recommendation.md)
42+
- Learn about Elasticsearch [Authentication Secret Rotation Recommendation](/docs/guides/elasticsearch/recommendation/rotate-auth-recommendation.md)

docs/guides/elasticsearch/recommendation/rotate-auth-recommendation.md

Lines changed: 248 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ menu:
55
identifier: es-rotate-auth-recommendation
66
name: Rotate Auth Recommendation
77
parent: es-recommendation-elasticsearch
8-
weight: 30
8+
weight: 40
99
menu_name: docs_{{ .version }}
1010
section_menu_id: guides
1111
---
@@ -14,15 +14,14 @@ section_menu_id: guides
1414
1515
# Elasticsearch Version Update Recommendation
1616

17-
Rotating authentication secrets in database management is vital to mitigate security risks, such as credential leakage or unauthorized access, and to comply with regulatory requirements. Regular rotation limits the exposure of compromised credentials, reduces the risk of insider threats, and enforces updated security policies like stronger passwords or algorithms. It also ensures operational resilience by testing the rotation process and revoking stale or unused credentials. KubeDB provides `RotateAuth OpsRequest` which reduces manual errors, and strengthens database security with minimal effort. KubeDB Ops-manager generates Recommendation for rotating authentication secrets via this OpsRequest.
17+
Rotating authentication secrets in database management is vital to mitigate security risks, such as credential leakage or unauthorized access, and to comply with regulatory requirements. Regular rotation limits the exposure of compromised credentials, reduces the risk of insider threats, and enforces updated security policies like stronger passwords or algorithms. It also ensures operational resilience by testing the rotation process and revoking stale or unused credentials. KubeDB provides `RotateAuth` which reduces manual errors, and strengthens database security with minimal effort. KubeDB Ops-manager generates Recommendation for rotating authentication secrets via this OpsRequest.
1818

19-
`Recommendation` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative recommendation for KubeDB managed databases like [Elasticsearch](https://www.elastic.co/products/elasticsearch) and [OpenSearch](https://opensearch.org/) in a Kubernetes native way. KubeDB generates Elasticsearch/Opensearch Rotate Auth recommendation regarding three particular cases.
19+
`Recommendation` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative recommendation for KubeDB managed databases like [Elasticsearch](https://www.elastic.co/products/elasticsearch) and [OpenSearch](https://opensearch.org/) in a Kubernetes native way. The recommendation will only be created if `.spec.authSecret.rotateAfter` is set. KubeDB generates Elasticsearch/Opensearch Rotate Auth recommendation regarding two particular cases.
2020

21-
1. There's been an update in the current version image
22-
2. There's a new version available with a minor/patch fix.
23-
3. There's a new major version available
21+
1. AuthSecret lifespan is more than one month and, less than one month remaining till expiry
22+
2. AuthSecret lifespan is less than one month and, less than one third of lifespan remaining till expiry
2423

25-
Let's go through a demo to see version update recommendations being generated. First, get the available Elasticsearch versions provided by KubeDB.
24+
Let's go through a demo to see `RotateAuth` recommendations being generated. First, get the available Elasticsearch versions provided by KubeDB.
2625

2726
```bash
2827
$ kubectl get elasticsearchversions | grep xpack
@@ -47,48 +46,254 @@ xpack-8.6.2 8.6.2 ElasticStack ghcr.io/appscode-images/elastic:8.6
4746
xpack-8.8.2 8.8.2 ElasticStack ghcr.io/appscode-images/elastic:8.8.2 17h
4847
```
4948

50-
Let's deploy an Elasticsearch cluster with version `xpack-8.15.0`.
49+
Let's deploy an Elasticsearch cluster with version `xpack-8.15.0`. We are going to create a cluster topology with 2 master nodes, 3 data nodes and 2 ingest node. We also have to provide an available storageclass for each of the node types.
5150

5251
```yaml
53-
apiVersion: kubedb.com/v1
54-
kind: Elasticsearch
52+
apiVersion: kubedb.com/v1
53+
kind: Elasticsearch
54+
metadata:
55+
name: elastic
56+
namespace: es
57+
spec:
58+
version: xpack-8.15.0
59+
storageType: Durable
60+
deletionPolicy: WipeOut
61+
authSecret:
62+
rotateAfter: 1h
63+
topology:
64+
master:
65+
replicas: 2
66+
storage:
67+
storageClassName: "local-path"
68+
accessModes:
69+
- ReadWriteOnce
70+
resources:
71+
requests:
72+
storage: 1Gi
73+
data:
74+
replicas: 2
75+
storage:
76+
storageClassName: "local-path"
77+
accessModes:
78+
- ReadWriteOnce
79+
resources:
80+
requests:
81+
storage: 1Gi
82+
ingest:
83+
replicas: 1
84+
storage:
85+
storageClassName: "local-path"
86+
accessModes:
87+
- ReadWriteOnce
88+
resources:
89+
requests:
90+
storage: 1Gi
91+
```
92+
93+
Wait for a while till elasicsearch cluster gets into `Ready` state. Required time depends on image pulling and node's physical specifications.
94+
95+
```bash
96+
$ kubectl get es elastic -n es -w
97+
NAME VERSION STATUS AGE
98+
elastic xpack-8.15.0 Provisioning 98s
99+
elastic xpack-8.15.0 Provisioning 5m43s
100+
elastic xpack-8.15.0 Provisioning 8m7s
101+
.
102+
.
103+
.
104+
elastic xpack-8.15.0 Ready 10m
105+
elastic xpack-8.15.0 Ready 10m
106+
```
107+
108+
Since, `.spec.authSecret.rotateAfter` is set as `1h`, it is expected that the recommendation engine will generate a rotate-auth recommendation at least after 40 minutes (two-third of lifespan) of the authsecret creation. Once generated you will get a similar recommendation as follows.
109+
110+
```bash
111+
$ kubectl get recommendation -n es | grep rotate-auth
112+
NAME STATUS OUTDATED AGE
113+
elastic-x-elasticsearch-x-rotate-auth-2juuee Pending false 10m
114+
```
115+
116+
The `Recommendation` custom resource will be named as `<DB-name>-x-<DB type>-x-<Recommendation type>-<random hash>`. Initially, the KubeDB `Supervisor` controller will mark the `Status` of this object to `Pending`. Let's check the complete Recommendation custom resource manifest:
117+
118+
```yaml
119+
$ kubectl get recommendation -n es elastic-x-elasticsearch-x-rotate-auth-2juuee -oyaml
120+
apiVersion: supervisor.appscode.com/v1alpha1
121+
kind: Recommendation
122+
metadata:
123+
creationTimestamp: "2025-02-25T09:12:29Z"
124+
generation: 1
125+
labels:
126+
app.kubernetes.io/instance: elastic
127+
app.kubernetes.io/managed-by: kubedb.com
128+
app.kubernetes.io/type: rotate-auth
129+
name: elastic-x-elasticsearch-x-rotate-auth-2juuee
130+
namespace: es
131+
resourceVersion: "80116"
132+
uid: 12f24cf6-2f02-420f-863d-3523e32a08dd
133+
spec:
134+
backoffLimit: 5
135+
deadline: "2025-02-25T09:20:53Z"
136+
description: Recommending AuthSecret rotation,elastic-auth AuthSecret needs to be
137+
rotated before 2025-02-25 09:30:53 +0000 UTC
138+
operation:
139+
apiVersion: ops.kubedb.com/v1alpha1
140+
kind: ElasticsearchOpsRequest
141+
metadata:
142+
name: rotate-auth
143+
namespace: es
144+
spec:
145+
databaseRef:
146+
name: elastic
147+
type: RotateAuth
148+
status: {}
149+
recommender:
150+
name: kubedb-ops-manager
151+
rules:
152+
failed: has(self.status) && has(self.status.phase) && self.status.phase == 'Failed'
153+
inProgress: has(self.status) && has(self.status.phase) && self.status.phase ==
154+
'Progressing'
155+
success: has(self.status) && has(self.status.phase) && self.status.phase == 'Successful'
156+
target:
157+
apiGroup: kubedb.com
158+
kind: Elasticsearch
159+
name: elastic
160+
status:
161+
approvalStatus: Pending
162+
failedAttempt: 0
163+
outdated: false
164+
parallelism: Namespace
165+
phase: Pending
166+
reason: WaitingForApproval
167+
```
168+
169+
In the generated Recommendation you will find a description, targeted db object, recommended operation or Ops-Request manifest, current status of the recommendation etc. Let's just focus on the recommendation description first.
170+
171+
```shell
172+
$ kubectl get recommendation -n es elastic-x-elasticsearch-x-rotate-auth-2juuee -o jsonpath='{.spec.operation}' | yq -y
173+
apiVersion: ops.kubedb.com/v1alpha1
174+
kind: ElasticsearchOpsRequest
55175
metadata:
56-
name: es
57-
namespace: demo
176+
name: rotate-auth
177+
namespace: es
58178
spec:
59-
version: xpack-8.15.0
60-
storageType: Durable
61-
deletionPolicy: WipeOut
62-
topology:
63-
master:
64-
replicas: 2
65-
storage:
66-
storageClassName: "standard"
67-
accessModes:
68-
- ReadWriteOnce
69-
resources:
70-
requests:
71-
storage: 1Gi
72-
data:
73-
replicas: 2
74-
storage:
75-
storageClassName: "standard"
76-
accessModes:
77-
- ReadWriteOnce
78-
resources:
79-
requests:
80-
storage: 1Gi
81-
ingest:
82-
replicas: 1
83-
storage:
84-
storageClassName: "standard"
85-
accessModes:
86-
- ReadWriteOnce
87-
resources:
88-
requests:
89-
storage: 1Gi
179+
databaseRef:
180+
name: elastic
181+
type: RotateAuth
182+
status: {}
90183
```
91184

185+
Let's check the status part of this recommendation.
186+
187+
```bash
188+
$ kubectl get recommendation -n es elastic-x-elasticsearch-x-rotate-auth-2juuee -o jsonpath='{.status}' | yq -y
189+
approvalStatus: Pending
190+
failedAttempt: 0
191+
outdated: false
192+
parallelism: Namespace
193+
phase: Pending
194+
reason: WaitingForApproval
195+
```
196+
197+
Now, This recommendation can be approved and operation can be executed immediately by setting `ApprovalStatus` to `Approved` and Setting `approvedWindow` to `Immediate`. You can approve this easily through Appscode UI or edit it manually. Also, You can use kubectl CLI for this -
198+
199+
```bash
200+
$ kubectl patch Recommendation elastic-x-elasticsearch-x-rotate-auth-2juuee \
201+
-n es \
202+
--type merge \
203+
--subresource='status' \
204+
-p '{"status":{"approvalStatus":"Approved","approvedWindow":{"window":"Immediate"}}}'
205+
recommendation.supervisor.appscode.com/elastic-x-elasticsearch-x-rotate-auth-2juuee patched
206+
```
207+
208+
Now, check the status part again. You will find a condition have appeared which says `OpsRequest is successfully created`.
209+
210+
```bash
211+
$ kubectl get recommendation -n es elastic-x-elasticsearch-x-rotate-auth-2juuee -o jsonpath='{.status}' | yq -y
212+
approvalStatus: Approved
213+
approvedWindow:
214+
window: Immediate
215+
conditions:
216+
- lastTransitionTime: '2025-02-25T09:23:29Z'
217+
message: OpsRequest is successfully created
218+
reason: SuccessfullyCreatedOperation
219+
status: 'True'
220+
type: SuccessfullyCreatedOperation
221+
createdOperationRef:
222+
name: elastic-1740475409-rotate-auth-auto
223+
failedAttempt: 0
224+
outdated: false
225+
parallelism: Namespace
226+
phase: InProgress
227+
reason: StartedExecutingOperation
228+
```
229+
230+
You will find an `ElasticsearchOpsRequest` custom resource have been created and, it is rotating the authsecret of `elastic` cluster with negligible downtime. Let's wait for it to reach `Successful` status.
231+
232+
```bash
233+
$ kubectl get elasticsearchopsrequest -n es elastic-1740475409-rotate-auth-auto -w
234+
NAME TYPE STATUS AGE
235+
elastic-1740475409-rotate-auth-auto UpdateVersion Progressing 3m12s
236+
elastic-1740475409-rotate-auth-auto UpdateVersion Progressing 3m34s
237+
.
238+
.
239+
elastic-1740475409-rotate-auth-auto UpdateVersion Successful 11m
240+
```
241+
242+
Let's recheck the recommendation for one last time. We should find that `.status.phase` has been marked as `Succeeded`.
243+
244+
```bash
245+
$ kubectl get recommendation -n es elastic-x-elasticsearch-x-rotate-auth-2juuee
246+
NAME STATUS OUTDATED AGE
247+
elastic-x-elasticsearch-x-rotate-auth-2juuee Succeeded false 78m
248+
```
249+
250+
You may not want to do trigger recommended operations manually. Rather, trigger them autonomously in a preferred schedule when infrastructure is idle or traffic rate is at the lowest. For this purpose, You can create a `MaintenanceWindow` custom resource where you can set your desired schedule/period for triggering these recommended operations automatically. Here's a sample one:
251+
252+
```yaml
253+
apiVersion: supervisor.appscode.com/v1alpha1
254+
kind: MaintenanceWindow
255+
metadata:
256+
name: elastic-maintenance
257+
namespace: es
258+
spec:
259+
timezone: Asia/Dhaka
260+
days:
261+
Wednesday:
262+
- start: 5:40AM
263+
end: 7:00PM
264+
dates:
265+
- start: 2025-01-25T00:00:18Z
266+
end: 2025-01-25T23:41:18Z
267+
```
268+
269+
You can now create a `ApprovalPolicy` custom resource to refer this `MaintenanceWindow` for particular DB type. Following is a sample `ApprovalPolicy` for any `Elasticsearch` custom resource deployed in `es` namespace. This `ApprovalPolicy` custom resource is referring to the `elastic-maintenance` MaintenanceWindow created in the same namespace. You can also create `ClusterMaintenanceWindow` instead which is effective for cluster-wide operations and refer it here. The following ApprovalPolicy will trigger recommended operations when referred maintenance window timeframe is reached.
270+
271+
```yaml
272+
apiVersion: supervisor.appscode.com/v1alpha1
273+
kind: ApprovalPolicy
274+
metadata:
275+
name: es-policy
276+
namespace: es
277+
maintenanceWindowRef:
278+
name: elastic-maintenance
279+
targets:
280+
- group: kubedb.com
281+
kind: Elasticsearch
282+
operations:
283+
- group: ops.kubedb.com
284+
kind: ElasticsearchOpsRequest
285+
```
286+
287+
Lastly, If you want to reject a recommendation, you can just set `ApprovalStatus` to `Rejected` in the recommendation status section. Here's how you can do it using kubectl cli.
288+
289+
```bash
290+
$ kubectl patch Recommendation elastic-x-elasticsearch-x-rotate-auth-2juuee \
291+
-n es \
292+
--type merge \
293+
--subresource='status' \
294+
-p '{"status":{"approvalStatus":"Rejected"}}'
295+
recommendation.supervisor.appscode.com/elastic-x-elasticsearch-x-rotate-auth-2juuee patched
296+
```
92297

93298

94299
## Next Steps

0 commit comments

Comments
 (0)