Skip to content

Commit 2700c97

Browse files
gtoonstragtoonstra
authored
docs: update instructions (#376)
1 parent e585108 commit 2700c97

5 files changed

Lines changed: 490 additions & 8 deletions

File tree

‎docs/cluster-addons.md‎

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
# Cluster Add-ons: metrics-server, Cluster Autoscaler, AWS Load Balancer Controller, Datadog Operator
2+
3+
These are standard Kubernetes cluster components, not part of the Datafold
4+
application itself. The cluster should have them installed and healthy before
5+
(or alongside) the [prerequisites](prerequisites.md) — Datafold's Horizontal
6+
Pod Autoscaling, node capacity, Service/Ingress-provisioned load balancers,
7+
and monitoring all depend on them.
8+
9+
> **Support scope:** metrics-server, Cluster Autoscaler, the AWS Load Balancer
10+
> Controller, and the Datadog Operator are third-party components. Datafold
11+
> provides these instructions as guidance, but support is scoped to the
12+
> Datafold application. For issues with these components themselves, refer to
13+
> their upstream documentation (linked in each section below).
14+
15+
---
16+
17+
## Which of these do I need?
18+
19+
| Component | Cloud scope | Why Datafold needs it |
20+
|-----------|-------------|------------------------|
21+
| [metrics-server](#metrics-server) | Any Kubernetes distribution | Powers `kubectl top` and any CPU/memory-based HPA on Datafold components |
22+
| [Cluster Autoscaler](#cluster-autoscaler) | AWS (EKS) | Adds/removes nodes as Datafold and Temporal workloads scale. GKE and AKS have native node-pool autoscaling instead — see [note](#gcp--azure-note) |
23+
| [AWS Load Balancer Controller](#aws-load-balancer-controller) | AWS (EKS) | Provisions ALB/NLB resources from `Ingress`/`Service` objects (e.g. the Datafold UI ingress). GKE and AKS provision load balancers natively — see [note](#gcp--azure-note) |
24+
| [Datadog Operator](#datadog-operator) | Any Kubernetes distribution | Only needed if this deployment monitors with Datadog (`monitoring.type: datadog`). Installs the operator only — the `DatadogAgent` CR is applied later, automatically, by the Datafold chart |
25+
26+
### GCP / Azure note
27+
28+
- **GKE**: node-pool autoscaling is enabled per pool (`--enable-autoscaling`
29+
on `gcloud container node-pools create`) — no separate chart to install.
30+
Ingress is provisioned natively via GKE's `gce`/`gce-internal` Ingress
31+
classes.
32+
- **AKS**: the cluster autoscaler ships as an AKS add-on, enabled per node
33+
pool (`az aks nodepool update --enable-cluster-autoscaler`). Load balancers
34+
are provisioned via the built-in Azure cloud provider or Application
35+
Gateway Ingress Controller (AGIC).
36+
37+
metrics-server is still required on GKE and AKS the same way as EKS — see
38+
below.
39+
40+
---
41+
42+
## metrics-server
43+
44+
[metrics-server](https://github.com/kubernetes-sigs/metrics-server) collects
45+
resource usage (CPU/memory) from kubelets and exposes it via the
46+
`metrics.k8s.io` API. Required for `kubectl top nodes`/`kubectl top pods` and
47+
for any `HorizontalPodAutoscaler` that scales on CPU/memory.
48+
49+
### Install
50+
51+
```bash
52+
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server
53+
helm repo update
54+
55+
helm install metrics-server metrics-server/metrics-server \
56+
--namespace kube-system
57+
```
58+
59+
No cloud-specific values are required — the chart's defaults work on EKS,
60+
GKE, and AKS alike.
61+
62+
### Verify
63+
64+
```bash
65+
kubectl rollout status deploy/metrics-server -n kube-system
66+
kubectl get apiservice v1beta1.metrics.k8s.io
67+
kubectl top nodes
68+
```
69+
70+
Expected: the `v1beta1.metrics.k8s.io` APIService reports `AVAILABLE: True`,
71+
and `kubectl top nodes` returns CPU/memory figures instead of an error.
72+
73+
---
74+
75+
## Cluster Autoscaler
76+
77+
[Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
78+
adjusts the number of nodes in an EKS managed node group's Auto Scaling Group
79+
based on pending/unschedulable pods.
80+
81+
> **AWS shortcut:** If this cluster is provisioned with Datafold's
82+
> [`terraform-aws-datafold`](https://github.com/datafold/terraform-aws-datafold)
83+
> module, the IAM role and the required Auto Scaling Group discovery tags
84+
> (`k8s.io/cluster-autoscaler/enabled`, `k8s.io/cluster-autoscaler/<CLUSTER_NAME>`)
85+
> are already created for you — see the `cluster_autoscaler_role` module block
86+
> in [`modules/eks/roles.tf`](https://github.com/datafold/terraform-aws-datafold/blob/main/modules/eks/roles.tf).
87+
> Confirm the role exists (`<DEPLOYMENT_NAME>-cluster-autoscaler`), then skip
88+
> straight to [Install](#install-1) — no manual IAM setup is needed.
89+
90+
### IAM / Workload Identity (EKS)
91+
92+
If you are **not** using the `terraform-aws-datafold` module, create an IAM
93+
role with an OIDC trust policy scoped to
94+
`system:serviceaccount:kube-system:cluster-auto-scaler`, granting the standard
95+
[Cluster Autoscaler IAM policy](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#iam-policy)
96+
(`autoscaling:DescribeAutoScalingGroups`, `autoscaling:SetDesiredCapacity`,
97+
`autoscaling:TerminateInstanceInAutoScalingGroup`,
98+
`ec2:DescribeInstanceTypes`, etc., scoped by the
99+
`k8s.io/cluster-autoscaler/<CLUSTER_NAME>` tag). Tag the node group's Auto
100+
Scaling Group with:
101+
102+
- `k8s.io/cluster-autoscaler/enabled: true`
103+
- `k8s.io/cluster-autoscaler/<CLUSTER_NAME>: owned`
104+
105+
### Install
106+
107+
```bash
108+
helm repo add autoscaler https://kubernetes.github.io/autoscaler
109+
helm repo update
110+
111+
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
112+
--namespace kube-system \
113+
--version 9.35.0 \
114+
--set awsRegion=<AWS_REGION> \
115+
--set rbac.create=true \
116+
--set rbac.serviceAccount.name=cluster-auto-scaler \
117+
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<ACCOUNT_ID>:role/<DEPLOYMENT_NAME>-cluster-autoscaler \
118+
--set autoDiscovery.clusterName=<CLUSTER_NAME> \
119+
--set autoDiscovery.enabled=true
120+
```
121+
122+
### Verify
123+
124+
```bash
125+
kubectl rollout status deploy/cluster-autoscaler-aws-cluster-autoscaler -n kube-system
126+
kubectl logs -n kube-system -l "app.kubernetes.io/name=aws-cluster-autoscaler,app.kubernetes.io/instance=cluster-autoscaler" --tail=20
127+
```
128+
129+
Expected: the deployment reports `1/1` ready, and logs show informer caches
130+
populating (`Caches populated for *v1.Node ...`) with no IRSA/auth errors. If
131+
the pod logs `AccessDenied` or similar, double-check the service account
132+
annotation matches the IAM role's trust policy subject exactly
133+
(`system:serviceaccount:kube-system:cluster-auto-scaler`).
134+
135+
---
136+
137+
## AWS Load Balancer Controller
138+
139+
The [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/)
140+
watches `Ingress` and `Service` (type `LoadBalancer`) objects and provisions
141+
matching ALB/NLB resources. Datafold's UI ingress and any `LoadBalancer`-type
142+
Services depend on this controller running.
143+
144+
> **AWS shortcut:** If this cluster is provisioned with Datafold's
145+
> [`terraform-aws-datafold`](https://github.com/datafold/terraform-aws-datafold)
146+
> module, the IAM role and policy (`AWSLoadBalancerControllerIAMPolicy`
147+
> equivalent) already exist — see the `k8s_load_balancer_controller_role`
148+
> module block in
149+
> [`modules/eks/roles.tf`](https://github.com/datafold/terraform-aws-datafold/blob/main/modules/eks/roles.tf).
150+
> Confirm the role exists (`<DEPLOYMENT_NAME>-lb-controller`), then skip
151+
> straight to [creating the service account](#create-the-service-account) —
152+
> do **not** also run `eksctl create iamserviceaccount` or
153+
> `aws iam create-policy` below, that would create a second, untracked IAM
154+
> role/policy alongside the Terraform-managed one.
155+
156+
### IAM / Workload Identity (EKS) — only if not using the Terraform module
157+
158+
Full instructions:
159+
[AWS docs — Install AWS Load Balancer Controller with Helm](https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html).
160+
Summary:
161+
162+
```bash
163+
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.14.1/docs/install/iam_policy.json
164+
165+
aws iam create-policy \
166+
--policy-name AWSLoadBalancerControllerIAMPolicy \
167+
--policy-document file://iam_policy.json
168+
169+
eksctl create iamserviceaccount \
170+
--cluster=<CLUSTER_NAME> \
171+
--namespace=kube-system \
172+
--name=aws-load-balancer-controller \
173+
--attach-policy-arn=arn:aws:iam::<ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
174+
--override-existing-serviceaccounts \
175+
--region <AWS_REGION> \
176+
--approve
177+
```
178+
179+
### Create the service account
180+
181+
If you used the **Terraform shortcut** above, `eksctl` did not create the
182+
service account for you — create it directly, pointing at the
183+
Terraform-managed role:
184+
185+
```yaml
186+
apiVersion: v1
187+
kind: ServiceAccount
188+
metadata:
189+
name: aws-load-balancer-controller
190+
namespace: kube-system
191+
labels:
192+
app.kubernetes.io/name: aws-load-balancer-controller
193+
app.kubernetes.io/component: controller
194+
annotations:
195+
eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<DEPLOYMENT_NAME>-lb-controller
196+
eks.amazonaws.com/sts-regional-endpoints: "true"
197+
```
198+
199+
```bash
200+
kubectl apply -f aws-load-balancer-controller-sa.yaml
201+
```
202+
203+
### Install
204+
205+
```bash
206+
helm repo add eks https://aws.github.io/eks-charts
207+
helm repo update eks
208+
209+
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
210+
-n kube-system \
211+
--set clusterName=<CLUSTER_NAME> \
212+
--set serviceAccount.create=false \
213+
--set serviceAccount.name=aws-load-balancer-controller \
214+
--version 1.14.0
215+
```
216+
217+
`serviceAccount.create=false` is required in both paths (Terraform shortcut
218+
or manual `eksctl`) — the service account must already exist with the IRSA
219+
annotation before the chart installs, otherwise Helm creates a second,
220+
un-annotated one.
221+
222+
### Verify
223+
224+
```bash
225+
kubectl rollout status deploy/aws-load-balancer-controller -n kube-system
226+
kubectl get deployment -n kube-system aws-load-balancer-controller
227+
kubectl logs -n kube-system deploy/aws-load-balancer-controller --tail=30
228+
```
229+
230+
Expected: `2/2` ready, leader election acquired
231+
(`successfully acquired lease kube-system/aws-load-balancer-controller-leader`),
232+
and controllers for `ingress`, `service`, and `targetGroupBinding` all
233+
starting cleanly with no auth errors.
234+
235+
The deployed chart does not receive security updates automatically — check
236+
the [release page](https://github.com/aws/eks-charts/releases) periodically
237+
and re-run `helm upgrade` with a newer `--version` when needed.
238+
239+
---
240+
241+
## Datadog Operator
242+
243+
The [Datadog Operator](https://docs.datadoghq.com/containers/kubernetes/installation/?tab=operator)
244+
watches for `DatadogAgent` custom resources and reconciles the node
245+
Agent/Cluster Agent DaemonSet+Deployment from them. Only install this if the
246+
deployment monitors with Datadog (`DatafoldApplication.spec.monitoring.type:
247+
datadog`).
248+
249+
> **This step only installs the operator itself.** Unlike the other add-ons on
250+
> this page, you do **not** hand-write a `DatadogAgent` CR here. The Datafold
251+
> chart ships its own `datadog` subchart
252+
> (`charts/datafold/charts/datadog/templates/datadog_operator.yaml`) that
253+
> renders the `DatadogAgent` CR automatically from
254+
> `DatafoldApplication.spec.monitoring.datadog.*` (APM, NPM, log collection,
255+
> `monitorPostgres`, `monitorKeda`, `monitorTemporal`, etc.) once the Datafold
256+
> application is deployed — see [Deploy with Operator](deploy-operator.md).
257+
> There is nothing to apply manually at this stage beyond the operator and the
258+
> API/App key secret below.
259+
260+
### Install
261+
262+
```bash
263+
helm repo add datadog https://helm.datadoghq.com
264+
helm repo update
265+
```
266+
267+
Datadog's own quickstart installs into the default namespace with a
268+
standalone `datadog-secret` — for Datafold deployments, install the operator
269+
into the **same namespace as the Datafold deployment** instead, since the
270+
`DatadogAgent` CR created later by the `datadog` subchart lives there too:
271+
272+
```bash
273+
helm install datadog-operator datadog/datadog-operator \
274+
--namespace <DATAFOLD_NAMESPACE>
275+
```
276+
277+
### API / App key secret
278+
279+
The `DatadogAgent` CR the `datadog` subchart renders later reads its
280+
credentials from `DATAFOLD_DD_API_KEY` / `DATAFOLD_DD_APP_KEY` keys on the
281+
Datafold application's own secret (the same Secret referenced by
282+
`DatafoldApplication.spec.monitoring.monitoringApiKey`), not a separate
283+
`datadog-secret`. Populate those keys before enabling the `datadog` component
284+
— see [Deploy with Operator](deploy-operator.md) for how application secrets
285+
are provisioned.
286+
287+
### Verify
288+
289+
```bash
290+
kubectl rollout status deploy/datadog-operator -n <DATAFOLD_NAMESPACE>
291+
```
292+
293+
Expected: `1/1` ready. At this point there is intentionally **no**
294+
`DatadogAgent` object yet:
295+
296+
```bash
297+
kubectl get datadogagent -n <DATAFOLD_NAMESPACE>
298+
```
299+
300+
The `DatadogAgent` (and the resulting node Agent DaemonSet + Cluster Agent
301+
Deployment) appears only after the Datafold application is deployed with
302+
`monitoring.type: datadog` — see
303+
[Deploy with Operator](deploy-operator.md). If it doesn't appear, check the
304+
Datadog Operator's own logs first:
305+
306+
```bash
307+
kubectl logs -n <DATAFOLD_NAMESPACE> deploy/datadog-operator --tail=50
308+
```
309+
310+
For the full set of configurable `DatadogAgent` fields (beyond what the
311+
`datadog` subchart already exposes), see the
312+
[Datadog Operator configuration reference](https://github.com/DataDog/datadog-operator/blob/main/docs/configuration.v2alpha1.md).
313+
Some features (e.g. multi-line log aggregation) are configured via pod
314+
annotations instead — see the
315+
[advanced log collection docs](https://docs.datadoghq.com/agent/logs/advanced_log_collection/?tab=kubernetes#multi-line-aggregation).
316+
317+
---
318+
319+
## Placeholder Reference
320+
321+
| Placeholder | Description | Example |
322+
|-------------|-------------|---------|
323+
| `<DEPLOYMENT_NAME>` | Your Datafold deployment name | `acme`, `production` |
324+
| `<CLUSTER_NAME>` | EKS cluster name | `acme-datafold` |
325+
| `<ACCOUNT_ID>` | AWS account ID | `123456789012` |
326+
| `<AWS_REGION>` | AWS region | `us-east-2` |
327+
| `<DATAFOLD_NAMESPACE>` | Kubernetes namespace the Datafold deployment runs in | `acme-datafold` |
328+
329+
---
330+
331+
## Next Step
332+
333+
Continue with the [Prerequisites overview](prerequisites.md) for PostgreSQL,
334+
Temporal, and KEDA, then [deploy the Datafold application](deploy-operator.md).

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ used. It requires extra care to get values correct.
5252
## Reference
5353

5454
- [Prerequisites overview](prerequisites.md)
55+
- [Cluster add-ons: metrics-server, Cluster Autoscaler, AWS Load Balancer Controller, Datadog Operator](cluster-addons.md)
5556
- [Temporal hosting: self-hosted vs Temporal Cloud](temporal-hosting.md)
5657
- [KEDA worker autoscaling](keda.md)
5758
- [Temporal Cloud payload encryption](temporal-cloud-encryption.md)

‎docs/postgres-zalando.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,27 @@ installation, IAM for backups, and custom resources.
2626

2727
---
2828

29+
> **AWS shortcut:** If this cluster is provisioned with Datafold's
30+
> [`terraform-aws-datafold`](https://github.com/datafold/terraform-aws-datafold)
31+
> module, Steps A1 and A2 below are already handled for you — the module
32+
> creates the Temporal backup S3 bucket and the `postgres-pod` IAM role (via
33+
> IRSA) whenever `deploy_temporal` is `true` (the default). See the
34+
> [`temporal_backup` module block in `main.tf`](https://github.com/datafold/terraform-aws-datafold/blob/main/main.tf#L375-L384)
35+
> for the bucket and the
36+
> [`# temporal` section of `modules/eks/roles.tf`](https://github.com/datafold/terraform-aws-datafold/blob/main/modules/eks/roles.tf#L393-L443)
37+
> for the IAM role and policy. Confirm `deploy_temporal` hasn't been disabled
38+
> for this deployment, then skip ahead to
39+
> [Step A3](#step-a3-install-zalando-postgres-operator).
40+
2941
## Step A1: Create Backup Storage
3042

3143
Create an object storage bucket for PostgreSQL logical backups. Enable
3244
server-side encryption and set a lifecycle policy to expire old backups
3345
(7 days recommended).
3446

47+
> **AWS:** skip this if you're using the `terraform-aws-datafold` module —
48+
> see the [AWS shortcut](#deployment-order) above.
49+
3550
| Cloud | Service | Example bucket name |
3651
|-------|---------|---------------------|
3752
| AWS | S3 | `<DEPLOYMENT_NAME>-postgres-backups` |
@@ -47,6 +62,9 @@ write to the backup bucket.
4762

4863
### AWS (EKS)
4964

65+
> Skip this if you're using the `terraform-aws-datafold` module — see the
66+
> [AWS shortcut](#deployment-order) above.
67+
5068
Create an IAM role with an OIDC trust policy for the EKS cluster. The role must
5169
grant the following S3 actions, restricted to the backup bucket ARN:
5270

@@ -309,7 +327,9 @@ server:
309327
enableHostVerification: false
310328
```
311329

312-
Continue with [Temporal install Step 3](temporal-install.md#step-3-install-temporal).
330+
Continue with [Optional: Datadog Metrics Collection](temporal-install.md#optional-datadog-metrics-collection)
331+
if this cluster monitors with Datadog, otherwise skip straight to
332+
[Temporal install Step 3](temporal-install.md#step-3-install-temporal).
313333

314334
---
315335

0 commit comments

Comments
 (0)