Skip to content

Commit dbaf51a

Browse files
authored
Merge pull request #1314 from rashmichandrashekar/rashmi/hpa
Add HPA docs
2 parents 55fc922 + ca3b703 commit dbaf51a

File tree

5 files changed

+71
-4
lines changed

5 files changed

+71
-4
lines changed
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
title: Autoscaling Support for Addon Pods
3+
description: Documentation regarding addon pod autoscaling support for Azure Managed Prometheus.
4+
ms.topic: conceptual
5+
ms.date: 2/28/2024
6+
ms.reviewer: rashmy
7+
---
8+
9+
# Horizontal Pod Autoscaling (HPA) for Collector Replica set
10+
11+
### Overview
12+
Azure Managed Prometheus supports Horizontal Pod Autoscaling(HPA) for the ama-metrics replica set pod by default.
13+
The HPA allows the ama-metrics replica set pod, which scrapes Prometheus metrics with custom jobs, to scale automatically based on memory utilization to prevent OOMKills. By default, the HPA is configured with a minimum of two replicas and a maximum of 12 replicas. Users can configure the number of shards within the range of 2 to 12 replicas.
14+
15+
[Kubernetes support for HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
16+
17+
### Update Min and Max shards
18+
The HPA object named **ama-metrics-hpa** in the kube-system namespace can be edited to update the min and max shards/replica set instances.
19+
If the changes aren't within the supported range of 2 to 12 they are ineffective and fall back to the last known good.
20+
21+
**Update Min replicas**
22+
```bash
23+
kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 4}}'
24+
```
25+
26+
**Update Max replicas**
27+
```bash
28+
kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"maxReplicas": 10}}'
29+
```
30+
31+
**Update Min and Max replicas**
32+
```bash
33+
kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 3, "maxReplicas": 10}}'
34+
```
35+
36+
**or**
37+
38+
The min and max replicas can also be edited by doing a **kubectl edit** and updating the spec in the editor
39+
```bash
40+
kubectl edit hpa ama-metrics-hpa -n kube-system
41+
```
42+
43+
### Update min and max shards to disable HPA scaling
44+
If the default HPA settings don't meet the customer's requirements, they can configure the minimum and maximum number of shards to be the same.
45+
This prevents the HPA from scaling the replicas based on varying loads, ensuring a consistent number of replicas.
46+
47+
Ex - If the customer wants to set the shards to 8 and not have the HPA update the shards, update the min and max shards to 8.
48+
49+
**Update Min and Max replicas**
50+
```bash
51+
kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 8, "maxReplicas": 8}}'
52+
```
53+
54+
*A kubectl edit on the ama-metrics-hpa spec gives more information about the scale up and scale down configurations used for HPA*
55+
56+
## Next steps
57+
58+
* [Troubleshoot issues with Prometheus data collection](prometheus-metrics-troubleshoot.md).

articles/azure-monitor/containers/prometheus-metrics-scrape-scale.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,9 @@ This article provides guidance on performance that can be expected when collecti
1414

1515
The CPU and memory usage is correlated with the number of bytes of each sample and the number of samples scraped. These benchmarks are based on the [default targets scraped](prometheus-metrics-scrape-default.md), volume of custom metrics scraped, and number of nodes, pods, and containers. These numbers are meant as a reference since usage can still vary significantly depending on the number of time series and bytes per metric.
1616

17-
The upper volume limit per pod is currently about 3-3.5 million samples per minute, depending on the number of bytes per sample. This limitation is addressed when sharding is added in future.
17+
The upper volume limit per pod is currently about 3-3.5 million samples per minute, depending on the number of bytes per sample.
1818

19-
The agent consists of a deployment with one replica and DaemonSet for scraping metrics. The DaemonSet scrapes any node-level targets such as cAdvisor, kubelet, and node exporter. You can also configure it to scrape any custom targets at the node level with static configs. The replica set scrapes everything else such as kube-state-metrics or custom scrape jobs that utilize service discovery.
19+
The agent consists of a deployment with two replicas by default (which will be automatically configured by HPA based on memory utilization) and DaemonSet for scraping metrics. The DaemonSet scrapes any node-level targets such as cAdvisor, kubelet, and node exporter. You can also configure it to scrape any custom targets at the node level with static configs. The replica set scrapes everything else such as kube-state-metrics or custom scrape jobs that utilize service discovery.
2020

2121
## Comparison between small and large cluster for replica
2222

@@ -38,7 +38,7 @@ For more custom metrics, the single pod behaves the same as the replica pod depe
3838

3939
## Schedule ama-metrics replica pod on a node pool with more resources
4040

41-
A large volume of metrics per pod needs a node with enough CPU and memory. If the *ama-metrics* replica pod isn't scheduled on a node or node pool with enough resources, it might get OOMKilled and go into CrashLoopBackoff. To fix this, you can add the label `azuremonitor/metrics.replica.preferred=true` to a node or node pool on your cluster with higher resources (in [system node pool](/azure/aks/use-system-pools#system-and-user-node-pools)). This ensures the replica pod gets scheduled on that node. You can also create extra system pools with larger nodes and add the same label. It's better to label node pools rather than individual nodes so new nodes in the pool can also be used for scheduling.
41+
A large volume of metrics per pod needs a node with enough CPU and memory. If the *ama-metrics* replica pods aren't scheduled on nodes or node pools with enough resources, they might get OOMKilled and go into CrashLoopBackoff. To fix this, you can add the label `azuremonitor/metrics.replica.preferred=true` to nodes or node pools on your cluster with higher resources (in [system node pool](/azure/aks/use-system-pools#system-and-user-node-pools)). This ensures the replica pods get scheduled on those nodes. You can also create extra system pools with larger nodes and add the same label. It's better to label node pools rather than individual nodes so new nodes in the pool can also be used for scheduling.
4242

4343
```
4444
kubectl label nodes <node-name> azuremonitor/metrics.replica.preferred="true"

articles/azure-monitor/containers/prometheus-metrics-troubleshoot.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,8 @@ If creation of Azure Monitor Workspace fails with an error saying "*Resource 're
197197
198198
When you create an Azure Monitor workspace, by default a data collection rule and a data collection endpoint in the form "*azure-monitor-workspace-name*" will automatically be created in a resource group in the form "*MA_azure-monitor-workspace-name_location_managed*". Currently there's no way to change the names of these resources, and you'll need to set an exemption on the Azure Policy to exempt the above resources from policy evaluation. See [Azure Policy exemption structure](/azure/governance/policy/concepts/exemption-structure).
199199
200-
## Next steps
200+
## High Scale considerations
201201
202+
If you are collecting metrics at high scale, check the sections below for HPA and high scale guidance.
202203
- [Check considerations for collecting metrics at high scale](prometheus-metrics-scrape-scale.md).
204+
- [Horizontal Pod Autoscaling for collector replicaset](prometheus-metrics-scrape-autoscaling.md)

articles/azure-monitor/essentials/prometheus-metrics-overview.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ Azure Monitor managed service for Prometheus provides preconfigured alerts, rule
2929

3030
Pricing is based on ingestion and query with no additional storage cost. For more information, see the **Metrics** tab in [Azure Monitor pricing](https://azure.microsoft.com/pricing/details/monitor/).
3131

32+
> [!NOTE]
33+
> Azure Managed Prometheus supports Horizontal Pod Autoscaling for replicaset pods in AKS Kubernetes clusters.
34+
See [Autoscaling](../containers/prometheus-metrics-scrape-autoscaling.md) to learn more.
35+
3236
### Enable Azure Monitor managed service for Prometheus
3337

3438
Azure Monitor managed service for Prometheus collects data from AKS and Azure Arc-enabled Kubernetes.

articles/azure-monitor/toc.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -492,6 +492,9 @@ items:
492492
- name: High scale
493493
displayName: Prometheus
494494
href: containers/prometheus-metrics-scrape-scale.md
495+
- name: Autoscale support
496+
displayName: Prometheus
497+
href: containers/prometheus-metrics-scrape-autoscaling.md
495498
- name: Integrations
496499
items:
497500
- name: Set up exporters for common workloads

0 commit comments

Comments
 (0)