Merge pull request #1314 from rashmichandrashekar/rashmi/hpa

Add HPA docs

articles/azure-monitor/containers/prometheus-metrics-scrape-autoscaling.md

@@ -0,0 +1,58 @@
+---
+title: Autoscaling Support for Addon Pods
+description: Documentation regarding addon pod autoscaling support for Azure Managed Prometheus.
+ms.topic: conceptual
+ms.date: 2/28/2024
+ms.reviewer: rashmy
+---
+
+# Horizontal Pod Autoscaling (HPA) for Collector Replica set
+
+### Overview
+Azure Managed Prometheus supports Horizontal Pod Autoscaling(HPA) for the ama-metrics replica set pod by default. 
+The HPA allows the ama-metrics replica set pod, which scrapes Prometheus metrics with custom jobs, to scale automatically based on memory utilization to prevent OOMKills. By default, the HPA is configured with a minimum of two replicas and a maximum of 12 replicas. Users can configure the number of shards within the range of 2 to 12 replicas.
+
+[Kubernetes support for HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) 
+
+### Update Min and Max shards
+The HPA object named **ama-metrics-hpa** in the kube-system namespace can be edited to update the min and max shards/replica set instances.
+If the changes aren't within the supported range of 2 to 12 they are ineffective and fall back to the last known good.
+
+**Update Min replicas**
+```bash
+kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 4}}'
+```
+
+**Update Max replicas**
+```bash
+kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"maxReplicas": 10}}'
+```
+
+**Update Min and Max replicas**
+```bash
+kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 3, "maxReplicas": 10}}'
+```
+
+**or**
+
+The min and max replicas can also be edited by doing a **kubectl edit** and updating the spec in the editor
+```bash
+kubectl edit hpa ama-metrics-hpa -n kube-system
+```
+
+### Update min and max shards to disable HPA scaling
+If the default HPA settings don't meet the customer's requirements, they can configure the minimum and maximum number of shards to be the same.
+This prevents the HPA from scaling the replicas based on varying loads, ensuring a consistent number of replicas.
+
+Ex - If the customer wants to set the shards to 8 and not have the HPA update the shards, update the min and max shards to 8.
+
+**Update Min and Max replicas**
+```bash
+kubectl patch hpa ama-metrics-hpa -n kube-system --type merge --patch '{"spec": {"minReplicas": 8, "maxReplicas": 8}}'
+```
+
+*A kubectl edit on the ama-metrics-hpa spec gives more information about the scale up and scale down configurations used for HPA*
+
+## Next steps
+
+* [Troubleshoot issues with Prometheus data collection](prometheus-metrics-troubleshoot.md).

articles/azure-monitor/containers/prometheus-metrics-scrape-scale.md

@@ -14,9 +14,9 @@ This article provides guidance on performance that can be expected when collecti
 
 The CPU and memory usage is correlated with the number of bytes of each sample and the number of samples scraped. These benchmarks are based on the [default targets scraped](prometheus-metrics-scrape-default.md), volume of custom metrics scraped, and number of nodes, pods, and containers. These numbers are meant as a reference since usage can still vary significantly depending on the number of time series and bytes per metric.
 
-The upper volume limit per pod is currently about 3-3.5 million samples per minute, depending on the number of bytes per sample. This limitation is addressed when sharding is added in future.
+The upper volume limit per pod is currently about 3-3.5 million samples per minute, depending on the number of bytes per sample. 
 
-The agent consists of a deployment with one replica and DaemonSet for scraping metrics. The DaemonSet scrapes any node-level targets such as cAdvisor, kubelet, and node exporter. You can also configure it to scrape any custom targets at the node level with static configs. The replica set scrapes everything else such as kube-state-metrics or custom scrape jobs that utilize service discovery.
+The agent consists of a deployment with two replicas by default (which will be automatically configured by HPA based on memory utilization) and DaemonSet for scraping metrics. The DaemonSet scrapes any node-level targets such as cAdvisor, kubelet, and node exporter. You can also configure it to scrape any custom targets at the node level with static configs. The replica set scrapes everything else such as kube-state-metrics or custom scrape jobs that utilize service discovery.
 
 ## Comparison between small and large cluster for replica
 
@@ -38,7 +38,7 @@ For more custom metrics, the single pod behaves the same as the replica pod depe
 
 ## Schedule ama-metrics replica pod on a node pool with more resources 
 
-A large volume of metrics per pod needs a node with enough CPU and memory. If the *ama-metrics* replica pod isn't scheduled on a node or node pool with enough resources, it might get OOMKilled and go into CrashLoopBackoff. To fix this, you can add the label `azuremonitor/metrics.replica.preferred=true` to a node or node pool on your cluster with higher resources (in [system node pool](/azure/aks/use-system-pools#system-and-user-node-pools)). This ensures the replica pod gets scheduled on that node. You can also create extra system pools with larger nodes and add the same label. It's better to label node pools rather than individual nodes so new nodes in the pool can also be used for scheduling.
+A large volume of metrics per pod needs a node with enough CPU and memory. If the *ama-metrics* replica pods aren't scheduled on nodes or node pools with enough resources, they might get OOMKilled and go into CrashLoopBackoff. To fix this, you can add the label `azuremonitor/metrics.replica.preferred=true` to nodes or node pools on your cluster with higher resources (in [system node pool](/azure/aks/use-system-pools#system-and-user-node-pools)). This ensures the replica pods get scheduled on those nodes. You can also create extra system pools with larger nodes and add the same label. It's better to label node pools rather than individual nodes so new nodes in the pool can also be used for scheduling.
 
 ```
 kubectl label nodes <node-name> azuremonitor/metrics.replica.preferred="true"

articles/azure-monitor/containers/prometheus-metrics-troubleshoot.md

@@ -197,6 +197,8 @@ If creation of Azure Monitor Workspace fails with an error saying "*Resource 're
 
 When you create an Azure Monitor workspace, by default a data collection rule and a data collection endpoint in the form "*azure-monitor-workspace-name*" will automatically be created in a resource group in the form "*MA_azure-monitor-workspace-name_location_managed*". Currently there's no way to change the names of these resources, and you'll need to set an exemption on the Azure Policy to exempt the above resources from policy evaluation. See [Azure Policy exemption structure](/azure/governance/policy/concepts/exemption-structure).
 
-## Next steps
+## High Scale considerations
 
+If you are collecting metrics at high scale, check the sections below for HPA and high scale guidance.
 - [Check considerations for collecting metrics at high scale](prometheus-metrics-scrape-scale.md).
+- [Horizontal Pod Autoscaling for collector replicaset](prometheus-metrics-scrape-autoscaling.md)

articles/azure-monitor/essentials/prometheus-metrics-overview.md

@@ -29,6 +29,10 @@ Azure Monitor managed service for Prometheus provides preconfigured alerts, rule
 
 Pricing is based on ingestion and query with no additional storage cost. For more information, see the **Metrics** tab in [Azure Monitor pricing](https://azure.microsoft.com/pricing/details/monitor/).
 
+> [!NOTE]
+> Azure Managed Prometheus supports Horizontal Pod Autoscaling for replicaset pods in AKS Kubernetes clusters.
+See [Autoscaling](../containers/prometheus-metrics-scrape-autoscaling.md) to learn more.
+
 ### Enable Azure Monitor managed service for Prometheus
 
 Azure Monitor managed service for Prometheus collects data from AKS and Azure Arc-enabled Kubernetes.

articles/azure-monitor/toc.yml

@@ -492,6 +492,9 @@ items:
         - name: High scale
           displayName: Prometheus
           href: containers/prometheus-metrics-scrape-scale.md
+        - name: Autoscale support
+          displayName: Prometheus
+          href: containers/prometheus-metrics-scrape-autoscaling.md
       - name: Integrations
         items:
         - name: Set up exporters for common workloads