Merge pull request #3575 from MicrosoftDocs/main638787097259164706sync_temp

For protected branch, push strategy should use PR and merge to target branch method to work around git push error

Author: learn-build-service-prod[bot]

AKS-Arc/TOC.yml

@@ -133,6 +133,8 @@
             href: kubernetes-monitor-object-events.md
           - name: Get kubelet logs
             href: aks-get-kubelet-logs.md
+          - name: Monitor control plane metrics
+            href: control-plane-metrics.md
           - name: Enable Container Insights
             href: /azure/azure-monitor/containers/kubernetes-monitoring-enable
           - name: Monitor Kubernetes audit events
@@ -331,6 +333,8 @@
         href: vnet.yml
 - name: AKS on Windows Server
   items:
+  - name: AKS on Windows Server retirement
+    href: aks-windows-server-retirement.md
   - name: Overview
     href: overview.md
   - name: System requirements

AKS-Arc/aks-windows-server-retirement.md

@@ -0,0 +1,92 @@
+---
+title: Retirement of AKS architecture on Windows Server 2019 and Windows Server 2022
+description: Learn about retirement of AKS on Windows Server 2019 and Windows Server 2022.
+ms.topic: how-to
+ms.custom: linux-related-content
+author: sethmanheim
+ms.author: sethm
+ms.date: 03/26/2025
+
+# Intent: As an IT Pro, I want to move my workloads from AKS on Windows Server to the latest version of AKS on Azure Local.
+# Keyword: retirement
+---
+
+# Announcing the 3-year retirement of AKS on Windows Server current architecture
+
+AKS enabled by Azure Arc uses Azure Arc to create new Kubernetes clusters on Azure Local directly from Azure. It enables you to use familiar tools such as the Azure portal, Azure CLI, Azure Resource Manager, and Bicep and Terraform templates to create and manage your Kubernetes clusters running on Azure Local. Microsoft continues to focus on delivering consistent user experience for all your AKS clusters. To continue ensuring Azure remains the best possible experience with the highest standards of safety and reliability, **we are retiring the current architecture of AKS on Windows Server 2019 and AKS on Windows Server 2022 in 3 years, on March 27, 2028**.
+
+## What is AKS on Azure Local?
+
+AKS on Azure Local uses Azure Arc to create new Kubernetes clusters on Azure Local directly from Azure. Since clusters are automatically connected to Azure Arc when they're created, you can use your Microsoft Entra ID for connecting to your clusters from anywhere. This ensures your developers and application operators can provision and configure Kubernetes clusters in accordance with company policies.
+
+The following Kubernetes cluster deployment and management capabilities are available:
+
+- **Pricing**: AKS is now included in Azure Local pricing, effective January 2025. This means that you only need to pay for Azure Local. There are no separate costs for running AKS clusters, including Linux and Windows node pools.
+- **Simplified infrastructure deployment on Azure Local**. Infrastructure components of AKS Arc like Arc Resource Bridge, Custom Location and the Kubernetes Extension for the AKS Arc operator, are all deployed as part of the Azure Local. The whole lifecycle management of AKS Arc infrastructure follows the same approach as the other components on Azure Local.
+- **Cloud-based management**: Create and manage Kubernetes clusters on Azure Local with familiar tools such as the Azure portal, Azure CLI, Azure Resource Manager, and Bicep and Terraform templates.
+- **Arc Gateway integration**: Deploy AKS Arc clusters with pod-level Arc Proxy and communicate with the Arc gateway, reducing the list of outbound URLs to configure in an isolated network environment.
+- **Integration with Entra ID and Azure RBAC**: Enable Azure RBAC for Kubernetes while creating AKS Arc clusters. Deploy AKS Arc clusters with workload identity enabled and deploy application pods with the workload identity label to access Microsoft Entra ID protected resources, such as Azure Key Vault.
+- **Support for NVIDIA T4**: Create Linux node pools in new VM sizes with GPU NVIDIA T4.
+- **K8s Audit Logs**: Export audit logs and other control plane logs to one or more destinations.
+- **Improved certificate management**: Shut down AKS Arc clusters for up to 7 days without any certificate expiration issues. Automatically repair certificates, managed by cert-tattoo, that expired when the cluster was shut down.
+
+## If you're using Azure Kubernetes Service on Windows Server 2019 or Windows Server 2022
+
+The Azure Kubernetes Service current architecture on Windows Server 2019 and Windows Server 2022 will be retired on 27 March 2028. Starting on March 27 2028, you no longer get support, security and quality updates for your existing Azure Kubernetes Service clusters. Additionally, you won't be able to deploy, upgrade or scale the current architecture of Azure Kubernetes Service on Windows Server 2019 and Windows Server 2022.
+
+## If you're using Azure Kubernetes Service on Azure Local, version 22H2
+
+If you're using AKS on Azure Local, version 22H2, be aware that Azure Local, version 22H2 will reach end of service on May 31 2025. After that, you won't receive monthly security and quality updates that provide protection from the latest security threats. To continue receiving updates, we recommend updating to the latest version of Azure Local.
+
+## Deploy AKS on Azure Local, version 23H2 or later
+
+### [From Windows Server 2019, 2022](#tab/ws)
+
+AKS on Azure Local has a dependency on deploying a supported version of Azure Local. This means that local, PowerShell, or Windows Admin Center commands such as `Update-AksHciCluster` that worked on Windows Server don't work on Azure Local, version 23H2 or later, since AKS deployments on Azure Local, version 23H2 or later are managed via Azure Resource Manager (Azure CLI, Azure portal, etc.).
+
+#### Evaluate if Azure Local is right for you
+
+[Compare Windows Server](/azure/azure-local/concepts/compare-windows-server) explains key differences between Azure Local and Windows Server and provides guidance on when to use each. Both products are actively supported and maintained by Microsoft. Many organizations choose to deploy both, as they are intended for different and complementary purposes.
+
+#### Uninstall AKS on Windows Server
+
+Before you move to Azure Local, follow these steps to disconnect AKS workload clusters from Azure Arc and then uninstall AKS:
+
+- Identify all your Arc-connected AKS workload clusters, and then [disconnect your AKS workload clusters from Azure Arc](connect-to-arc.md#disconnect-your-aks-cluster-from-azure-arc)
+- Uninstall AKS using [`Uninstall-AksHci`](/azure/aks/aksarc/reference/ps/uninstall-akshci). This removes all AKS-related configuration from Windows Server.
+
+#### Deploy a supported version of Azure Local
+
+[Deploy Azure Local from Azure portal or an Azure Resource Manager template](/azure/azure-local/deploy/deployment-introduction).
+
+#### Deploy an AKS cluster on Azure Local
+
+- [Review the networking pre-requisites](aks-hci-network-system-requirements.md) for deploying AKS on Azure Local.
+- [Deploy an AKS cluster on Azure Local using Az CLI, Azure portal and ARM templates, etc.](aks-create-clusters-cli.md).
+
+### [From Azure Local, version 22H2](#tab/22H2)
+
+AKS on Azure Local has a dependency on deploying a supported version of Azure Local. This means that local, PowerShell, or Windows Admin Center commands such as `Update-AksHciCluster` that worked on Azure Local, version 22H2 don't work on Azure Local, version 23H2 or later, since AKS deployments on Azure Local, version 23H2 or later are managed via Azure Resource Manager (Az CLI, Azure portal, etc).
+
+#### Uninstall AKS on Azure Local, version 22H2
+
+Before you upgrade to a supported version of Azure Local, follow these steps to disconnect AKS workload clusters from Azure Arc and then uninstall AKS:
+
+- Identify all your Arc-connected AKS workload clusters, and then [disconnect your AKS workload clusters from Azure Arc](connect-to-arc.md#disconnect-your-aks-cluster-from-azure-arc)
+- Uninstall AKS using [`Uninstall-AksHci`](/azure/aks/aksarc/reference/ps/uninstall-akshci). This removes all AKS-related configuration from Windows Server.
+
+#### Upgrade to a supported version of Azure Local
+
+[Upgrade Azure Local 22H2 to a supported version of Azure Local](/azure/azure-local/upgrade/about-upgrades-23h2).
+
+#### Deploy an AKS cluster on Azure Local
+
+- [Review the networking pre-requisites](aks-hci-network-system-requirements.md) for deploying AKS on Azure Local.
+- [Deploy an AKS cluster on Azure Local using Az CLI, Azure portal and ARM templates, etc.](aks-create-clusters-cli.md).
+
+---
+
+## Next steps
+
+- [Compare AKS deployment options](https://techcommunity.microsoft.com/blog/azurearcblog/comparing-feature-sets-for-aks-enabled-by-azure-arc-deployment-options/4188163).
+- [Compare Windows Server with Azure Local](/azure/azure-local/concepts/compare-windows-server)

AKS-Arc/control-plane-metrics.md

@@ -0,0 +1,146 @@
+---
+title: Monitor control plane metrics
+description: Learn how to enable and query control plane metrics from AKS on Azure Local 23H2.
+ms.date: 03/26/2025
+ms.topic: how-to
+author: sethmanheim
+ms.author: sethm
+ms.reviewer: haojiehang
+
+---
+
+# Monitor control plane metrics
+
+[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)]
+
+This article describes how to enable and query control plane metrics from AKS on Azure Local 23H2. The workflow is as follows:
+
+- Enable Managed Prometheus extension
+- Enable control plane metrics
+- (Optional) view metrics in Grafana
+
+Control plane metrics provide critical visibility into the availability and performance of Kubernetes control plane components, such as the API server, scheduler, or controller manager. You can use these metrics to maximize observability and maintain operational excellence for your cluster.
+
+## Prerequisites
+
+Before you begin, make sure the following prerequisites are met:
+
+- A running AKS on Azure Local instance.
+- Install the latest version of the **aksarc**, **connectedk8s**, and **k8s-extension** CLI extensions.
+- [Download and install **kubectl**](https://kubernetes.io/docs/tasks/tools/) on your development machine.  
+- [Understand the basics of Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/examples/).
+- [Understand the basics of Kubernetes system component metrics](https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/).
+
+## Enable Managed Prometheus extension
+
+Azure Monitor collects and aggregates important metrics from your AKS Arc running on Azure Local instance. In addition to [platform metrics](kubernetes-monitor-metrics.md#metrics) collected from your cluster, you can view granular Kubernetes metrics using the managed Prometheus extension. This extension collects Prometheus metrics from your deployment and stores them in an Azure Monitor workspace in Azure. Once they are ingested, you can analyze them in Metrics Explorer or use prebuilt dashboards in Azure Managed Grafana.
+
+### Step 1: Install the managed Prometheus extension
+
+You can install the extension either from the Azure portal, or using CLI.
+
+# [Azure portal](#tab/azureportal)
+
+Go to your Kubernetes instance, then select **Monitoring > Insights > Monitor Settings**:
+
+:::image type="content" source="media/control-plane-metrics/monitor-settings.png" alt-text="Screenshot of portal showing monitor settings." lightbox="media/control-plane-metrics/monitor-settings.png":::
+
+#### [Azure CLI](#tab/azurecli)
+
+The following command installs the managed Prometheus extension with a default Azure Monitor workspace:
+
+```azurecli
+az k8s-extension create --name azuremonitor-metrics --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers.Metrics
+```
+
+---
+
+See the [guidance for the managed Prometheus extension onboarding](/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#enable-prometheus-and-grafana).
+
+### Step 2: Verify extension and metrics pod deployment
+
+To verify the extension installation, you can run `az connectedk8s proxy` to connect to the cluster and use **kubectl** to list the metrics pods. The pods should start with the name **ama-metrics-** and are in a running state.
+
+```azurecli
+kubectl get pods -n kube-system
+```
+
+The output of the command is similar to the following:
+
+```output
+NAME READY STATUS RESTARTS AGE
+akshci-telemetry-5df56fd5-s5wtm 1/1 Running 1 (37h ago) 44d
+ama-logs-nqf9h 3/3 Running 0 5h29m
+ama-logs-pvvb2 3/3 Running 2 (5h21m ago) 5h29m
+ama-logs-rs-86bc9dd898-4p7pv 2/2 Running 0 5h29m
+**ama-metrics-98bb54876-dndrh** 2/2 Running 2 (3h33m ago) 5h30m
+**ama-metrics-ksm-6544c98f5f-ph6sp** 1/1 Running 0 5h30m
+**ama-metrics-node-6dl7p** 2/2 Running 2 (3h33m ago) 5h30m
+**ama-metrics-node-ztwzt** 2/2 Running 1 (3h33m ago) 5h30m
+….
+```
+
+## Enable control plane metrics with custom configuration
+
+After you enable the extension, you can view Prometheus Metrics from [targets scraped by default](/azure/azure-monitor/containers/prometheus-metrics-scrape-default#targets-scraped-by-default) in the Azure Monitor workspace. The [default ON targets](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration-minimal#minimal-ingestion-for-default-on-targets) include kubelet, kube-state-metrics, node-exporter, etc. To get started with kubelet metrics, use the PromQL below:
+
+```bash
+kubelet_running_pods{cluster="<cluster_name>", instance="<instance_name>", job="kubelet"}
+```
+
+:::image type="content" source="media/control-plane-metrics/metrics.png" alt-text="Screenshot showing metrics query." lightbox="media/control-plane-metrics/metrics.png":::
+
+To view control plane metrics such as APIServer and ETCD, you can customize the scraping of Prometheus metrics by applying the config maps to your cluster. The metrics pods pick up the config maps and pods restart in 2-3 minutes. Follow these steps to enable.
+
+### Step 1: connect to Kubernetes
+
+Connect to your cluster using `az connectedk8s proxy` and run `kubectl get pods -A` to make sure you're connected.
+
+### Step 2: download the configuration files and review the content
+
+Managed Prometheus uses an [agent-based solution](https://github.com/Azure/prometheus-collector) to collect Prometheus metrics and send them to the Azure Monitor workspace. There are two configuration files to download and review: **ama-metrics-settings-configmap.yaml** and **ama-metrics-prometheus-config-configmap.yaml**. For more information about customizing metrics scraping using configuration files, see [Customize scraping of Prometheus metrics](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration).
+
+- To enable **APIServer** metrics, modify the value of `apiserver` under the default OFF targets and set it to **true** in **ama-metrics-settings-configmap.yaml**. For the list of metrics, see [minimal ingestion for default OFF targets](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration-minimal#minimal-ingestion-for-default-off-targets).
+- To enable metrics from components not listed under default OFF targets such as **ETCD, Controller Manager, Kube Scheduler**, add a new scraping job in **ama-metrics-prometheus-config-configmap.yaml**.
+
+[You can download](https://github.com/Azure/aksArc/tree/main/scripts/ControlPlaneMetrics) these two configuration files to your local machine and review the content before going to the next step.
+
+### Step 3: apply custom configuration files
+
+Run the following commands to apply the changes, then wait several minutes for the metrics pods to restart.
+
+```bash
+kubectl apply -f ama-metrics-settings-configmap.yaml
+kubectl apply -f ama-metrics-prometheus-config-configmap.yaml
+```
+
+### Step 4: query metrics in Azure Monitor workspace
+
+Go to the linked **Azure Monitor Workspace > Metric Explorer** and use PromQL to validate that the metrics are ingested. In the following sample query, it shows a stable kube-scheduler metrics `scheduler_schedule_attempts_total` from a specific Kubernetes cluster.
+
+:::image type="content" source="media/control-plane-metrics/metrics-ingested.png" alt-text="Screenshot of portal showing metrics ingestion." lightbox="media/control-plane-metrics/metrics-ingested.png":::
+
+## View metrics in Grafana
+
+Metrics Explorer is convenient for metrics validation. To operationalize Kubernetes monitoring with Azure Monitor, it's recommended that you monitor the metrics using Azure Managed Grafana.
+
+### Step 1: install Azure Managed Grafana
+
+[Follow these instructions](/azure/managed-grafana/how-to-connect-azure-monitor-workspace) to create a Grafana workspace, link to an Azure Monitor workspace, and view the metrics in Grafana dashboards. You can view the dashboard under **Monitoring > Insights > Monitor Settings**. Multiple instances can be linked to the same Azure Monitor workspace, so make sure to choose the right dashboard.
+
+### Step 2: import a prebuilt dashboard for control plane metrics
+
+[Download the API server dashboard ](https://grafana.com/grafana/dashboards/20331-kubernetes-api-server/) to your local machine, copy the JSON content, then import it to the managed Grafana dashboard.
+
+:::image type="content" source="media/control-plane-metrics/dashboards.png" alt-text="Screenshot showing metrics dashboard." lightbox="media/control-plane-metrics/dashboards.png":::
+
+### Step 3: view metrics in the dashboard
+
+Ensure that the data source and cluster names are correct. You can view the metrics in Grafana and customize them as needed.
+
+:::image type="content" source="media/control-plane-metrics/metrics-status.png" alt-text="Screenshot showing control plane metrics status." lightbox="media/control-plane-metrics/metrics-status.png":::
+
+## Next steps
+
+- [AKS Arc monitoring data reference](kubernetes-monitor-metrics.md)
+- [Prometheus scrape configuration](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration)