The primary motivation for producing AKS specific Windows VHDs is to improve the reliability and deployment time for the configuration phase of Windows nodes configured by AKS Engine. By performing expensive configuration options and pre-downloading installation artifacts as part of the VHD build process we can accomplish both of these goals.
Producing and publishing AKS specific Windows VHDs to the Azure marketplace also allow us to get targeted windows patches to customers sooner. Today the Windows/Windows Server team only publish new images containing the latest B week patches. Several times recently C week patches contained fixes for issues customers are facing running Kubernetes on Windows nodes. This means customers would either need to wait an additional 3-4 weeks to get these fixes (until the fixes are released in next months cumulative B week update and new marketplace images are published) or some mechanism to deliver these patches to Windows nodes would need to be developed.
Lastly publishing AKS specific Windows VHDs allows us to perform adequate testing on new patches before allowing customers to customers to upgrade their Windows nodes. Given the challenging nature of validating private Windows fixes, we will work with the Windows team to incorporate Kubernetes testing before patches get released publicly. (see Testing Private Fixes section below for more details)
Windows AKS VHDs are produced using packer in an Azure DevOps build pipeline.
The build pipeline:
- Runs a packer job which provisions a new VM in Azure based off a specified Windows Server image and:
- Installs/enables required Windows components/features
- Download multiple versions of components which users are allowed to specify the version of (Kubernetes, Docker, etc)
- Downloads commonly used container images
- Configure system settings (windows update, page file sizes, etc)
- Creates release notes detailing what is installed (features/QFEs/services) and what artifacts are cached on the image
- Runs AKS Engine E2E tests using the VHD produced in the same pipeline
- Note: A full upstream E2E test pass of Kubernetes will come later
- Optionally copies the VHD to another Azure storage account for extra validation and/or publishing
Note: See scripts referenced below to for further details
- Azure DevOps build yaml build definition: .pipelines/vhd-builder-windows.yaml
- Packer job definition: vhd/packer/windows-vhd-builder.json
- Customization script used in packer: vhd/packer/configure-windows-vhd.ps1
Occasionally it may be necessary to validate private fixes provided by the Windows team. This can be done without providing public access to the fixes by following these steps:
- Fork Azure/aks-engine into a private git repo hosted in Azure DevOps
- Configure a private build pipeline using .pipelines/vhd-builder-windows.yaml as the pipeline definition
- Set up the build pipeline variables to point to private Azure subscriptions
- Pipeline variables are documented in the yaml file
- Update vhd/packer/configure-windows-vhd.ps1 to install privates
- Run the pipeline
Both Windows and Linux nodes are configured by executing Custom Script Extensions as part of VM deployment operations.
The scripts are generated by aks-engine generate or aks-engine deploy, which populates instance-specific information from the cluster definition API model and get embedded into the ARM templates produced by AKS engine. These scripts are enlightened to understand the work in conjunction with Windows AKS marketplace images to not duplicate configuration steps and/or utilize files cached during the VHD build process.
Parts/k8s/kuberneteswindowssetup.ps1 and associated ps1 files are used as templates for the extension scripts.
It is an explicit goal to maintain AKS Engine compatibility with Windows Server marketplace images published by MicrosoftWindowsServer (for most scenarios) and to treat the Windows AKS marketplace images as an optimization for the reasons stated above.
It is a non-goal to produce Windows Server marketplace images with only patches installed.