Skip to content

TMP: Clean up self hosted runaways #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6b7959e
Self hosted runners: make VM image configurable through environment v…
dennisameling Nov 22, 2023
59003a9
Self hosted runners: update image from 22h2 to 23h2
dennisameling Nov 22, 2023
f428d14
Self hosted runners: add cleanup script
dennisameling Nov 22, 2023
9beb913
Merge branch 'cleanup-self-hosted-runners'
dscho Sep 14, 2024
7bd6ad3
cleanup-self-hosted-runners: use a newer GitHub CLI version
dscho Sep 14, 2024
91c4017
cleanup-self-hosted-runner: order the conditional blocks from small t…
dscho Sep 14, 2024
cb7c524
Add a shell script to authenticate with `gh` as an App
dscho Sep 14, 2024
692b6f4
cleanup-self-hosted-runner: avoid deleting VMs that are busy
dscho Sep 14, 2024
55fbe54
cleanup-self-hosted-runner: clean up idle runners earlier
dscho Sep 14, 2024
ded928e
cleanup-self-hosted-runner: fix some white-space issue
dscho Sep 14, 2024
8acd97b
cleanup-self-hosted-runner: use workflow commands for better mark-up
dscho Sep 14, 2024
2c67ce6
TO-DROP: run on push
dennisameling Sep 15, 2024
89a5352
TO-DROP: run on pull_request instead of push
dennisameling Sep 15, 2024
332a05d
Azure Actions: upgrade to v2 and pin Azure CLI to 2.63.0
dennisameling Sep 15, 2024
1fcd7a8
set -x for debugging
dennisameling Sep 15, 2024
577630b
Fix missing quote in jq command
dennisameling Sep 15, 2024
b90b227
Trim ISO date string
dennisameling Sep 15, 2024
cc19e05
Another attempt to fix things
dennisameling Sep 15, 2024
070a161
Another attempt to fix
dennisameling Sep 15, 2024
300182d
Another attempt to fix
dennisameling Sep 15, 2024
a99784a
Add quotes
dennisameling Sep 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/cleanup-self-hosted-runners.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Cleanup Azure self hosted runners
run-name: Cleanup Azure self hosted runners

on:
schedule:
# Run every 6 hours
- cron: "0 */6 * * *"
workflow_dispatch:
pull_request:

# The following secrets are required for this workflow to run:
# AZURE_CREDENTIALS - Credentials for the Azure CLI. It's recommended to set up a resource
# group specifically for self-hosted Actions Runners.
# az ad sp create-for-rbac --name "{YOUR_DESCRIPTIVE_NAME_HERE}" --role contributor \
# --scopes /subscriptions/{SUBSCRIPTION_ID_HERE}/resourceGroups/{RESOURCE_GROUP_HERE} \
# --sdk-auth
# AZURE_RESOURCE_GROUP - Resource group to create the runner(s) in
jobs:
delete-runner:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Azure Login
uses: azure/login@v2
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Discover VMs to delete
uses: azure/CLI@v2
env:
GH_APP_ID: ${{ secrets.GH_APP_ID }}
GH_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }}
with:
# Stick to 2.63.0 until jq is added to 2.64.0+ https://github.com/Azure/azure-cli/issues/29830
azcliversion: 2.63.0
inlineScript: |
set -x
active_vms=$(az vm list -g ${{ secrets.AZURE_RESOURCE_GROUP }} | jq -c '.[] | {name,timeCreated}')
current_time=$(date +%s)
one_hour_ago=$(($current_time - 3600))

if [ -z "$active_vms" ]; then
echo "No active VMs found, nothing to do."
exit 0
else
echo "Found these active VMs:"
echo $active_vms
fi

for active_vm in ${active_vms[@]}; do
vm_name=$(echo $active_vm | jq '.name')
# Use jq to extract and format the date-time string
vm_creation_time_string=$(echo $active_vm | jq -r '.timeCreated | sub("\\.[0-9]+[+-][0-9]+:[0-9]+$"; "") | sub("T"; " ")')
vm_creation_time=$(TZ=UTC date -d "$vm_creation_time_string" +%s)

if [ "$one_hour_ago" -lt "$vm_creation_time" ]; then
echo "::notice::The VM ${vm_name} was created less then 1 hour ago and shouldn't be deleted yet. Skipping."
elif test true = "$(if test ! -f .cli-authenticated; then
./gh-cli-auth-as-app.sh &&
>.cli-authenticated # only authenticate once
fi &&
gh api repos/$GITHUB_REPOSITORY/actions/runners \
--jq '.runners[] | select(.name == "'$vm_name'") | .busy')"; then
echo "::notice::The VM ${vm_name} is still busy."
else
echo "::warning::The VM ${vm_name} was created more than 3 hours ago and wasn't deleted. Let's do that now."
az vm delete -n "$vm_name" -g ${{ secrets.AZURE_RESOURCE_GROUP }} --yes
az network nsg delete -n "$vm_name"-nsg -g ${{ secrets.AZURE_RESOURCE_GROUP }}
az network vnet delete -n "$vm_name"-vnet -g ${{ secrets.AZURE_RESOURCE_GROUP }}
az network public-ip delete -n "$vm_name"-ip -g ${{ secrets.AZURE_RESOURCE_GROUP }}
fi
done
25 changes: 25 additions & 0 deletions gh-cli-auth-as-app.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/sh

node -e '(async () => {
const [owner, repo] = process.env.GITHUB_REPOSITORY.split("/")
const getAppInstallationId = require("./get-app-installation-id")
const installationId = await getAppInstallationId(
console,
process.env.GH_APP_ID,
process.env.GH_APP_PRIVATE_KEY,
owner,
repo
)
const getInstallationAccessToken = require("./get-installation-access-token")
const token = await getInstallationAccessToken(
console,
process.env.GH_APP_ID,
process.env.GH_APP_PRIVATE_KEY,
installationId
)
process.stderr.write(`::add-mask::${token.token}\n`)
process.stdout.write(token.token)
})().catch(e => {
process.stderr.write(JSON.stringify(e, null, 2))
process.exit(1)
})' | gh auth login --with-token
Loading