Skip to content
This repository was archived by the owner on Jan 16, 2025. It is now read-only.

Commit fef8d65

Browse files
npalmnavdeepg2021
npalm
and
navdeepg2021
committed
feat!: Add multi-runner capability (#2472)
Migration: - Does the main module needs to be redeployed after this feature is released? There is no change in Terraform version for the main module, so it will not require TF upgrade. There is only one change in main module, which is a breaking change, the variable runner_enable_workflow_job_labels_check is not available under this module as we now only support the flagrunner_enable_workflow_job_labels_check_all to check all/partial workflow labels to be present in the runner labels. Running TF apply on existing deployment will apply some changes done in internal modules to existing infrastructure. - Impact on using internal (submodules). There are some changes in the webhook internal module as per this feature. So, If you have been using it directly, the changes will be towards accepting runner configuration (queue details and label matchers) as a single object instead of earlier configuration of scattered queue details and criteria to match the workflow labels against the runner labels. Co-authored-by: Niek Palm <[email protected]> Co-authored-by: Niek Palm <[email protected]> Co-authored-by: navdeepg2021 <[email protected]>
1 parent 7506e9d commit fef8d65

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2197
-1099
lines changed

‎.github/workflows/terraform.yml

+31-4
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
name: Verify module
1515
strategy:
1616
matrix:
17-
terraform: [1.1.3, "latest"]
17+
terraform: [1.3.2, "latest"]
1818
runs-on: ubuntu-latest
1919
container:
2020
image: hashicorp/terraform:${{ matrix.terraform }}
@@ -28,7 +28,7 @@ jobs:
2828
touch modules/runner-binaries-syncer/lambdas/runner-binaries-syncer/runner-binaries-syncer.zip
2929
- name: terraform init
3030
run: terraform init -get -backend=false -input=false
31-
- if: contains(matrix.terraform, '1.1.')
31+
- if: contains(matrix.terraform, '1.3.')
3232
name: check terraform formatting
3333
run: terraform fmt -recursive -check=true -write=false
3434
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
@@ -43,7 +43,7 @@ jobs:
4343
strategy:
4444
fail-fast: false
4545
matrix:
46-
terraform: [1.0.11, 1.1.3, "latest"]
46+
terraform: [1.0.11, 1.1.9, 1.2.9, "latest"]
4747
example:
4848
["default", "ubuntu", "prebuilt", "arm64", "ephemeral", "windows"]
4949
defaults:
@@ -56,7 +56,7 @@ jobs:
5656
- uses: actions/checkout@v3
5757
- name: terraform init
5858
run: terraform init -get -backend=false -input=false
59-
- if: contains(matrix.terraform, '1.1.')
59+
- if: contains(matrix.terraform, '1.3.')
6060
name: check terraform formatting
6161
run: terraform fmt -recursive -check=true -write=false
6262
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
@@ -65,3 +65,30 @@ jobs:
6565
continue-on-error: true
6666
- name: validate terraform011
6767
run: terraform validate
68+
69+
70+
verify_multi_runner_example:
71+
name: Verify Multi-Runner examples
72+
strategy:
73+
fail-fast: false
74+
matrix:
75+
terraform: [1.3.2, "latest"]
76+
defaults:
77+
run:
78+
working-directory: examples/multi-runner
79+
runs-on: ubuntu-latest
80+
container:
81+
image: hashicorp/terraform:${{ matrix.terraform }}
82+
steps:
83+
- uses: actions/checkout@v3
84+
- name: terraform init
85+
run: terraform init -get -backend=false -input=false
86+
- if: contains(matrix.terraform, '1.3.')
87+
name: check terraform formatting
88+
run: terraform fmt -recursive -check=true -write=false
89+
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
90+
name: check terraform formatting
91+
run: terraform fmt -recursive -check=true -write=false
92+
continue-on-error: true
93+
- name: validate terraform
94+
run: terraform validate

‎README.md

+16-17
Large diffs are not rendered by default.

‎examples/ephemeral/main.tf

-3
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,6 @@ module "runners" {
4343
enable_organization_runners = true
4444
runner_extra_labels = "default,example"
4545

46-
# enable workflow labels check
47-
# runner_enable_workflow_job_labels_check = true
48-
4946
# enable access to the runners via SSM
5047
enable_ssm_on_runners = true
5148

‎examples/multi-runner/.terraform.lock.hcl

+60
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/multi-runner/README.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Action runners deployment of Multiple-Runner-Configurations-Together example
2+
3+
This module shows how to create GitHub action runners with multiple runner configuration together in one deployment. This example has the configurations for the following runner types with the relevant labels supported by them as matchers:
4+
5+
- Linux ARM64 `["self-hosted", "linux", "arm64", "amazon"]`
6+
- Linux Ubuntu `["self-hosted", "linux", "x64", "ubuntu"]`
7+
- Linux X64 `["self-hosted", "linux", "x64", "amazon"]`
8+
- Windows X64 `["self-hosted", "windows", "x64", "servercore-2022"]`
9+
10+
The module will decide the runner for the workflow job based on the match in the labels defined in the workflow job and runner configuration. Also the runner configuration allows the match to be exact or non-exact match. We recommend to use only exact matches.
11+
12+
For exact match, all the labels defined in the workflow should be present in the runner configuration matchers and for non-exact match, some of the labels in the workflow, when present in runner configuration, shall be enough for the runner configuration to be used for the job. First the exact matchers are applied, next the non exact ones.
13+
14+
## Webhook
15+
16+
For the list of provided runner configurations, there will be a single webhook and only a single Github App to receive the notifications for all types of workflow triggers.
17+
18+
## Lambda distribution
19+
20+
Per combination of OS and architecture a lambda distribution syncer will be created. For this example there will be three instances (windows X64, linux X64, linux ARM).
21+
22+
## Usages
23+
24+
Steps for the full setup, such as creating a GitHub app can be found in the root module's [README](../../README.md). First download the Lambda releases from GitHub. Alternatively you can build the lambdas locally with Node or Docker, there is a simple build script in `<root>/.ci/build.sh`. In the `main.tf` you can simply remove the location of the lambda zip files, the default location will work in this case.
25+
26+
> Ensure you have set the version in `lambdas-download/main.tf` for running the example. The version needs to be set to a GitHub release version, see https://github.com/philips-labs/terraform-aws-github-runner/releases
27+
28+
```bash
29+
cd lambdas-download
30+
terraform init
31+
terraform apply
32+
cd ..
33+
```
34+
35+
Before running Terraform, ensure the GitHub app is configured. See the [configuration details](../../README.md#usages) for more details.
36+
37+
```bash
38+
terraform init
39+
terraform apply
40+
```
41+
42+
You can receive the webhook details by running:
43+
44+
```bash
45+
terraform output -raw webhook_secret
46+
```
47+
48+
Be-aware some shells will print some end of line character `%`.

‎examples/multi-runner/lambdas-download/main.tf

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
locals {
2+
version = "<REPLACE_BY_GITHUB_RELEASE_VERSION>"
3+
}
4+
5+
module "lambdas" {
6+
source = "../../../modules/download-lambda"
7+
lambdas = [
8+
{
9+
name = "webhook"
10+
tag = local.version
11+
},
12+
{
13+
name = "runners"
14+
tag = local.version
15+
},
16+
{
17+
name = "runner-binaries-syncer"
18+
tag = local.version
19+
}
20+
]
21+
}
22+
23+
output "files" {
24+
value = module.lambdas.files
25+
}

‎examples/multi-runner/main.tf

+165
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
locals {
2+
environment = var.environment != null ? var.environment : "multi-runner"
3+
aws_region = "eu-west-1"
4+
}
5+
6+
resource "random_id" "random" {
7+
byte_length = 20
8+
}
9+
data "aws_caller_identity" "current" {}
10+
11+
12+
################################################################################
13+
### Hybrid account
14+
################################################################################
15+
16+
module "multi-runner" {
17+
source = "../../modules/multi-runner"
18+
multi_runner_config = {
19+
"linux-arm64" = {
20+
matcherConfig : {
21+
labelMatchers = ["self-hosted", "linux", "arm64", "amazon"]
22+
exactMatch = true
23+
}
24+
fifo = true
25+
delay_webhook_event = 0
26+
redrive_build_queue = {
27+
enabled = false
28+
maxReceiveCount = null
29+
}
30+
runner_config = {
31+
runner_os = "linux"
32+
runner_architecture = "arm64"
33+
runner_extra_labels = "arm"
34+
enable_ssm_on_runners = true
35+
instance_types = ["t4g.large", "c6g.large"]
36+
runners_maximum_count = 1
37+
scale_down_schedule_expression = "cron(* * * * ? *)"
38+
}
39+
},
40+
"linux-ubuntu" = {
41+
matcherConfig : {
42+
labelMatchers = ["self-hosted", "linux", "x64", "ubuntu"]
43+
exactMatch = true
44+
}
45+
fifo = true
46+
delay_webhook_event = 0
47+
redrive_build_queue = {
48+
enabled = false
49+
maxReceiveCount = null
50+
}
51+
runner_config = {
52+
runner_os = "linux"
53+
runner_architecture = "x64"
54+
runner_extra_labels = "ubuntu"
55+
enable_ssm_on_runners = true
56+
instance_types = ["m5ad.large", "m5a.large"]
57+
runners_maximum_count = 1
58+
scale_down_schedule_expression = "cron(* * * * ? *)"
59+
runner_run_as = "ubuntu"
60+
userdata_template = "./templates/user-data.sh"
61+
ami_owners = ["099720109477"] # Canonical's Amazon account ID
62+
63+
ami_filter = {
64+
name = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
65+
}
66+
block_device_mappings = [{
67+
# Set the block device name for Ubuntu root device
68+
device_name = "/dev/sda1"
69+
delete_on_termination = true
70+
volume_type = "gp3"
71+
volume_size = 30
72+
encrypted = true
73+
iops = null
74+
throughput = null
75+
kms_key_id = null
76+
snapshot_id = null
77+
}]
78+
runner_log_files = [
79+
{
80+
log_group_name = "syslog"
81+
prefix_log_group = true
82+
file_path = "/var/log/syslog"
83+
log_stream_name = "{instance_id}"
84+
},
85+
{
86+
log_group_name = "user_data"
87+
prefix_log_group = true
88+
file_path = "/var/log/user-data.log"
89+
log_stream_name = "{instance_id}/user_data"
90+
},
91+
{
92+
log_group_name = "runner"
93+
prefix_log_group = true
94+
file_path = "/opt/actions-runner/_diag/Runner_**.log",
95+
log_stream_name = "{instance_id}/runner"
96+
}
97+
]
98+
}
99+
},
100+
"windows-x64" = {
101+
matcherConfig : {
102+
labelMatchers = ["self-hosted", "windows", "x64", "servercore-2022"]
103+
exactMatch = true
104+
}
105+
fifo = true
106+
delay_webhook_event = 5
107+
runner_config = {
108+
runner_os = "windows"
109+
runner_architecture = "x64"
110+
enable_ssm_on_runners = true
111+
instance_types = ["m5.large", "c5.large"]
112+
runner_extra_labels = "servercore-2022"
113+
runners_maximum_count = 1
114+
scale_down_schedule_expression = "cron(* * * * ? *)"
115+
runner_boot_time_in_minutes = 20
116+
ami_filter = {
117+
name = ["Windows_Server-2022-English-Core-ContainersLatest-*"]
118+
}
119+
}
120+
},
121+
"linux-x64" = {
122+
matcherConfig : {
123+
labelMatchers = ["self-hosted", "linux", "x64", "amazon"]
124+
exactMatch = false
125+
}
126+
fifo = true
127+
delay_webhook_event = 0
128+
runner_config = {
129+
runner_os = "linux"
130+
runner_architecture = "x64"
131+
create_service_linked_role_spot = true
132+
enable_ssm_on_runners = true
133+
instance_types = ["m5ad.large", "m5a.large"]
134+
runner_extra_labels = "amazon"
135+
runners_maximum_count = 1
136+
enable_ephemeral_runners = true
137+
scale_down_schedule_expression = "cron(* * * * ? *)"
138+
}
139+
}
140+
}
141+
aws_region = local.aws_region
142+
vpc_id = module.vpc.vpc_id
143+
subnet_ids = module.vpc.private_subnets
144+
runners_scale_up_lambda_timeout = 60
145+
runners_scale_down_lambda_timeout = 60
146+
prefix = local.environment
147+
tags = {
148+
Project = "ProjectX"
149+
}
150+
github_app = {
151+
key_base64 = var.github_app_key_base64
152+
id = var.github_app_id
153+
webhook_secret = random_id.random.hex
154+
}
155+
156+
# Assuming local build lambda's to use pre build ones, uncomment the lines below and download the
157+
# lambda zip files lambda_download
158+
# webhook_lambda_zip = "lambdas-download/webhook.zip"
159+
# runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"
160+
# runners_lambda_zip = "lambdas-download/runners.zip"
161+
162+
# override delay of events in seconds
163+
164+
# log_level = "debug"
165+
}

‎examples/multi-runner/outputs.tf

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
output "webhook_endpoint" {
2+
value = module.multi-runner.webhook.endpoint
3+
}
4+
5+
output "webhook_secret" {
6+
sensitive = true
7+
value = random_id.random.hex
8+
}

‎examples/multi-runner/providers.tf

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
provider "aws" {
2+
region = local.aws_region
3+
}

0 commit comments

Comments
 (0)