Queue Workflows for Specific Instance Types / Arch / OS

[nc-billingsj]

Hello!

I have a use case where I need to allocate certain Github Actions Workflows to specific EC2 instance types. For example, Workflow_1 needs to run on a t3.small, while Workflow_2 needs to run on a t3.large. I thought that #1019 would enable this support. I expected that I could specify instance_types = ["t3.small", "t3.large"] and runner_extra_labels = "ci-runner-small,ci-runner-large and then set Workflow_1 runs-on: "ci-runner-small and Workflow_2 runs-on: "ci-runner-large"; basically mapping a runner label to an instance type. However, that doesn't seem to work since all labels are used for each launch template. Is there a way to achieve this? Thank you in advance.

[nahsi]

@nc-billingsj I use several module invocations to achieve this.

i.e. I have module "runner_t3small", module "runner_t3large" etc

[pbrewer-ifit]

Can you elaborate a little? Did you call this on the top level module or on some of the sub-modules?

[nahsi]

Sorry for a very late answer. I create a module for each instance type.

[npalm]

Adding the use case

1. group
2. runners with os linux and windows

[npalm]

How to support different instance types / architectures / runner os?

Option 1 - deployng the stack multiple times

Deploy the stack one time per instance / arch / os type

This requires one webhook or webhook app per deployment to route events for scaling. Requires the webhook only accept jobs meant for the stack. The lambdas for scaling could use the same app in the multiple stacks.

Pros:

• no change to the modue required
• no extra complecity on the module is introduced

Cons:

• complexer deployment
• multiple webhooks required

Option 2 - Build in via mutliple deployments of the runner stack

A second option could be let the webhook distribute the evetns based on the label to a queue. Per queue connect a runner module that scales for the required instance type / arch / os. The solution cuould be relalied by a submodule that acts a root module which can read config for mutlipel type of runners. Creates a queue and runner per type.

Pros:

• One webhook
• Easier te deploy
• works with features like scale down and the pool (maybe it will work out of the box via the environment variable)

Cons:

• Needs to be implementd
• More lambda, and queues to monitor
• Require also changes to the scale down en pool mechaism.

Option 3

In this option all the logic is handled in the lambda. Based on the job labels it should look up an config and do the scaling.

Pro:

• ?

Cons:

• Complexity of the lamba's will increas
• Complexer to implement

[ScottGuymer]

I drew a nice picture of option 2

[pbrewer-ifit]

I would love to see option 2 implemented.
But I'd like to hear the experiences of any one currently using Option 1. I have embarked on trying to implement multiple pools to support multiple OS and multiple architectures. I have to imagine this is a common need as every where I have worked for last 5 years has had similar needs for CI build systems.

[ScottGuymer]

This is on our radar for the future. First we want to bring in some monitoring to allow teams to see what is happening in their runners.

[rashidnhm]

Chiming in on this discussion as I feel we have a similar use case pending regarding workflows.

Question:

• For option 1: Is there a downside to having each webhook on a repository triggering if there are multiple webhook setup against it?
• For option 3: How much more complexity would it add to lambdas to handle the new logic?

Having a parameter like the following would be nice:

instance_type_labels = {
   gh-runner-small = ["t2.micro", "t2.small"],
   gh-runner-large = ["t2.xlarge"]
}

[npalm]

Thanks for jumping on the discussion. Downside of this qpproach is supporting in the same way also other architecture or os side. Since that part is defined in Terraform. Also features like max number of instances, a pool and so on will not work with this option out of the box. Option 2 will make all current features available together wih having multiple instance types / configurations

[rashidnhm]

So if I understand option 2 -> The webhook lambda would put the events into different SQS queues based on label of the workflow job.

Would there still be one scale up / down lambda checking runners for all queues? Or different lambda per queue?

[npalm]

The idea is to deploy the runner module per queue, so that means the lambda's in that module are deployed multiple times.

[rashidnhm]

Thinking of option 1, if the webhook lambda is triggered with workflow_job event but the label provided does not match any label to runner_extra_labels, then is the event dropped by the webhook lambda? Or is it put in the SQS queue and then dropped by the scale up lambda?

[npalm]

We started on implementation, see https://github.com/philips-labs/terraform-aws-github-runner/pull/2472