workflows: Rewrite build-ci-container to work on larger runners

[tstellar]

Also switch them over to the new depot runners.

[boomanaiden154]

Was just about to start hacking on this today. 😆

Not too familiar with the depot runners. It seems like they're paid self hosted runners? Is the foundation paying for them?

llvm-premerge-linux-runners should also work and will point to self hosted runners on the GCP project handling premerge. We intend to use that runner set to run the transitioned premerge jobs once we migrate to GHA.

[tstellar]

Not too familiar with the depot runners. It seems like they're paid self hosted runners? Is the foundation paying for them?

The Foundation is providing a small budget for testing them out. We're using 'Depot Managed' which means the instances are deployed in the Foundation's AWS account rather than using the machines from Depot's pool.

llvm-premerge-linux-runners should also work and will point to self hosted runners on the GCP project handling premerge. We intend to use that runner set to run the transitioned premerge jobs once we migrate to GHA.

Are these ready to use now? I don't have a preference for which runners to use.

[boomanaiden154]

The Foundation is providing a small budget for testing them out. We're using 'Depot Managed' which means the instances are deployed in the Foundation's AWS account rather than using the machines from Depot's pool.

Ah, okay. I didn't realize that had taken place. Thanks for the additional context.

Are these ready to use now? I don't have a preference for which runners to use.

Theoretically, yes, they are ready to use and should just work. We can only run one job at a time currently though because we don't have enough quota in the region where the cluster is deployed, but that should be enough for testing. @lnihlen is working on getting additional quota. I think it would probably make more sense to use the Google hosted runners given it doesn't require any budget from the foundation, and lets us consolidate on a single system.

I think our runners will also be more powerful too (configured to be 56 vCPUs), but depends upon how the depot runners are configured.

[tstellar]

The Foundation is providing a small budget for testing them out. We're using 'Depot Managed' which means the instances are deployed in the Foundation's AWS account rather than using the machines from Depot's pool.

Ah, okay. I didn't realize that had taken place. Thanks for the additional context.

Are these ready to use now? I don't have a preference for which runners to use.

Theoretically, yes, they are ready to use and should just work. We can only run one job at a time currently though because we don't have enough quota in the region where the cluster is deployed, but that should be enough for testing. @lnihlen is working on getting additional quota. I think it would probably make more sense to use the Google hosted runners given it doesn't require any budget from the foundation, and lets us consolidate on a single system.

OK, do you think I should update this PR to use those runners instead or wait a few weeks?

I think our runners will also be more powerful too (configured to be 56 vCPUs), but depends upon how the depot runners are configured.

Depot runners are up-to 64 CPUs. The number of CPUs is part of the runs-on label so you can choose the right size machine that fits the job.

[boomanaiden154]

OK, do you think I should update this PR to use those runners instead or wait a few weeks?

I would say update it to use the new runners. This job doesn't run very often and probably shouldn't take that long to run on a decent sized machine.

Depot runners are up-to 64 CPUs. The number of CPUs is part of the runs-on label so you can choose the right size machine that fits the job.

Ah, interesting. We don't have that much flexibility currently but could theoretically add the capability. Everything that I have seen/can imagine though either requires not much compute and should probably run on the free Github runners or requires a lot and would benefit from as many cores as possible.

[tstellar]

OK, do you think I should update this PR to use those runners instead or wait a few weeks?

I would say update it to use the new runners. This job doesn't run very often and probably shouldn't take that long to run on a decent sized machine.

Depot runners are up-to 64 CPUs. The number of CPUs is part of the runs-on label so you can choose the right size machine that fits the job.

Ah, interesting. We don't have that much flexibility currently but could theoretically add the capability. Everything that I have seen/can imagine though either requires not much compute and should probably run on the free Github runners or requires a lot and would benefit from as many cores as possible.

Yeah, the nice thing about the builds is that they are highly parallel, so if you are paying for runners, you may as well use the one with the most CPUs, because the cost/job ends up being roughly the same no matter how many CPUs you have.

[tstellar]

@boomanaiden154 It looks like thellvm-premerge-linux-runners require jobs to use a container and having nested containers doesn't work, so I may have to switch it back to the depot runners.

[boomanaiden154]

It looks like thellvm-premerge-linux-runners require jobs to use a container and having nested containers doesn't work, so I may have to switch it back to the depot runners.

Ah, right. I forgot about that detail. My plan was to use https://github.com/GoogleContainerTools/kaniko to build things on the cluster, but that's definitely something I can get to later.

[boomanaiden154]

Maybe add a comment on why the rm -rf ./build is here? I believe it was originally to avoid out of disk space errors. Assuming the depot runners have enough disk, it would still be useful as it probably reduces checkpointing time.

[tstellar]

I'm inclined to drop it. I think this is a common pattern when doing something on the final image, but since this is just a builder image, I'm not sure if it makes sense to do. I don't have a strong opinion though.

[boomanaiden154]

Seems reasonable enough to me.

[boomanaiden154]

LGTM after addressing the final comment.

[boomanaiden154]

@tstellar What are your plans on landing this? I wanted to try porting it to the premerge cluster this week, but would prefer to build off of this patch in tree.