ASG warm pool support#519
Conversation
…SG - remove scale-in protection and allow ASG to terminate them or move them to warm pool.
|
This change breaks the use case of configuring a maximum number of builds. When an instance reaches the maximum number of builds, the desired capacity may not change, but the plugin will set the instance offline within Jenkins then execute the termination. If the plugin is changed to no longer issue terminations, then those instances would never be replaced and just sit idle not able to process any jobs. |
|
Added logic - terminate if max uses exhausted, remove scale in protection otherwise. |
|
I've not had time to fully look this over and test this change, however I would say that I would be more comfortable with this change if it did a check for the existence of a warm pool with instance reuse policy and only applied the new behavior if that was found. Otherwise, it can keep doing things how it does today. |
I've recalled an additional detail relevant to my desire to keep the existing behavior if not using an instance reuse policy. The reason which the plugin terminates instances in the way it does came about after discussions with AWS engineers. Essentially, due to some of the internals of how the service works, in high volume environments you can end up with some significant lag in scaling activities, which is something we encountered in our environment before I updated the plugin to do what it currently does. The TerminateInstanceInAutoScalingGroup call tells the ASG to take immediate action rather than rely on the internal processes which run on a schedule and can throttle scaling activity. |
The idea behind this PR is to allow ASG to use warm pool, for faster builds - it's faster to get stopped instance from warm pool, than spin up it from image, and stopped instance will have caches/checked out code from last build.
Implementation details:
Instead of terminating instance - plugin removes scale-in protection from the instance it wants to terminate and allows ASG to handle instance termination - AGS will move this instance back to warm pool if there is one configured.
Related to #352
Testing done
Tested on live jenkins instance, and corrected automated tests.
Submitter checklist