Improve support for multiple schedulers #47
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rosa
force-pushed
the
multiple-schedulers
branch
2 times, most recently
from
f8aba1f to
98418ed
Compare
…are scheduled This change helps multiple schedulers running at the same time not waiting on each others' locks when deleting scheduled executions. Besides, it improves the selection of jobs to schedule and their deletion, to make sure only executions that have been in fact scheduled are deleted. The previous implementation assumed all selected jobs would be correctly promoted into ready executions, but if this was not the case, they'd simply be deleted and lost. Now we only delete the ones that have been, within the same transaction.
This comes from an idea from @djmb, so that, when we have multiple schedulers, they don't start all at the same time after a deploy, enqueuing All The Things and causing a thundering herd.
I had missed that we were running 5 and not 4 there, oops.
rosa
force-pushed
the
multiple-schedulers
branch
2 times, most recently
from
c375ca3 to
b2035b4
Compare
And wait until all semaphores have been signalled, as it might happen that jobs finish and before they've signaled the semaphore, we try to wait on it, having the test fail, if the timing aligns.
rosa
force-pushed
the
multiple-schedulers
branch
from
b2035b4 to
42e33d4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until now, everything around the scheduler was done under the assumption that a single process would be run for this purpose. This might not be true for everyone, and in particular, it won't be true for us very soon 😅 This PR introduces two changes:
SKIP LOCKED, to help multiple schedulers that are running at the same time not have to wait on each others' locks when deleting scheduled executions. Besides, it improves the selection of jobs to schedule and their deletion to make sure only executions that have been in fact scheduled are deleted. The previous implementation assumed all selected jobs would be correctly promoted to ready executions, but if this was not the case, they'd simply be deleted and lost. Now we only delete the ones that have been, within the same transaction.cc @mdkent