⚠️ [Warm Replicas] Implement warm replica support for controllers. #3192
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: godwinpang The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @godwinpang. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
do-not-merge/work-in-progress
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
godwinpang
force-pushed
the
warm-replica-impl
branch
from
5d1be84 to
5db61c7
Compare
|
/retest |
| @@ -439,6 +439,11 @@ func (cm *controllerManager) Start(ctx context.Context) (err error) { | |||
| return fmt.Errorf("failed to start other runnables: %w", err) | |||
| } | |||
|
|
|||
| // Start and wait for sources to start. | |||
| if err := cm.runnables.Warmup.Start(cm.internalCtx); err != nil { | |||
There was a problem hiding this comment.
I think this should come right after the caches and the comment shouldn't make assumptions about what the Warmup internally does.
The other issue: This needs to block until the Warmup has terminated, otherwise we may end up starting the controller before the sources are started, as the check we added only checks if we started to start the sources, not if we finished doing so
There was a problem hiding this comment.
Will change to coming right after the caches, but don't completely understand the significance of having it after vs. before the non-leader election runnables; mind explaining a bit?
There was a problem hiding this comment.
This needs to block until the Warmup has terminated, otherwise we may end up starting the controller before the sources are started, as the check we added only checks if we started to start the sources, not if we finished doing so
Is this a big problem? Replicas will only start the controller after they win leader election so I don't see an issue in the leader election failover case; are you saying that in non-leader election cases the behavior of warmup should be that it completely blocks controller startup?
| @@ -314,6 +314,12 @@ type LeaderElectionRunnable interface { | |||
| NeedLeaderElection() bool | |||
| } | |||
|
|
|||
| // WarmupRunnable knows if a Runnable should be a warmup runnable. | |||
| type WarmupRunnable interface { | |||
| // Warmup returns true if the Runnable should be run as warmup. | |||
| @@ -314,6 +314,12 @@ type LeaderElectionRunnable interface { | |||
| NeedLeaderElection() bool | |||
| } | |||
|
|
|||
| // WarmupRunnable knows if a Runnable should be a warmup runnable. | |||
| type WarmupRunnable interface { | |||
There was a problem hiding this comment.
Please explain the purpose of it in the godoc
godwinpang
force-pushed
the
warm-replica-impl
branch
from
5db61c7 to
be1b1c2
Compare
|
/retest |
|
/retest |
|
/retest |
This change implements the proposal for warm replicas as proposed in #3121.
It adds a
NeedWarmUpoption for controllers to optionally start as warmed replicas.Note for reviewers: This draft PR is feature complete with tests but the main purpose is to make sure that things are on the right track. Some of the naming / comments are a bit inconsistent, I will address them in a followup cleanup tomorrow.
Builds upon #3190.