-
Notifications
You must be signed in to change notification settings - Fork 53
Controller worker thread times out during config set #652
Comments
tagging at rc1 since this is something we need to fix before we cut a stable release. |
ping @helgi; has this been fixed recently? |
No, and it won't be done in RC. This isn't a kubernetes problem, it's more the fact we are trying to execute any operation within the timeout of the |
Oh, what has helped is doing deploys in batches, and by default rolling as many pods as available nodes but that only mitigates the issue in some scenarios |
@kmala are you still able to reproduce this in your environment with |
This situation should be improved by the batching operation of the current controller, although probably not fixed definitively. |
This issue was moved to teamhephy/controller#66 |
when we try to do a config:set on an app with 30-40 pods the controller worker thread is timing out because the operation is taking more than 20min(default timeout of the worker thread) keeping the cluster in an unstable state with pods of both releases.
The text was updated successfully, but these errors were encountered: