feat(kernelLogWatcher): enable revive kmsg parser if channel closed #1004
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
Hi @daveoy. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: daveoy The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
size/M
|
should close #1003 |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
Is CI okay? Looks like the same failure on my other PRs marked ok to test.... |
|
Yeah, I haven't had time to look into them. Do you have time to help debug and fix? /retest |
|
Yes I can take a look |
|
@daveoy: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| return nil | ||
| } | ||
|
|
||
| // revive ourselves if the kmsg channel is closed |
There was a problem hiding this comment.
I am curious about the error you observe when the channel is closed. Do you see the "Kmsg channel closed" log? Is it always associated with high load of the node?
| @@ -99,6 +100,9 @@ func (k *kernelLogWatcher) watchLoop() { | |||
| return | |||
| case msg, ok := <-kmsgs: | |||
| if !ok { | |||
| if val, ok := k.cfg.PluginConfig["revive"]; ok && val == "true" { | |||
There was a problem hiding this comment.
Can you add a sample config for using revive?
| @@ -99,6 +100,9 @@ func (k *kernelLogWatcher) watchLoop() { | |||
| return | |||
| case msg, ok := <-kmsgs: | |||
| if !ok { | |||
| if val, ok := k.cfg.PluginConfig["revive"]; ok && val == "true" { | |||
| k.reviveMyself() | |||
| } | |||
| klog.Error("Kmsg channel closed") | |||
There was a problem hiding this comment.
This log line should be before the revive.
| @@ -99,6 +100,9 @@ func (k *kernelLogWatcher) watchLoop() { | |||
| return | |||
| case msg, ok := <-kmsgs: | |||
| if !ok { | |||
| if val, ok := k.cfg.PluginConfig["revive"]; ok && val == "true" { | |||
| k.reviveMyself() | |||
There was a problem hiding this comment.
Before revive, add a log line about we are starting to revive.
| // enter the watch loop again | ||
| func (k *kernelLogWatcher) reviveMyself() { | ||
| // if k.reviveCount >= reviveRetries { | ||
| // klog.Errorf("Failed to revive kmsg parser after %d retries", reviveRetries) |
There was a problem hiding this comment.
For the retries, is there any time boundary needed? For example, if the revive happens 10 times over a year, it may be fine. But if revive happens 10 times over a minute, something is wrong.
| // close the old kmsg parser and create a new one | ||
| // enter the watch loop again | ||
| func (k *kernelLogWatcher) reviveMyself() { | ||
| // if k.reviveCount >= reviveRetries { |
this PR adds a revival mechanism for a recurring issue im facing.
in kubernetes, when a node is under significant load, the connection to /dev/kmsg can be closed unexpectedly.
instead of exiting the watcher and restarting the whole pod (subsequently clearing any conditions set by NPD during the new pod's problem daemon init), i would like to revive the kmsg channel and continue execution.