fix: fix node resolution cache for nodes in maintenance mode #890
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
utkuozdemir
force-pushed
the
fix-maintenance-node-resolution
branch
3 times, most recently
from
2e199c1 to
419c21d
Compare
Unix4ever
reviewed
Jan 30, 2025
Unix4ever
approved these changes
Jan 30, 2025
Sure, will do. |
There was a problem with the node resolution (a.k.a. DNS) cache of the nodes. When a machine is in maintenance mode, there is a corresponding `MachineStatus` resource for it, but there isn't any `ClusterMachineIdentity`. Both of these types trigger updates in the node resolution cache. When a machine was never part of a cluster, the only source is `MachineStatus`, and the cache updates on it did not populate the machine ID in the cache. This caused the GRPC router to pick the wrong destination. Furthermore, we did not remove the cluster and node name information from the cache when a machine was removed from a cluster. This caused the cache to contain obsolete cluster information, causing Talos GRPC proxy to not proxy the requests correctly after a machine was removed from a cluster. Co-authored-by: Artem Chernyshev <[email protected]> Signed-off-by: Utku Ozdemir <[email protected]>
utkuozdemir
force-pushed
the
fix-maintenance-node-resolution
branch
from
419c21d to
6f014b1
Compare
|
/m |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem with the node resolution (a.k.a. DNS) cache of the nodes. When a machine is in maintenance mode, there is a corresponding
MachineStatusresource for it, but there isn't anyClusterMachineIdentity. Both of these types trigger updates in the node resolution cache. When a machine was never part of a cluster, the only source isMachineStatus, and the cache updates on it did not populate the machine ID in the cache. This caused the GRPC router to pick the wrong destination.Furthermore, we did not remove the cluster and node name information from the cache when a machine was removed from a cluster. This caused the cache to contain obsolete cluster information, causing Talos GRPC proxy to not proxy the requests correctly after a machine was removed from a cluster.
Closes #888.