Bumping revisions after restore

[tjungblu]

We're currently looking into a revamp of our etcd restore logic. While testing, we observed many operators that are holding on to newer revisions in cache, which then might cause some undesired behavior in whatever they are controlling.
Imagine you're currently at rev 100, then restore etcd back to rev 80. There's a difference in 20 revisions, which will need to be caught up again to invalidate the caches.

Restarting all operators/nodes certainly works to start clean - but also means downtime for workload running on the cluster. We're looking for a "nicer" way to tell etcd to restore from a snapshot - but with revision +X, ideally bigger than it was before.

Is there a better way to achieve this? The way I understand it, we can simply load the snapshot and do +X put operations on an ephemeral key that can be deleted after.

[serathius]

I have been looking into this as it affects K8s 1.25+ versions. Tested couple of approaches, but due to lack of time needed to postpone the work. Feel free to file a feature request.

[tjungblu]

if you find the time, please let us know the approaches you've tried already. Happy to take it from there.

[serathius]

I have been looking into bumping compaction revision fields in bbolt meta bucket. Idea is to hack db file so etcd will compact old data and start from higher revision. However this is hacky and I encountered some edge cases that lead to bad etcd behavior. Didn't have time to debug them.

[dusk125]

This is being addressed in #16028.

[ahrtr]

Restarting all operators/nodes certainly works to start clean - but also means downtime for workload running on the cluster.

Points:

• Usually it is risky for an offline tool to modify the db data, see also cli: Add etcdutl snapshot hashkv command #15965 (comment). So usually it is NOT recommended unless there is no other alternative approaches. For example, we add bbolt surgery to fix/update db file, because the db is corrupted and we have no choice.
• When you restore an etcd cluster, usually it means that you already ran into a disastrous failure. In other words, there is already downtime. But if it's a huge cluster (e.g. with hundreds of or thousands of nodes), it might be inconvenient to restart all operators and nodes (actually kubelet, kube-proxy and CNI?). But if it's just a small cluster with a couple of nodes, it might be fine to just restart them.
• When you restore an etcd cluster, usually it also means that there might be some data loss when the snapshot isn't up to date.
• If you bump the revision, it means etcd may re-notify watchers with identical or stale key/values. This might not be a big problem for identical records. FYI. clientv3: correct nextRev on receiving progress notification response #15248 (comment). But for stale data, it means the snapshot isn't up to date, and there are some data loss....what' the potential impact?

cc @ptabor @tjungblu @cenkalti