kvserver: introduce blocking vfs.FS; add some tests #98852
Conversation
This introduces a `vfs.FS` implementation that can be toggled to block on any write activity. Add two tests: 1. check that even though a durable commit is inflight, we can still write non-durably. I.e. at the pebble level there is no immediate reason why we couldn't keep *applying* the raft log even though log *appends* are stuck. 2. end-to-end test where we set up a three node cluster, send unique writes and stall the leaseholder's disk, hoping for the writes to keep going through. This fails, as is expected, because raft does not even put up entries for application if they haven't been durably appended to the leader's log (which they can't be due to the stall we're injecting). If we can get 2) to pass, we can have reason to believe that we can address cockroachdb#88699 (though we'd need to verify with "real" disk stalls and higher write volumes/stall durations and would certainly need to configure pebble to allow much more data to accumulate in memory). The additional benefit test 2) provides is that it prints who is blocked. At the time of writing, it's encouraging - we see just the LogWriter blocked on a call to `Write` (via `flushPending`).o The next step would be to hack in the ability to apply entries that aren't durable on the leader and to see what we get. Touches cockroachdb#88699. Release note: None
|
This change is |
|
cc @jbowens @nvanbenschoten @erikgrinaker since we talked about #88699. |
|
Going to try to hook this up with etcd-io/raft#37 |
Surprised me, but TestWriteWithStuckLeaseholder now passes. Epic: none Release note: None
|
From the above Slack thread:
Jackson then had some responses on where the stalls would be. At this point I'm going to disengage from the prototype, we can pick this back up if we're interested in making progress in earnest. There seems to be reason to believe that with some pebble work we can possibly mask short disk stalls or latency degradations without necessarily exposing them to users. What's unexplored is how "real" disk stalls affect disk reads. We need to be doing all pebble reads out of the cache during evaluation and replication or requests get stuck anyway. |
|
Yeah, maybe we should start a IIRC, there's also a few tests that use errorfs and a custom InjectorFunc to induce blocking. We could also move |
edit: with the third commit in place, the test in 2) now passes.
This introduces a
vfs.FSimplementation that can be toggled to blockon any write activity.
Add two tests:
write non-durably. I.e. at the pebble level there is no immediate
reason why we couldn't keep applying the raft log even though log
appends are stuck.
writes and stall the leaseholder's disk, hoping for the writes to
keep going through. This fails, as is expected, because raft does
not even put up entries for application if they haven't been durably
appended to the leader's log (which they can't be due to the stall
we're injecting).
If we can get 2) to pass, we can have reason to believe that we can
address #88699 (though we'd need to verify with "real" disk stalls
and higher write volumes/stall durations and would certainly need to
configure pebble to allow much more data to accumulate in memory).
The additional benefit test 2) provides is that it prints who is
blocked. At the time of writing, it's encouraging - we see just
the LogWriter blocked on a call to
Write(viaflushPending).The next step would be to hack in the ability to apply entries
that aren't durable on the leader and to see what we get.
Touches #88699.
Release note: None