NEP-568: Resharding V3 #568
Conversation
flmel
added
the
S-draft/needs-author-revision
|
Hi @wacban – thank you for starting this proposal. As the moderator, I labeled this PR as "Needs author revision" because we assume you are still working on it since you submitted it in "Draft" mode. |
Filling in the future possibilities section - and looking for more ideas :)
Filling the specification section about flat state.
Some beautification courtesy ChatGPT. I double checked everything, we aren't changing any meaning.
walnut-the-cat
added
S-review/needs-sme-review
|
As the moderator, I believe this proposal is ready for SME review. @near/wg-protocol , could you help assign SMEs who can review the proposal? Thank you. |
There was a problem hiding this comment.
As a reviewer I would like to thank the team for putting together such a well-written and detailed proposal! Designing this feature was no easy task, and the proposal here seems to address all the intricacies nicely.
I have left two questions, but otherwise the design looks good to me.
|
|
||
| #### Flat State's Status Persistence | ||
|
|
||
| Every shard's Flat State has a status associated with it and stored in the database, called `FlatStorageStatus`. We propose extending the existing object by adding a new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is persisted to disk, making the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat State creation (see `FlatStorageShardCreator`), reducing code duplication. |
There was a problem hiding this comment.
If the node crashes during resharding then the HybridArena MemTrie objects also need to be reconstructed to resume block processing. How does this recovery happen? I assume it is easy to reconstruct the FrozenArena from the Flat state, but what about the changes on top of it? Are they persisted somewhere? If they are not then I suppose the node would need to recreate the MemTrie by re-applying the chunks that have happened so far in the epoch.
There was a problem hiding this comment.
@Trisfald Can you reply please?
There was a problem hiding this comment.
Yes the HybridArena Memtrie will need to be reconstructed upon crash or even a simple restart.
I think at the moment stage of nearcore/master we don't support seamless node restarts yet. There are a couple possible solution to this problem since all changes to the state are always persisted, in a way or another. In increasing order of complexity:
- Wait for flat storage to complete resharding and load the memtrie for children shards as usual
- Recreate the memtrie of the parent shard as it was at the resharding block, split it again, and apply changes
- Create memtries from current flat storage state of children (even if not complete) and apply remaining changes
- More complex solutions which leverage decoupling of memtrie from flat storage
There was a problem hiding this comment.
Thanks for the clarification @Trisfald . I brought this up here because crash recovery is mentioned as part of the motivation for the design. Maybe it is worth adding a section to the MemTrie specification about how restarts during resharding will be handled.
|
@mfornet , could you be able to review this NEP? |
|
@mfornet , another ping to make sure it catches your eyes. As we are trying to release resharding v3 with upcoming release, it will be great if you could share your thoughts. Thank you. |
# Feature to stabilize This PR stabilizes Resharding V3. It introduces a new implementation for resharding and two new shard layouts for the production networks. # Context - NEP: near/NEPs#568 - Implementation: #11881 # Testing and QA This feature was extensively tested in unit tests, testloop tests and in forknet. # Checklist - [ ] Link to nightly nayduck run https://nayduck.nearone.org/#/run/1142 - [x] Update CHANGELOG.md to include this protocol feature in the `Unreleased` section.
There was a problem hiding this comment.
Hi @wacban, @walnut-the-cat, sorry for the delay reviewing this PR.
I really appreciate all the details layed out in this document, given the complexity of the proposal. The questions I was annotating were answered in the document itself.
The design looks good.
walnut-the-cat
added
S-voting/needs-wg-voting-indication
|
@near/wg-protocol Can you please fully read this NEP and comment in the thread if you are leaning towards approving or rejecting it? Please make sure to include your rationale and any feedback that you have for the author. |
|
As a subject matter expert and working group member I lean towards approving this proposal. You can see more detailed comments from me in my SME review. |
|
As a working group member I lean towards approving this proposal. Thank you for writing such a detailed proposal! |
|
As a working group member I support approving this proposal. |
NEP Status (Updated by NEP Moderators)
Status: Voting
SME reviews:
Protocol Work Group voting indications (❔ | 👍 | 👎 ):