Block processing should use storage atomically

[tzdybal]

Currently ApplyBlock, SaveBlock, Commit, SaveBlockResponses and SetHeight are executed as completely separate operations - this potentially leads to inconsistencies in database.
https://github.com/celestiaorg/optimint/blob/b1df4346508c124d59504555f34210773a4e5b50/block/manager.go#L216-L247
https://github.com/celestiaorg/optimint/blob/b1df4346508c124d59504555f34210773a4e5b50/block/manager.go#L392-L433

Ideally all those operations should use single KV batch to ensure atomicity.

Ref: rollkit/rollkit#457, rollkit/rollkit#474

[bd21]

Something like this?

Wanted to check if this was the right approach before refactoring everything

[Manav-Aggarwal]

Talked to @tzdybal about this and he mentioned "this can only be visible when a validator process is killed brutally". Removed va tag.

[tzdybal]

It seems this can also be visible in certain circumstances (high load of node machine and very quick RPC queries) when for example BlockResults is called with height returned by Status.

[Manav-Aggarwal]

A team ran into an issue when using status and block_results endpoint that noticed this issue.

From @tzdybal: We save block information and update height in the code just before saving block responses - and it's not atomic/transactional. because of that status may report a block for which block responses are not yet available.
As this is a race condition, high load and/or low-perf machine/VM makes this more visible.

[tzdybal]

This was not fixed by rollkit/rollkit#1782.