perf/kv generation plus preprocessed jumpdest

[Oppen]

• add benching util
• edit root cargo.toml
• basic pathbased impl
• improve serialization
• read optimization
• fixes
• start fixing snapsync
• cleanup
• validate node fetches
• fix opening hash
• fix state healing node fetch
• fix merge
• cleanup
• remove contains_X_node APIs
• lint
• heal terminal leaves too
• fix rocksdb test
• save rollback logs
• Revert "save rollback logs"
• re-add full-path leaves
• lint&fix
• basic trie layer implementation
• add passthrough for trie wrapper, for now
• flush batch to cache instead of db
• clippy
• fix insert ordering
• better find the state root
• fix storage handling
• fix insert-after-remove
• revert head to last state
• direct trie and fixes
• fix incorrect miss
• snapsync fixes
• also delete from db
• add healing logic
• healing fixes
• docs: add comment
• refactor: add delete_range function in Store
• Revert "refactor: add delete_range function in Store"
• delete range improvement
• Reapply "refactor: add delete_range function in Store"
• fix extension range delete
• fix: add logs and add fix to storage healing
• fix delete path calculation
• Improved logs for speed
• Update storage_healing.rs
• test: add test for computing subtree ranges
• chore: comment out spammy print
• perf: remove unnecessary clone
• fix: use full path when computing storage range
• chore: remove logs
• chore: remove commented print
• do not return empty value nodes
• fix: remove nodes when writing empty
• refactor: improve InconsistentTree error messages
• move storage healing to db thread
• also delete leaf-related paths
• chore: enable warn-on-block feature in spawned
• fix
• chore: disable warn-on-block again
• fix failed insertion error
• Change vec for btreemap on nodes to write state healing
• avoid costly deletions when not needed
• Add delete range batch
• Add delete ranges log
• fix deleting the wrong nodes
• remove unneeded apply_prefix
• feat: add failed block hash to log
• also free peers in healing
• avoid concurrent writes
• regnerate snapshot from leaves after healing
• rename and update val
• fixes
• disable snapshots
• restore snap cycle size
• remove spammy log
• chore: run cargo fmt
• chore: remove libmdbx
• chore: fix some clippy issues
• chore: add some changes from old branch
• chore: remove unused delete_range functions
• fix: implement eth_getProof again
• fix bug in storage insertion
• re-enable snapshots
• fix: implement new functionality for InMemoryDb
• chore: remove get_node_checked
• chore: UNCOMMENT CONDITION
• chore: fix lints
• fix leaf regen path
• fix: update Trie::get_node
• chore: update archive_sync
• chore: bump max diff-layers to 128
• chore: ignore unused parameter
• chore: fix lint issue and add comment
• Path based merge optimizations (Path based merge optimizations #4825)
• fix import
• split snapshot to separate table
• implement snapshoot
• launch in thread
• fixes
• read empty last_written is missing
• refactor snapshooter into separate thread
• conditionally advance iterators
• add logging
• add debug logging
• add logging and improve locked trie
• empty snapshot table before starting
• fix snapshot adding condition
• block until continue
• fix last_written advance
• wait until write to signal snapshotter
• mark snapshot as complete when done
• cleanup
• fix merge
• remove unwanted changes
• fix duplicated wait
• add docs
• fix fkv-done marker
• rename snapshot->FlatKeyValue
• save last fkv batch
• check done marker
• fmt
• fix tests
• ignore fkv in test
• perf: preprocess code for jump destinations

Motivation

Description

Closes #issue_number

[github-actions]

Lines of code report

Total lines added: 296
Total lines removed: 54
Total lines changed: 350

Detailed view

+------------------------------------------------------------------------+-------+------+
| File                                                                   | Lines | Diff |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/common/trie/db.rs                                        | 89    | -5   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/common/trie/node.rs                                      | 249   | +3   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/common/trie/trie.rs                                      | 929   | +15  |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/common/trie/trie_sorted.rs                               | 440   | +3   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/sync.rs                                   | 1356  | +1   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/api.rs                                           | 241   | +1   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/error.rs                                         | 41    | +2   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs                                         | 1529  | +5   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/in_memory.rs                            | 629   | +3   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/rocksdb.rs                              | 1409  | +171 |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/trie_db/layering.rs                              | 138   | +4   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/trie_db/rocksdb.rs                               | 207   | +42  |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/storage/trie_db/rocksdb_locked.rs                        | 85    | +27  |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/vm/levm/src/call_frame.rs                                | 374   | +19  |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/vm/levm/src/opcode_handlers/stack_memory_storage_flow.rs | 243   | -6   |
+------------------------------------------------------------------------+-------+------+
| ethrex/crates/vm/levm/src/utils.rs                                     | 381   | -43  |
+------------------------------------------------------------------------+-------+------+

[github-actions]

Benchmark for b78b3a1

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.4±0.59ms	35.7±0.95ms	+0.85%
Trie/cita-trie insert 1k	3.5±0.04ms	3.6±0.01ms	+2.86%
Trie/ethrex-trie insert 10k	62.4±0.67ms	66.6±2.99ms	+6.73%
Trie/ethrex-trie insert 1k	8.0±0.10ms	8.3±0.04ms	+3.75%

[github-actions]

Benchmark for e105a19

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.2±0.78ms	35.4±0.91ms	+0.57%
Trie/cita-trie insert 1k	3.5±0.01ms	3.7±0.15ms	+5.71%
Trie/ethrex-trie insert 10k	62.3±0.89ms	65.7±1.17ms	+5.46%
Trie/ethrex-trie insert 1k	8.2±0.20ms	8.3±0.23ms	+1.22%

[github-actions]

Benchmark Results Comparison

No significant difference was registered for any benchmark run.

Detailed Results

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
main_revm_BubbleSort	2.910 ± 0.032	2.882	2.983	1.01 ± 0.01
main_levm_BubbleSort	3.160 ± 0.007	3.152	3.175	1.09 ± 0.01
pr_revm_BubbleSort	2.893 ± 0.016	2.879	2.934	1.00
pr_levm_BubbleSort	3.162 ± 0.009	3.150	3.177	1.09 ± 0.01

Benchmark Results: ERC20Approval

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Approval	943.1 ± 13.4	933.9	974.3	1.00 ± 0.02
main_levm_ERC20Approval	1121.2 ± 10.3	1114.4	1148.6	1.19 ± 0.02
pr_revm_ERC20Approval	941.8 ± 10.7	932.4	964.4	1.00
pr_levm_ERC20Approval	1116.7 ± 2.4	1113.1	1120.6	1.19 ± 0.01

Benchmark Results: ERC20Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Mint	124.5 ± 0.6	123.8	125.9	1.00
main_levm_ERC20Mint	152.5 ± 0.9	151.9	155.0	1.23 ± 0.01
pr_revm_ERC20Mint	125.0 ± 1.8	123.9	129.6	1.00 ± 0.02
pr_levm_ERC20Mint	152.6 ± 0.7	151.8	153.9	1.23 ± 0.01

Benchmark Results: ERC20Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ERC20Transfer	221.3 ± 3.2	219.2	228.4	1.00
main_levm_ERC20Transfer	276.4 ± 3.7	273.8	285.7	1.25 ± 0.02
pr_revm_ERC20Transfer	222.1 ± 2.8	219.2	227.5	1.00 ± 0.02
pr_levm_ERC20Transfer	274.6 ± 0.8	273.8	276.4	1.24 ± 0.02

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Factorial	184.2 ± 1.1	183.2	187.0	1.00 ± 0.01
main_levm_Factorial	209.9 ± 1.4	208.5	213.1	1.14 ± 0.01
pr_revm_Factorial	183.7 ± 0.4	183.0	184.3	1.00
pr_levm_Factorial	208.5 ± 0.5	207.6	209.2	1.14 ± 0.00

Benchmark Results: FactorialRecursive

Command	Mean [s]	Min [s]	Max [s]	Relative
main_revm_FactorialRecursive	1.318 ± 0.023	1.285	1.366	1.02 ± 0.07
main_levm_FactorialRecursive	2.131 ± 0.043	2.089	2.206	1.65 ± 0.12
pr_revm_FactorialRecursive	1.290 ± 0.087	1.128	1.356	1.00
pr_levm_FactorialRecursive	2.110 ± 0.011	2.097	2.128	1.64 ± 0.11

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Fibonacci	155.0 ± 1.0	153.8	157.5	1.00 ± 0.01
main_levm_Fibonacci	192.9 ± 3.8	190.2	201.2	1.25 ± 0.03
pr_revm_Fibonacci	154.5 ± 0.8	153.5	155.9	1.00
pr_levm_Fibonacci	192.8 ± 4.1	190.5	203.1	1.25 ± 0.03

Benchmark Results: FibonacciRecursive

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_FibonacciRecursive	708.9 ± 4.1	703.5	714.7	1.00
main_levm_FibonacciRecursive	963.8 ± 8.9	956.6	984.1	1.36 ± 0.01
pr_revm_FibonacciRecursive	710.0 ± 5.6	701.7	722.1	1.00 ± 0.01
pr_levm_FibonacciRecursive	960.7 ± 5.9	955.4	970.7	1.36 ± 0.01

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_ManyHashes	7.9 ± 0.1	7.8	8.1	1.00
main_levm_ManyHashes	9.4 ± 0.2	9.3	10.1	1.19 ± 0.03
pr_revm_ManyHashes	8.0 ± 0.3	7.8	8.7	1.01 ± 0.03
pr_levm_ManyHashes	9.5 ± 0.1	9.3	9.8	1.20 ± 0.02

Benchmark Results: MstoreBench

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_MstoreBench	250.9 ± 1.6	249.1	253.9	1.00
main_levm_MstoreBench	352.6 ± 2.4	349.9	356.8	1.41 ± 0.01
pr_revm_MstoreBench	251.7 ± 1.5	250.0	254.7	1.00 ± 0.01
pr_levm_MstoreBench	354.9 ± 5.1	350.3	366.0	1.41 ± 0.02

Benchmark Results: Push

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_Push	257.5 ± 0.9	256.6	258.9	1.00
main_levm_Push	378.4 ± 5.5	373.3	388.4	1.47 ± 0.02
pr_revm_Push	257.7 ± 2.5	255.7	263.4	1.00 ± 0.01
pr_levm_Push	384.8 ± 10.2	374.9	411.9	1.49 ± 0.04

Benchmark Results: SstoreBench_no_opt

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
main_revm_SstoreBench_no_opt	145.3 ± 0.5	144.6	146.4	1.71 ± 0.02
main_levm_SstoreBench_no_opt	84.9 ± 1.2	83.4	87.0	1.00
pr_revm_SstoreBench_no_opt	145.2 ± 0.4	144.5	145.8	1.71 ± 0.02
pr_levm_SstoreBench_no_opt	85.7 ± 3.3	83.4	94.9	1.01 ± 0.04

[github-actions]

Benchmark for 43e7a85

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	39.6±2.18ms	35.4±1.19ms	-10.61%
Trie/cita-trie insert 1k	3.5±0.02ms	3.6±0.01ms	+2.86%
Trie/ethrex-trie insert 10k	64.8±1.31ms	65.5±0.89ms	+1.08%
Trie/ethrex-trie insert 1k	8.0±0.04ms	8.7±0.37ms	+8.75%

[github-actions]

Benchmark for edbd4aa

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.5±0.62ms	36.1±1.81ms	+1.69%
Trie/cita-trie insert 1k	3.5±0.02ms	3.5±0.03ms	0.00%
Trie/ethrex-trie insert 10k	62.2±0.52ms	66.4±3.69ms	+6.75%
Trie/ethrex-trie insert 1k	8.1±0.17ms	8.3±0.03ms	+2.47%

[github-actions]

Benchmark for 6d6e27d

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	34.7±0.42ms	35.0±0.49ms	+0.86%
Trie/cita-trie insert 1k	3.5±0.01ms	3.6±0.02ms	+2.86%
Trie/ethrex-trie insert 10k	61.5±0.55ms	65.7±2.72ms	+6.83%
Trie/ethrex-trie insert 1k	7.9±0.02ms	8.4±0.03ms	+6.33%

[github-actions]

Benchmark for 113045e

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	34.3±0.29ms	36.1±0.78ms	+5.25%
Trie/cita-trie insert 1k	3.5±0.01ms	3.6±0.01ms	+2.86%
Trie/ethrex-trie insert 10k	62.6±1.24ms	66.8±1.28ms	+6.71%
Trie/ethrex-trie insert 1k	8.1±0.33ms	8.5±0.32ms	+4.94%

[github-actions]

Benchmark for ef23f46

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.4±1.53ms	35.4±0.74ms	0.00%
Trie/cita-trie insert 1k	3.5±0.01ms	3.6±0.15ms	+2.86%
Trie/ethrex-trie insert 10k	62.3±1.39ms	65.0±1.72ms	+4.33%
Trie/ethrex-trie insert 1k	8.0±0.04ms	8.4±0.06ms	+5.00%

[github-actions]

Benchmark for 8f438e2

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	37.6±0.77ms	37.7±0.93ms	+0.27%
Trie/cita-trie insert 1k	3.6±0.02ms	3.6±0.21ms	0.00%
Trie/ethrex-trie insert 10k	63.6±1.05ms	67.1±0.71ms	+5.50%
Trie/ethrex-trie insert 1k	8.1±0.08ms	8.3±0.07ms	+2.47%

[MegaRedHand]

This was removed in main:

Suggested change

ethrex-trie.workspace = true

[MegaRedHand]

Same

Suggested change

"ethrex-trie 3.0.0",