Enhance ckb export/import subcommand with range and verifier selection

[eval-exec]

What problem does this PR solve?

Issue Number: close #4921

Problem Summary:

What is changed and how it works?

+ rm -rf /tmp/a
+ rm -rf /tmp/export
+ /tmp/ckb init -C /tmp/a
WARN: Mining feature is disabled because of the lack of the block assembler config options.
Initialized CKB directory in /tmp/a
Create ckb.toml
Create ckb-miner.toml
Create default.db-options
Genesis Hash: 0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5
+ /tmp/ckb export -C /home/exec/box/ckb_data --target /tmp/export --from 1 --to 200
Writing to /tmp/export/ckb-1-200.jsonl
[00:00:00] ##################################################    201/201    done!
+ ls -ahl /tmp/export
total 1.1M
drwxr-xr-x   2 exec users   60 Jul 22 15:05 .
drwxrwxrwt 109 root root  2.7K Jul 22 15:05 ..
-rw-r--r--   1 exec users 1.1M Jul 22 15:05 ckb-1-200.jsonl
+ /tmp/ckb export -C /home/exec/box/ckb_data --target /tmp/export --from 200 --to 300
Writing to /tmp/export/ckb-200-300.jsonl
[00:00:00] ##################################################    301/301    done!
+ ls -ahl /tmp/export
total 3.3M
drwxr-xr-x   2 exec users   80 Jul 22 15:05 .
drwxrwxrwt 109 root root  2.7K Jul 22 15:05 ..
-rw-r--r--   1 exec users 1.1M Jul 22 15:05 ckb-1-200.jsonl
-rw-r--r--   1 exec users 2.3M Jul 22 15:05 ckb-200-300.jsonl
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-200-300.jsonl
Before import, current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)
Import error: Custom { kind: Other, error: "In /tmp/export/ckb-200-300.jsonl, the first block is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff), and its parent (hash: Byte32(0x5c75b4fc749d23238a1999296c27359b28043578af84fe0e11f126b75b1d8e53)) was not found in the database. The current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)." }
+ true
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-1-200.jsonl
Before import, current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)
[00:00:00] ################################################## 1.03MiB/1.03MiB done!
After import, Current tip is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff)
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-200-300.jsonl
Before import, current tip is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff)
[00:00:01] ################################################## 2.22MiB/2.22MiB done!
After import, Current tip is 300-Byte32(0x2461b3c944f041db2fcf414b0368abae64aa7d72365709bfdd02a41f8ca2a8c2)

Related changes

• Support range feature for ckb export
• Support verifier selection for ckb import
• Export to stdout
• Import from stdin
• Todo Benchmark

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code ci-runs-only: [ quick_checks,linters ]

Side effects

• None

Release note

Title Only: Include only the PR title in the release note.

[driftluo]

serde-rs/json#635 Need to limit the number of parallel serde json, its memory usage is very large

[eval-exec]

[New LWP 1345358]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/zdpby3l6azi78sl83cpad2qjpfj25aqx-glibc-2.40-66/lib/libthread_db.so.1".
Core was generated by `./target/ckb import -C /tmp/import /tmp/export/ckb-0-500000.jsonl --skip-all-verify'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f80c9a99d1c in __pthread_kill_implementation () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
[Current thread is 1 (Thread 0x7f80aafff6c0 (LWP 1345362))]
Python Exception <class 'ModuleNotFoundError'>: No module named 'libstdcxx'
(gdb) bt full
#0  0x00007f80c9a99d1c in __pthread_kill_implementation () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
No symbol table info available.
#1  0x00007f80c9a41406 in raise () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
No symbol table info available.
#2  0x00007f80c9a2893a in abort () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
No symbol table info available.
#3  0x00005562906d11fb in rocksdb::port::Mutex::Lock() ()
No symbol table info available.
#4  0x0000556290583e68 in rocksdb::PeriodicTaskScheduler::Unregister(rocksdb::PeriodicTaskType) ()
No symbol table info available.
#5  0x0000556290471bee in rocksdb::DBImpl::CancelAllBackgroundWork(bool) ()
No symbol table info available.
#6  0x0000556290473663 in rocksdb::DBImpl::CloseHelper() ()
No symbol table info available.
#7  0x0000556290474549 in rocksdb::DBImpl::CloseImpl() ()
No symbol table info available.
#8  0x000055629049abf6 in rocksdb::DBImpl::~DBImpl() ()
No symbol table info available.
#9  0x000055629049bbe9 in rocksdb::DBImpl::~DBImpl() ()
No symbol table info available.
#10 0x00005562907a673f in rocksdb::OptimisticTransactionDBImpl::~OptimisticTransactionDBImpl() ()
No symbol table info available.
#11 0x0000556290453552 in rocksdb_optimistictransactiondb_close ()
No symbol table info available.
#12 0x000055628f0d89e3 in core::ptr::drop_in_place<ckb_rocksdb::optimistic_transaction_db::OptimisticTransactionDB> ()
No symbol table info available.
#13 0x000055628f0d8970 in alloc::sync::Arc<T,A>::drop_slow ()
No symbol table info available.
#14 0x000055628f7bce06 in core::ptr::drop_in_place<ckb_store::snapshot::StoreSnapshot> ()
No symbol table info available.
#15 0x000055628f7bccec in alloc::sync::Arc<T,A>::drop_slow ()
No symbol table info available.
#16 0x000055628f7884d1 in core::ptr::drop_in_place<ckb_snapshot::SnapshotMgr> ()
No symbol table info available.
#17 0x000055628f7845d0 in alloc::sync::Arc<T,A>::drop_slow ()
No symbol table info available.
#18 0x000055628efd62d1 in core::ptr::drop_in_place<ckb_shared::shared::Shared> ()
No symbol table info available.
#19 0x000055628efe32b9 in core::ptr::drop_in_place<ckb_chain::verify::ConsumeUnverifiedBlocks> ()
No symbol table info available.
#20 0x000055628efe20f9 in ckb_chain::verify::ConsumeUnverifiedBlocks::start ()
No symbol table info available.
#21 0x000055628efddc55 in std::sys::backtrace::__rust_begin_short_backtrace ()
No symbol table info available.
#22 0x000055628f06ff63 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
No symbol table info available.
#23 0x000055629014df3b in std::sys::pal::unix::thread::Thread::new::thread_start ()
No symbol table info available.
#24 0x00007f80c9a97ea3 in start_thread () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
No symbol table info available.
#25 0x00007f80c9b1bdfc in __clone3 () from /run/current-system/sw/share/nix-ld/lib/libc.so.6
No symbol table info available.
(gdb)