Validator emits Possible compression bomb encountered for ah-next

[skunert]

First reported by @kianenigma on element.

Spawning a zombienet network from these instructions, the relay chain does not back any blocks.

On the validator logs I see:

2025-02-21 13:33:12 Failed to validate candidate para_id=Id(1100) error=Preparation(CouldNotDecompressCodeBlob("Possible compression bomb encountered"))
2025-02-21 13:33:12 Deterministic error occurred during preparation (should have been ruled out by pre-checking phase) para_id=Id(1100) e=CouldNotDecompressCodeBlob("Possible compression bomb encountered")
2025-02-21 13:33:12 Failed to validate and make available candidate_hash=0x688baddd2b0e2cf48a1e8e8a67e3af8e23a66344ab899af9189eb7a2656af0bd error=ValidationFailed(ValidationFailed("prepare: could not decompress code blob: Possible compression bomb encountered")) traceID=138964967204577931892639416271369383822

[kianenigma]

Does the failure in backing cause the parachain to author blocks 1 -> 2 -> 3 -> 1 -> 2 ... in a loop? this is the outcome that I see in the parachian log.

[alexggh]

CouldNotDecompressCodeBlob

Is coming from here:

polkadot-sdk/polkadot/node/core/pvf/prepare-worker/src/lib.rs

Line 307 in 21f6f07

sp_maybe_compressed_blob::decompress(&maybe_compressed_code, VALIDATION_CODE_BOMB_LIMIT)

, and that const seems to be 12MiB:

pub const VALIDATION_CODE_BOMB_LIMIT: usize = (MAX_CODE_SIZE * 4u32) as usize;
pub const MAX_CODE_SIZE: u32 = 3 * 1024 * 1024;

So, is it possible this parachain creates a bigger PvF than that ?

Does the failure in backing cause the parachain to author blocks 1 -> 2 -> 3 -> 1 -> 2 ... in a loop? this is the outcome that I see in the parachian log.

Yes, that's exactly what you are going to see, because with async backing the collator always tries to build ahead 3 blocks(1,2,3), then if those fail to be backed it tries again an so on, until the chain progresses.

[skunert]

Ah yes, @kianenigma your uncompressed PVF was 14mb right?

[kianenigma]

Yes:

[polkadot-sdk-2] ll target/release/wbuild/asset-hub-next-westend-runtime                                                                                                                                                                                                     donal-ah-next-westend  ✭ ✱
Permissions Size User       Date Modified Name
.rw-r--r--  2.7M kianenigma 24 Feb 16:05  asset_hub_next_westend_runtime.compact.compressed.wasm
.rw-r--r--   13M kianenigma 24 Feb 16:06  asset_hub_next_westend_runtime.compact.wasm
.rwxr-xr-x   14M kianenigma 24 Feb 16:05  asset_hub_next_westend_runtime.wasm

But I am using the compressed one to generate the chain spec, which is 2MB. Double checking now

[kianenigma]

The entire chain-spec is:

[polkadot-sdk-2] ll chain_spec.json                                     
Permissions Size User       Date Modified Name
.rw-r--r--  5.4M kianenigma 24 Feb 16:07  chain_spec.json

[kianenigma]

Ah, I see, this line tries to decompress it -- and it will be bigger than the limit.

Short term, I can hack a custom polkadot node with a higher limit.

But if I am not doing anything wrong, and the normal AH runtime will be more than 12MiB, we will have to do a node upgrade before AHM -- CC @seadanda. We would also take this into account for Westend, but we can use the hacky way.

@alexggh thank you very much for the input!

What constraints would we have towards increasing this limit in production?

[kianenigma]

It seems like AssetHub westend for example was close to this limit, but below it:

➜  polkadot-sdk-2 git:(donal-ah-next-westend) ✗ exa -lah target/release/wbuild/asset-hub-westend-runtime
Permissions Size User Date Modified Name
.rw-rw-r--  2.1M kian 24 Feb 16:37  asset_hub_westend_runtime.compact.compressed.wasm
.rw-rw-r--  9.9M kian 24 Feb 16:38  asset_hub_westend_runtime.compact.wasm
.rwxrwxr-x   10M kian 24 Feb 16:37  asset_hub_westend_runtime.wasm
➜  polkadot-sdk-2 git:(donal-ah-next-westend) ✗ exa -lah target/release/wbuild/asset-hub-next-westend-runtime 
Permissions Size User Date Modified Name
.rw-rw-r--  2.7M kian 24 Feb 16:37  asset_hub_next_westend_runtime.compact.compressed.wasm
.rw-rw-r--   13M kian 24 Feb 16:38  asset_hub_next_westend_runtime.compact.wasm
.rwxrwxr-x   14M kian 24 Feb 16:37  asset_hub_next_westend_runtime.wasm

Probably due to adding like a dozen pallets to AH, now it is above the limit.

[seadanda]

, and that const seems to be 12MiB:

What drives this limit? Is it just a safe but conservative limit or is there a calculation/reasoning behind this exact number?

[alexggh]

, and that const seems to be 12MiB:

What drives this limit? Is it just a safe but conservative limit or is there a calculation/reasoning behind this exact number?

I don't know the history of it, we will have into looking if this can be safely increased.

Another, thing to note here is that if we actually do need to increase all validator would probably have to be upgraded before we can use the new value, and that proved to be a really slow process on both kusama and polkadot.

[bkchr]

We should look a little bit into these wasm files and do some optimizations.

twiggy monos reveals the following:

 Apprx. Bloat Bytes │ Apprx. Bloat % │ Bytes   │ %      │ Monomorphizations
────────────────────┼────────────────┼─────────┼────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
             598202 ┊          5.77% ┊  600398 ┊  5.79% ┊ scale_info::build::Variants<F>::variant
                    ┊                ┊    2196 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::h2d1644580bdb5781
                    ┊                ┊    1967 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::h19c99bad4af3b959
                    ┊                ┊    1965 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::h6a8af945d9fed638
                    ┊                ┊    1561 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::h454e77d4e33fd99c
                    ┊                ┊    1561 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::h5544f1896b9c0c1e
                    ┊                ┊    1560 ┊  0.02% ┊     scale_info::build::Variants<F>::variant::hd25a02588ee0492f
                    ┊                ┊    1546 ┊  0.01% ┊     scale_info::build::Variants<F>::variant::h91f10794db9d9339
                    ┊                ┊    1538 ┊  0.01% ┊     scale_info::build::Variants<F>::variant::hbbdedb45661971d3
                    ┊                ┊    1412 ┊  0.01% ┊     scale_info::build::Variants<F>::variant::h3396a4d9fe07df98
                    ┊                ┊    1412 ┊  0.01% ┊     scale_info::build::Variants<F>::variant::hc30b09bad13b34c8
                    ┊                ┊  583680 ┊  5.63% ┊     ... and 1178 more.
             458579 ┊          4.42% ┊  465774 ┊  4.49% ┊ frame_support::storage::transactional::with_transaction
                    ┊                ┊    7195 ┊  0.07% ┊     frame_support::storage::transactional::with_transaction::h69cfacb972f1602c
                    ┊                ┊    6794 ┊  0.07% ┊     frame_support::storage::transactional::with_transaction::h6fde688feab4bf65
                    ┊                ┊    6354 ┊  0.06% ┊     frame_support::storage::transactional::with_transaction::h5e514fb2f822b0aa
                    ┊                ┊    5352 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::h30668d8e6ffb7fc0
                    ┊                ┊    5164 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::hd79fd28f15b269a7
                    ┊                ┊    4954 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::hd91ed371bca5444b
                    ┊                ┊    4819 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::h08096270838cc58a
                    ┊                ┊    4797 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::h54fdcb06e00a7f09
                    ┊                ┊    4749 ┊  0.05% ┊     frame_support::storage::transactional::with_transaction::hb1047d0ec8350126
                    ┊                ┊    4653 ┊  0.04% ┊     frame_support::storage::transactional::with_transaction::h24b57c6cfc71ceb0
                    ┊                ┊  410943 ┊  3.96% ┊     ... and 280 more.
             166624 ┊          1.61% ┊  169502 ┊  1.63% ┊ pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}
                    ┊                ┊    2878 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::hee9e9adb54b0ddad
                    ┊                ┊    2871 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::hd653261a1fcdea23
                    ┊                ┊    2778 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h8c84a8f9abaa3cd0
                    ┊                ┊    2767 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::ha4a9d45b0be62bc2
                    ┊                ┊    2675 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h3abf9ad74d17edee
                    ┊                ┊    2656 ┊  0.03% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h6c8cf2f7aff9b5ed
                    ┊                ┊    2040 ┊  0.02% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h12423a8f56c1ac25
                    ┊                ┊    2040 ┊  0.02% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h574ce141d65eb2be
                    ┊                ┊    1988 ┊  0.02% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h0f8eb6ea327c90bb
                    ┊                ┊    1984 ┊  0.02% ┊     pallet_assets::pallet::_::<impl parity_scale_codec::codec::Decode for pallet_assets::pallet::Call<T,I>>::decode::{{closure}}::h7505c1ed06b3658a
                    ┊                ┊  144825 ┊  1.40% ┊     ... and 206 more.