Proper elastic scaling pipeline for v3 candidates

[eskimor]

Problem

With v3 the block building pipeline is finally sound for normal async and sync backing. Elastic scaling is not yet — collators today can land a candidate in an earlier relay chain block than the slot budget intended for it.

The pipeline assumes 2s build + 2s validation + 2s network propagation & statement distribution = 6s total per candidate. If a candidate built mid-slot is squeezed into the previous slot's RC block, that budget is violated: validators outside the EU cluster don't have time to receive it and statement-distribute before backing closes. This is the symptom in #12028 / #10921 — non-EU validators see fewer backable candidates than EU authors put in blocks.

The relay chain runtime doesn't enforce this today. parse_ump_signals / check_core_index only check that the candidate's cq_offset has a core assigned for the para — not that the candidate is not getting backed earlier than intended.

Solution (v3 candidates)

Enforce a minimum claim-queue position on the relay chain (runtime + provisioner): a candidate may be backed at its declared position or later, never earlier.

Collator side:

• First 2s of the slot: cq_offset = 1 (grandchild of scheduling parent).
• Rest of the slot: cq_offset = 2 (grand-grandchild).

Picture (3 cores/slot)

A candidate lands in the RC block of slot X iff its 2s-build + 4s-tail ends before slot X starts. Under one block per 2s, only the first-of-slot candidate qualifies for offset 1; the other two cores roll into offset 2. Steady state: 3 candidates per RC block, full ES throughput preserved, fairness restored for distant validators.

Cooperative for now

The minimum is enforced on chain, but the collator-side rule ("offset 1 for first 2s, offset 2 after") is cooperative. We do not try to enforce intra-slot timing on the relay chain — for now.

Reason: we plan to allocate more cores to a para than it actually needs and leave them idle most of the time, keeping them as spare capacity. After a resubmission the para can burn through the spares to catch back up, instead of pausing block production while the unincluded segment buffer drains. Hard timing enforcement on the relay chain would interact badly with that: a candidate using a "spare" core legitimately needs to look like it's coming in early, and a strict time check would reject it.

Misbehavior also has little upside: a candidate that lies and sets cq_offset = 1 when it isn't actually ready will just miss the seal and be re-tried as offset 2 later.

We can revisit strict enforcement later if it ever becomes worthwhile.

Impact on resubmissions

Open: does the resubmission reasoning in #11903 still hold under the new timings? Specifically:

• The naive-resubmission illustrations in Collator Protocol V4 for resubmissions #11903 (comment)
• The naive vs smart comment in Collator Protocol V4 for resubmissions #11903 (comment)

Those drawings assumed candidates could be backed early. With minimum-offset enforcement the situation should improve — the naive path may no longer waste a core in the same way. Needs the diagrams redrawn against 2s build + 4s tail + the minimum-offset rule, then re-evaluate whether the smart-resubmission variant is still worth the extra complexity.

Scope

• Runtime: enforce minimum claim-queue position in check_descriptor_version_and_signals (paras_inherent)
• Provisioner: filter candidates that would violate the minimum
• Collator (cumulus, slot_based/block_builder_task.rs): pick cq_offset per-core based on elapsed-into-slot time, instead of a single value per slot iteration
• Tests: unit + zombienet for the per-core offset choice and the runtime reject path
• Redraw Collator Protocol V4 for resubmissions #11903 illustrations with the new timing; re-evaluate naive vs smart resubmission
• PRDoc

Notes

• v3 only. v1/v2 keep existing semantics: v1/v2 don't have a proper time model and trying to enforce one is breaking.
• Related: CoreSelector wraparound causes some skipped blocks #8893 (claim queue offset jumps for on-demand / interlaced paras)
• Related: statement-distribution: faster statement propagation #12028 (statement-distribution latency) — apart from improving distribution latency, we should also give enough time (we have it).

[sandreim]

It is not possible to change the cq_offset, this is currently enforced to be equal to the RP offset. If wanted to relax that we need to implement #9428

[eskimor]

Not true, see my reasoning there.

[sandreim]

Elastic scaling is not yet — collators today can land a candidate in an earlier relay chain block than the slot budget intended for it.

I don't understand why this is a problem. Why is it important where the candidate lands ? We cannot predict because it as it is async and also bad things can happen, block author is far away and learns too late about it or the first candidate in chain is not backed in time, so nothing is backed. Actually it is better to get anything on chain as fast as possible. Overall the budget remains the same, the number of cores on RC vs the velocity configured in the parachain runtime.

Reason: we plan to allocate more cores to a para than it actually needs and leave them idle most of the time, keeping them as spare capacity. After a resubmission the para can burn through the spares to catch back up, instead of pausing block production while the unincluded segment buffer drains.

For this we could use bulk or we could use on-demand.

[eskimor]

You filed an issue on why this is a problem (one symptom of it). You should not be allowed to go faster than what the protocol was designed to handle. You can optimize statement distribution all you want, but you will never get a candidate to all validators in time across the globe, if it was submitted at the very end of the slot. This is a blockchain and slots are designed with the assumption that within 6s all nodes will have received the last block (or the statement in this case). We are also not letting nodes produce blocks faster than 6s, just because they have seen the block sooner - no they have to wait, so everybody can keep up.

Worth mentioning: We are hitting this issues with empty blocks! Thus I do think optimizing statement-distribution makes sense as it is very likely that with full blocks we could not even match our actual slot times.

[sandreim]

You filed an issue on why this is a problem (one symptom of it). You should not be allowed to go faster than what the protocol was designed to handle.

Yes, but the issue is not that anyone is going faster than what the protocol allows. I haven't dived very deep into the details of how/when it happens, but one issue I see which I believe is related is that it only happens when the prospective parachains is empty and blocks get in really fast, favouring the votes of backers closer to the RCB author. If there are other blocks waiting in prospective parachains (which is the case with empty RCBs), then there is plenty of time for votes to arrive from all backers (even if they are very far away)

This is a blockchain and slots are designed with the assumption that within 6s all nodes will have received the last block (or the statement in this case).

The issue is that this slot is not enough sometimes. But shouldn't the fix be that we make propagation faster ? and not slow down everyone else because a few guys are far away.

[eskimor]

Yes, but the issue is not that anyone is going faster than what the protocol allows.

But that is exactly the issue, otherwise statement distribution would be fast enough. If a slot is not enough, then indeed we have severe optimizations ahead of us, but that's not the problem. The problem is that with elastic scaling we are building candidates throughout a slot. The first one already has less time than 6s (5s because of the offset we have on v2/v1), the next one has only 3s left and the last one 1s (for build + advertise + validate + distribute) ... this is much much less than a slot of 6s.

[sandreim]

otherwise statement distribution would be fast enough.

In my ticket I say that it is not fast enough: if the latency between A and B is 200ms, then B receives the AttestedCandidate after 600ms, and only then it can forward it to next guys. (B receives manifest, A receives request, B receives response).

I would say the protocol favours clustered validators, which is not good, because it works against geo decentralization. Fixing this would be a mix of faster statements and lowering incentives for backing, but it will never be perfect.

[eskimor]

You are talking here about a couple of hundred milliseconds per hop ... slot time is 6s, blocks are empty ... if candidates were not backed too early there should not be any problem. If we really could not make it reliably work within 6s (of course there can be hickups, but it should not be standard expectation), then we would need longer slots. If the slot time is generous enough, clustered validators should not have any benefit.