Feat: proof aggregation

[zhenfeizhang]

Description

• replace Proof aggregation circuit #523 and dynamic proof aggregation #564
• diff against dynamic aggregation
• update circuit to handle keccak with empty input
• mock prover test with aggregation circuit
• real prover test with aggregation circuit
  • log: super:/home/ubuntu/zhenfei/0227/zkevm-circuits/aggregator/agg_pass_0724.log
• e2e test and integration @ scroll-prover
• update document and spec w.r.t. static aggregation
• re-enable skip first pass
• re-enable feature flags to turn off different modules

Issue Link

[link issue here]

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Contents

• see aggregator/readme

Rationale

[design decisions and extended information]

How Has This Been Tested?

aggregator/test.sh

[zhenfeizhang]

The diff between dynamic aggregation and static aggregation can be found here.

[kunxian-xia]

When the current integration test that do not skip 1st pass succeed, then we need to test the aggregation circuit with the skip_first_pass feature enabled.

[zhenfeizhang]

When the current integration test that do not skip 1st pass succeed, then we need to test the aggregation circuit with the skip_first_pass feature enabled.

Yes. This is left as a todo for now. The code does not compile yet when we skip first pass

[zhenfeizhang]

hmmm some how 47beb9e reintroduced the issue with constraint system after using the new fixed cells method

[lispc]

Will merge it first. Review comments can be addressed in later prs

[lispc]

Oh there are 4 pending issues. I have to manually close them now to merge.....

[kunxian-xia]

It will be more managable if we can collect all bug fix PRs into one large pr.

[kunxian-xia]

I still think that directly using the keccak's input cells and output cells is not a good design.

1. It's very hard to handle the case in which number of input bytes to get batch data hash is not fixed. If you change the constant MAX_AGG_SNARKS to something larger than 10, then you have more cases to handle as the position of the output of batch data hash cells lie in a larger interval. This is not desireable in my opinion.
2. The cells in aggregation circuit are too tightly coupled with keccak sub-circuit. It's hard to audit the code.

Therefore, I have the following suggestion:

1. Allocate 1 advice column for holding the input and output cells for doing keccak;
2. Allocate 1 advice column for accumulating the length of keccak.
3. Allocate 1 selector for lookup into the keccak table.

The layout will be looking like (using the chunk_pi_hash as an example)

adv	len	q_keccak
chain_id[0]	1	0
..	..	0
chain_id[7]	8	0
prev_state_root[0]	9	0
..	..	0
chunk_data_hash[31]	136	0
rlc cells for getting rlc(chunk_pi_hash_preimage) ..	..	0
chunk_pi_hash[0]	1	0
chunk_pi_hash[1]	2	0
..	..	0
chunk_pi_hash[31]	32	0
rlc cells for getting rlc(chunk_pi_hash)	..	0
input_rlc	136	1
output_rlc	..	0

The layout for getting batch_data_hash can be drawn similarly.

[zhenfeizhang]

spend some time to think about this.

The cells in aggregation circuit are too tightly coupled with keccak sub-circuit. It's hard to audit the code.

is there a case where we modify the keccak circuit while not require changes to this pi aggregation circuit? I think not.

So the real complaint here is that the code is hard coded for MAX_AGG_SNARK. This I agree.

OTOH I think the keccak circuit itself is not well designed. I.e., currently, it uses 87 columns and 2^19 rows, i.e., roughly 43 million cells, supporting 1600ish keccak rounds; while we actually only used 25 rounds, or, 87 * 25 * 300 = 652 K cells. If we stick with k=20, then we should be able to fill all those in a single column or two, rather than 87 columns. This will result into 3x in the overall aggregator performance.

(there are also quite a few wasted cells within those 652k cells as per currently layout of the keccak table)

I think a refactor of (semi-static) keccak circuit is of a higher priority than making the above code less hard-coded.

[kunxian-xia]

is there a case where we modify the keccak circuit while not require changes to this pi aggregation circuit? I think not.

But the design goal is this aggregation circuit should not need any change even if we change the layout of keccak circuit to a more thinner one, right?
For now, if we shrink the keccak circuit's layout from 87 cols to like 29 cols (num_rows_per_round: 12 -> 50), then we need at least refactor the get_indices function which is an example of tight coupling that I worried.

[kunxian-xia]

flag3 is true iff num_of_valid_snarks_cell[0] > 8. The comparison here is buggy because if num_of_valid_snarks_cells[0] = 8, then this not_flag3 = false and flag3 will be true (which should not be the case!).

Therefore, I think we should change the comparison to

let flag3 = rlc_config.is_smaller_than(&mut region, &eight, &num_of_valid_snarks_cell[0], &mut offset)?;
let not_flag3 = rlc_config.not(&mut region, &flag3, &mut offset)?;

[zhenfeizhang]

fixed in #699

flag3 is true is #snarks > 8. the proposed solution does not work as well.
I changed it to

let not_flag3 = rlc_config.is_smaller_than(
    &mut region,
    &num_of_valid_snarks_cell[0],
    &nine,
    &mut offset,
)?;
let flag3 = rlc_config.not(&mut region, &not_flag3, &mut offset)?;

[kunxian-xia]

not_flag3 = (num_snarks < 9) in your proposed solution is equivalent to
flag3 = (num_snarks >= 9) which is equivalent to (8 < num_snarks)? Therefore I think my proposed solution should work.

[zhenfeizhang]

It will be more managable if we can collect all bug fix PRs into one large pr.

here it is: #698