integrate whir into Expander #293

zhenfeizhang · 2025-06-12T22:31:15Z

this PR

integrat WHIR PCS into Expander.
require Hyrax and KZG batch verification #290

future work

Some future work for integration:

original whir only works for Goldilocks and does not have SIMD support. We will need to
- either write Arkwork's FF trait for Goldilcoksx8,
- or fork whir and change its backend to Expander's FF
original whir uses sponge-fish as Fiat-Shamir. we will need to replace it with ours; or find a way to pass our transcript state to sponge-fish

On expander side:

we need to find a good way to batch commit. it is still not clear what is the best way to do it.
- we could use the FRI trick to hash all polynomials into a single one. But we will need to open all polynomials at a same point. this requires modifications to the GKR code
- we could also think about using sumcheck to merge the polynomials as introduced in Hyrax and KZG batch verification #290. how to properly merge non-homomorphic commitments requires some more thoughts

gemini-code-assist

Summary of Changes

Hello @zhenfeizhang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the cryptographic backend options by integrating the WHIR PCS for Goldilocks and introduces a generalized batch opening mechanism for existing PCS like Hyrax and KZG, leveraging a new SumCheck implementation. This enhances proof efficiency and adds flexibility to the commitment layer.

Highlights

WHIR PCS Integration: Integrated the WHIR Polynomial Commitment Scheme (PCS) into the Expander framework, specifically tailored for the Goldilocks field. This adds a new PCS option for circuits using Goldilocks.
Batch Opening Functionality: Introduced generic batch opening capabilities for multiple polynomials evaluated at multiple points. This functionality is implemented for both Hyrax and KZG PCS using a SumCheck-based approach.
Generic SumCheck Protocol: Added a new generic SumCheck protocol implementation for Sum-of-Products polynomials, which serves as a core component for the new multi-point batch opening scheme.
Dependency Updates and Additions: Updated numerous dependencies and added new ones, including the whir crate (from a specific fork), spongefish, spongefish-pow, and several Arkworks crates (ark-ff, ark-ec, etc.) required by WHIR.
KZG Module Refactoring: Restructured the KZG implementation into distinct univariate (uni_kzg) and bivariate (bi_kzg) modules for better organization and clarity.
Field Conversion Implementations: Added From trait implementations for converting between Expander's Goldilocks and GoldilocksExt2 fields and Arkworks' Field64 and Field64_2 types, enabling interoperability with Arkworks-based libraries like WHIR.
Polynomial Trait Enhancements: Enhanced the MultiLinearPoly struct with PartialEq and Eq derives and added methods like fix_bottom_variable and eval_reverse_order, along with Add and Mul trait implementations, supporting the new SumCheck logic.
PCS Trait Extension: Extended the core ExpanderPCS trait to include associated types and methods for batch opening (BatchOpening, multi_points_batch_open, multi_points_batch_verify), standardizing the interface for batch proofs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces significant enhancements by integrating the WHIR polynomial commitment scheme and adding batch opening capabilities to existing schemes like Hyrax and KZG. The KZG codebase has also been refactored for better organization between univariate and bivariate schemes. These are substantial changes that expand the toolkit's cryptographic functionalities.

My review focuses on the correctness of these new integrations and refactorings, particularly the trait modifications and the logic within the PCS implementations. I've identified a critical issue in the KZG verification logic and a few areas where clarity or error handling could be improved, especially concerning MPI support in new batching functions and trait bound changes. The use of git dependencies for whir and spongefish is noted and is acceptable for this stage of development, but should be revisited for stable releases.

zhenfeizhang · 2025-06-16T00:03:38Z

/gemini review

gemini-code-assist

Code Review

This PR integrates the WHIR Polynomial Commitment Scheme (PCS) into the Expander framework. Key changes include adding WhirPCS and its associated types, updating the PolynomialCommitmentScheme trait, and modifying dependencies. There are critical issues with the serialization of WhirCommitment and several unwrap() calls in the WhirPCS implementation that need to be addressed for robustness and correctness.

gemini-code-assist · 2025-06-16T00:05:28Z

poly_commit/src/traits.rs

+    type Params: Clone + Debug;
    type Poly: Clone + Debug + Default;
    type EvalPoint: Clone + Debug + Default;
-    type ScratchPad: Clone + Debug + Default + ExpSerde;
+    type ScratchPad: Debug;

    type SRS: Clone + Debug + Default + ExpSerde + StructuredReferenceString;
-    type Commitment: Clone + Debug + Default + ExpSerde;
+    type Commitment: ExpSerde;
    type Opening: Clone + Debug + Default + ExpSerde;


The Commitment: ExpSerde bound conflicts with the ExpSerde implementation for WhirPCS::Commitment in serdes/src/serdes.rs. The serialize_into for Witness is a no-op, and deserialize_from is unimplemented!(). Address this conflict by either implementing proper serialization/deserialization or removing the ExpSerde bound.

gemini-code-assist · 2025-06-16T00:05:29Z

serdes/src/serdes.rs

+impl ExpSerde for Witness<Field64_2, KeccakMerkleTreeParams<Field64_2>> {
+    fn serialize_into<W: Write>(&self, mut _writer: W) -> SerdeResult<()> {
+        Ok(())
+    }
+    fn deserialize_from<R: Read>(mut _reader: R) -> SerdeResult<Self> {
+        unimplemented!("Witness deserialization is not implemented for witness");
+    }


The ExpSerde implementation for Witness is non-functional. serialize_into is a no-op, and deserialize_from is unimplemented!(). Implement proper serialization/deserialization or remove the ExpSerde bound.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/whir.rs

+
+        let committer = CommitmentWriter::new(params.clone());
+
+        committer.commit(prover_state, whir_poly.clone()).unwrap()


Handle the Result from committer.commit to prevent panics. Propagate the error or implement error recovery logic.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/whir.rs

+        prover
+            .prove(prover_state, statement.clone(), witness)
+            .unwrap();


Handle the Result from prover.prove to prevent panics. Propagate the error or implement error recovery logic.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/whir.rs

+        let parsed_commitment = commitment_reader
+            .parse_commitment(&mut verifier_state)
+            .unwrap();


Handle the Result from commitment_reader.parse_commitment to prevent panics. Return false from verify if parsing fails.

gemini-code-assist · 2025-06-16T00:05:29Z

Cargo.toml

-derivative = "2.2.0"
-transpose = "0.2.3"
+# whir = { git = "https://github.com/WizardOfMenlo/whir", rev = "9d7a97b169542f8d0787db482ee58f7b9b660848" }
+ whir = { git = "https://github.com/zhenfeizhang/whir" }


Pinning the whir dependency to a specific commit hash or tag improves reproducibility and prevents unexpected changes from upstream.

gemini-code-assist · 2025-06-16T00:05:29Z

arith/goldilocks/src/goldilocks.rs

+impl From<Goldilocks> for Field64 {
+    #[inline(always)]
+    fn from(x: Goldilocks) -> Self {
+        Field64::from_bigint(BigInteger64::new([x.v])).unwrap()


Clarify the assumption that x.v is always a canonical Goldilocks field element to ensure from_bigint doesn't fail.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/whir.rs

+    type Opening = Vec<u8>;
+
+    fn init_scratch_pad(params: &Self::Params) -> Self::ScratchPad {
+        // todo: session identifier can be sampled from transcript?


Consider sampling the session identifier from the transcript to enhance security against cross-protocol or replay attacks.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/whir.rs

+        prover_state: &mut Self::ScratchPad,
+        _transcript: &mut impl gkr_engine::Transcript,
+    ) -> (GoldilocksExt2, Self::Opening) {
+        // todo: avoid cloning commitment if possible


Avoid cloning the commitment if possible, as it could impact performance. Explore alternative API structures or optimizations in whir.

gemini-code-assist · 2025-06-16T00:05:29Z

poly_commit/src/traits.rs

    fn open(
        params: &Self::Params,
+        commitment: &Self::Commitment,
        proving_key: &<Self::SRS as StructuredReferenceString>::PKey,
        poly: &Self::Poly,
        x: &Self::EvalPoint,
-        scratch_pad: &Self::ScratchPad,
+        scratch_pad: &mut Self::ScratchPad,
        transcript: &mut impl Transcript,
    ) -> (F, Self::Opening);


The signature of the open method has changed. Ensure these are intentional and all implementers of PolynomialCommitmentScheme are updated accordingly.

zhenfeizhang added 30 commits May 20, 2025 07:34

simplify benches

306fd69

wip

5447786

wip

598d574

kzg batch verify works

86dbbd5

clean up

f525e60

bring back the tests

6cef0f8

fmt

310683e

better API

5bd3d3e

update gkr verifier

c0b64ed

clippy

22d47da

wip

19fdff9

hyrax batch opening now works properly

65b8778

update bench

330425e

cargo fmt

51474a7

Merge branch 'main' into zz/defer_kzg_pairing_check

4b304fa

Merge branch 'zz/defer_kzg_pairing_check' into zz/hyrax_batch_verify

ab05261

ExpanderPCS api

0278307

impl batch opening for kzg

a3e0063

clean up

7cc57f8

add bench

231844d

fix warning

606c9f6

lint

09fafa3

fix warnings

46ba7dd

refactor pcs trait

7d97254

finally gets sumcheck correct

1cb3117

finished sumcheck

01c0b49

clean up generic sumcheck

4624176

wip

aed30f2

wip

9979816

wip

1e67e25

zhenfeizhang added 22 commits May 29, 2025 08:51

remove pairing accumulator

accad15

clean up

9b2aba8

Update test_kzg.rs

fb3f5d6

refactor hyrax

425eb5d

fix

c38d5b3

fix clippy

9c44e62

refactoring kzg

428567b

wip

c9faa9d

wip

4be7c38

wip

7dde99c

refactor bikzg and unikzg

ebdb4e0

performance tuning

8257bf5

mirco benches

9ec57e5

performance tuning

f72e2eb

performance tuning

a8ede0f

Merge branch 'main' into zz/hyrax_batch_verify

c733eb7

fix after merge

b74ed0f

fix tests

1f0d4e6

batch with variable length polys

f1e9dc7

zz/whir api

22022d8

fix bug with scratch pad

834595f

clean up

9035ab8

gemini-code-assist bot reviewed Jun 12, 2025

View reviewed changes

zhenfeizhang added 4 commits June 14, 2025 12:47

Merge remote-tracking branch 'origin/main' into zz/whir_api

c52e7f9

fix clippy

d3acbf8

clean up

e00e69e

clean up

6361e3f

gemini-code-assist bot reviewed Jun 16, 2025

View reviewed changes


		let committer = CommitmentWriter::new(params.clone());

		committer.commit(prover_state, whir_poly.clone()).unwrap()

integrate whir into Expander #293

Are you sure you want to change the base?

integrate whir into Expander #293

Uh oh!

Conversation

zhenfeizhang commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

this PR

future work

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

zhenfeizhang commented Jun 16, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhenfeizhang commented Jun 12, 2025 •

edited

Loading