feat: Initial V2 E2E Prototype #260

ethenotethan · 2025-01-24T16:33:39Z

Fixes Issue

Changes proposed

Introduces a working MVP for integration with EigenDA V2.

Known Issues

Screenshots (Optional)

Note to reviewers

… feature flag

* fix(#251): better error logging for RPC lookup errors against service manager (#254) * chore: modify verifier to not require eth archive node (#241) * chore: force verifier's EthConfirmationDepth to be <64 We panic in the flag's action, as well as in the verifier's constructor when this condition is not respected. This will make sure that an archival node is not required. * chore: modify verifier to load quorum parameters only once at initialization This removes the need for running with an eth archive node. * style: fix minor lint issue * docs: update README to mention that archival node is no longer needed * docs: clean-up README archival node requirement explanation * docs: fix verify/cert.go comment typo Co-authored-by: Ethen <[email protected]> * docs: for eg -> e.g. * style(cert): remove unecessary bound checks from inside loop * style: create consts package with EthHappyPathFinalizationDepthBlocks = 64 * style: change panic into error return * docs: change op reference for eth reference * docs: make flag comment simpler * Update verify/cert.go Co-authored-by: EthenNotEthan <[email protected]> --------- Co-authored-by: Ethen <[email protected]> * docs: pimp out readme with blob lifecycle diagrams (#233) * chore: move pull_request_template.md under .github/ dir * docs: reorder README sections to feel more natural (move flags to bottom) * docs (wip): add blob lifecycle diagrams to README * docs: remove Sidecar from README title (proxy is not necessarily a side) * docs: add blob lifecycle section to README * docs: add TOC to README * style(routing): rename raw_commitment -> payload We changed the name in README so should be consistent in the code * docs: update README sections to use Payload instead of Blob Posting Blobs -> Posting Payloads Retrieving Blobs -> Retrieving Payloads * docs: remove "obviously" word * Required quorums glitch (#255) * Avoid quorum 1 check on range of Holesky blocks * Improve SVC address check * Update verify/verifier.go Co-authored-by: Samuel Laferriere <[email protected]> * Update verify/verifier.go Co-authored-by: Samuel Laferriere <[email protected]> * Avoid unnecessary cast * Rename constant * Fix lint --------- Co-authored-by: Samuel Laferriere <[email protected]> * docs: update README posting payload image (#256) * fix: remove last eth_call that required archive node (#259) Forgot this one in #241 * docs: update SECURITY.md with audit commit + fix small typos (#263) * docs: update SECURITY.md * docs: update SECURITY.md with audited release number + release where findings were addressed * feat: EigenDAV2 commitment processing and generation * feat: EigenDAV2 commitment processing and generation - add note describing follow up todo --------- Co-authored-by: Samuel Laferriere <[email protected]> Co-authored-by: Gaston Ponti <[email protected]>

…sk--feat-init-v2-scaffolds

…bs/eigenda-proxy into epociask--feat-init-v2-scaffolds

…g with G2 point ingestion

…sk--feat-init-v2-scaffolds

samlaf

FIrst pass. Still need to review load_store and eigenda_v2 code

samlaf · 2025-02-07T02:55:37Z

.vscode/launch.json

+                "ENV_PATH": "../../.env",
+                "EIGENDA_PROXY_EIGENDA_CERT_VERIFICATION_DISABLED": "true",
+                "EIGENDA_PROXY_EIGENDA_TARGET_KZG_G1_PATH": "../../resources/g1.point",
+                "EIGENDA_PROXY_EIGENDA_TARGET_KZG_G2_POWER_OF_2_PATH": "../../resources/g2.point.powerOf2"


why ../..? Which folder is this being run from?

cmd/server

This is kind of weird though no? Typically a binary is ran from root, like go run cmd/server? Is this something specific to how vscode launches the binary?

How does this work actually? Does this add a button on the main() function in cmd/server/main.go on the vscode UI?

samlaf · 2025-02-08T21:16:59Z

server/config.go

+	EdaV1ClientConfig clients.EigenDAClientConfig
+	MemstoreConfig    memstore.Config
+	StorageConfig     store.Config
+	VerifierConfig    verify.Config
+	PutRetries        uint

 	MemstoreEnabled bool
+
+	EigenDAV2Enabled  bool
+	V2DispersalConfig clients_v2.PayloadDisperserConfig
+	V2RetrievalConfig clients_v2.RelayPayloadRetrieverConfig


Can you clean up this config a bit?
Like put all the memstore fields together.
Make the eigenda v1 and v2 close to each other.
etc

samlaf · 2025-02-08T21:17:28Z

server/config.go

+	MemstoreConfig    memstore.Config
+	StorageConfig     store.Config
+	VerifierConfig    verify.Config
+	PutRetries        uint


Can we add a comment here? Why is this a standalone config and not part of the other configs?

samlaf · 2025-02-08T21:20:29Z

verify/v1/cli.go

+		// &cli.StringFlag{
+		// 	Name:     G2PathFlagName,
+		// 	Usage:    "Directory path to g2.point file.",
+		// 	EnvVars:  []string{withEnvPrefix(envPrefix, "TARGET_KZG_G2_PATH")},
+		// 	Value:    "resources/g2.point",
+		// 	Category: category,
+		// },


why comment this out?

d684f1f

V2: Fix client side proving bug due to malformed G2 ingestion #272

wrong diff? its pointing to a merge commit.
I'm still confused as to why this is commented out. Is it because you're trying to hide it from clients given that its broken? Is this flag only for v2? Why is it under verify/v1/cli.go then?

Also if you're trying to hide prob best to not comment bu instead literally hide the flag (Hide: true). And also add a comment pointing to that issue and saying that its only hidden/commented because g2 generation is broken, and we should uncomment once its fixed.

verify/v1/cli.go

samlaf · 2025-02-08T22:00:48Z

flags/eigendaflags/cli.go

+	// dependencies migrate to using EigenDA V2 network
+	V2Enabled = withFlagPrefix("v2-enabled")
+
+	// disperser specific flags (interoperable && mutex for (v1 && v2))


what does "interoperable && mutex" mean?

I think v1 and v2 should have completely independent flags. possibly one might want to set a different confirmation timeout for v1 and v2 (which makes sense given that there's no bridging on v2 for eg)

But if we use a different flag namespace (e.g, v2) then we'd have to kill that namespace eventually in the future when we migrate to V2 which would cause eventual deprecation. Nbd but something to take into account

samlaf · 2025-02-08T22:06:20Z

flags/eigendaflags/cli.go

@@ -163,10 +181,96 @@ func CLIFlags(envPrefix, category string) []cli.Flag {
 			EnvVars:  []string{withEnvPrefix(envPrefix, "PUT_RETRIES")},
 			Category: category,
 		},
+	},
+		v2Flags(envPrefix, category)...,


think we should use a separate v2 category

samlaf · 2025-02-08T22:07:01Z

flags/eigendaflags/cli.go

+		},
+		&cli.DurationFlag{
+			Name:     RelayTimeoutName,
+			Usage:    "Timeout used when querying a relay for blob contents.",


Suggested change

Usage: "Timeout used when querying a relay for blob contents.",

Usage: "Timeout used when querying a relay for blobs.",

technically we should change this to payload

Technically what's being retrieved from the relay is the blob, and that's what the timeout applies to. Conversion from a blob to a payload happens after a successful blob retrieval.

We had discussed letting users be ignorant of the concept of a blob, though... so I'm a bit torn on technical correctness vs. simple mental model for users here

samlaf · 2025-02-08T22:15:38Z

flags/eigendaflags/cli.go

+		&cli.DurationFlag{
+			Name:     ContractCallTimeoutName,
+			Usage:    "Timeout used when performing smart contract eth_calls",
+			EnvVars:  []string{withEnvPrefix(envPrefix, "CONTRACT_CALL_TIMEOUT")},


is the only contract call verifyCertDAV2?

no we also call the RelayRegistry during bootstrap to read urls for payload fetching

samlaf · 2025-02-08T22:16:03Z

flags/eigendaflags/cli.go

+		},
+		&cli.UintFlag{
+			Name:     BlobVersionName,
+			Usage:    "Blob version used when dispersing. Currently only supports (0)",


what is a blob version? Is this the encoding version?

no this is the params version that's controlled via EigenDA governance

…bs/eigenda-proxy into epociask--feat-init-v2-scaffolds

…st path

litt3 · 2025-02-10T14:20:30Z

flags/eigendaflags/cli.go

+		},
+		&cli.StringFlag{
+			Name:     CertVerifierAddrName,
+			Usage:    "Address of the EigenDABlobVerifier contract. Required for performing eth_calls to verify EigenDA certificates.",


Usage string is out of date

"EigenDABlobVerifier" -> "EigenDACertVerifier"

litt3 · 2025-02-10T14:28:02Z

flags/eigendaflags/cli.go

+	noPolynomial := ctx.Bool(DisablePointVerificationModeFlagName)
+	polyMode := codecs.PolynomialFormCoeff
+
+	// if point verification mode is disabled then blob is treated as evaluations and


I think you have this reversed.

PolynomialFormEval is to be understood like this: "The encoded payload is interpreted as a polynomial in eval form. Blobs must be in coeff form, therefore we must take the IFFT to turn our encoded payload into the correct form".

And the other side of the coin, PolynomialFormCoeff is to be understood like this: "The encoded payload is interpreted as a polynomial in coeff form. Blobs must be in coeff form, therefore we don't need to do any IFFT encoding, the bytes can be sent as-is"

Originally I had understood the meaning of these terms as you wrote here, but talking with @bxue-l2 and @samlaf, we came to an agreement on the descriptions I wrote above. I'm still not super happy with our terminology here, but a perfect solution has proven elusive.

litt3 · 2025-02-10T14:34:43Z

server/load_store.go

@@ -52,12 +64,103 @@ func populateTargets(targets []string, s3 common.PrecomputedKeyStore, redis *red
 	return stores
 }

+func loadEigenDAV2Store(ctx context.Context, cfg CLIConfig, log logging.Logger) (*eigendav2.Store, error) {
+	// TODO: Replace with real logger once dependency PRs are merged


is this comment still relevant?

litt3 · 2025-02-10T14:36:32Z

server/load_store.go

+	cfg.EigenDAConfig.V2DispersalConfig.SignerPaymentKey = cfg.EigenDAConfig.EdaV1ClientConfig.SignerPrivateKeyHex
+
+	cfg.EigenDAConfig.V2DispersalConfig.BlobCertifiedTimeout = time.Second * 100
+	cfg.EigenDAConfig.V2DispersalConfig.Quorums = []uint8{0, 1, 2}


Why is 2 in this list?

litt3 · 2025-02-10T14:38:11Z

server/load_store.go

+			UseSecureGrpcFlag: !cfg.EigenDAConfig.EdaV1ClientConfig.DisableTLS,
+		},
+		&gethCfg,
+		nil,


Is there a reason you must use a nil prover, or is this just a WIP?

litt3 · 2025-02-10T14:45:28Z

store/generated_key/eigenda_v2/eigenda.go

+
+var _ common.GeneratedKeyStore = (*Store)(nil)
+
+func NewStore(log log.Logger, cfg *Config,


should this use logging.Logger instead of log.Logger?

litt3 · 2025-02-10T14:47:57Z

store/generated_key/eigenda_v2/eigenda.go

+			return e.disperser.SendPayload(ctx, value, salt)
+		},
+		retry.RetryIf(func(err error) bool {
+			salt++ // increment salt before retrying


Should we actually increment salt on failures? My limited understanding was that we'd only need to increment salt if there is a success, and then a reorg happens that requires blob resubmission

litt3 · 2025-02-10T14:50:19Z

store/generated_key/eigenda_v2/eigenda.go

+	return rlp.EncodeToBytes(cert)
+}
+
+// Backend returns the backend type for EigenDA Store


nit: it seems the method was renamed from Backend to BackendType, but the doc wasn't updated

litt3 · 2025-02-10T14:56:24Z

store/manager.go

+		data, err := m.eigendaV2.Get(ctx, key)
+		if err == nil {
+			// verify v2 (payload, cert)
+			err = m.eigendaV2.Verify(ctx, key, data)


I think we ought to have a different order of operations here (since cert verification will likely be faster than retrieval of the full blob, right?):

Verify the cert on chain

If the cert is verified, only then proceed to try to retrieve the blob, from the available backend sources in order of priority

litt3 · 2025-02-10T14:57:33Z

store/generated_key/eigenda_v2/eigenda.go

+// Key is used to recover certificate fields and that verifies blob
+// against commitment to ensure data is valid and non-tampered.
+// TODO: tap into actual verification
+func (e Store) Verify(_ context.Context, _ []byte, _ []byte) error {


I don't think this method signature will need to accept the actual blob bytes. It should just need to verify the certificate on chain, and then the blob bytes will be checked against the certificate inside the eigenDA client code, while doing the fetch

ethenotethan added 4 commits January 24, 2025 23:32

feat: Initial V2 scaffolds

ae6a199

feat: Initial V2 scaffolds - add boilerplate dependency injection and…

6806969

… feature flag

feat: Initial V2 scaffolds - go fmt

f92ba1d

feat: Initial V2 scaffolds - wire up disperser and retriever clients

355fc24

ethenotethan changed the base branch from main to eigenda_v2 January 30, 2025 01:52

ethenotethan changed the title ~~feat: Initial V2 scaffolds~~ feat: Initial V2 E2E Prototype Jan 30, 2025

ethenotethan changed the base branch from eigenda_v2 to main January 30, 2025 05:28

ethenotethan added 10 commits January 30, 2025 12:29

Merge branch 'main' of github.com:Layr-Labs/eigenda-proxy into epocia…

8c52290

…sk--feat-init-v2-scaffolds

Merge branch 'epociask--feat-init-v2-scaffolds' of github.com:Layr-La…

b8a3fd6

…bs/eigenda-proxy into epociask--feat-init-v2-scaffolds

feat: Initial V2 scaffolds - dependency injection working, current bu…

534a29e

…g with G2 point ingestion

feat: Initial V2 scaffolds - E2E proof of concept sort've

165dd85

feat: Initial V2 scaffolds - bump EigenDA

a6970a3

feat: Initial V2 scaffolds - update dependency injection

f211b5d

feat: Initial V2 scaffolds - hardcode quorums

b0d014f

Merge branch 'main' of github.com:Layr-Labs/eigenda-proxy into epocia…

65090b4

…sk--feat-init-v2-scaffolds

feat: Initial V2 scaffolds - working mvp

10a3b7f

feat: Initial V2 scaffolds - working lint and tests

5ea4d48

ethenotethan marked this pull request as ready for review February 6, 2025 19:05

Merge branch 'main' into epociask--feat-init-v2-scaffolds

c47aa1b

ethenotethan changed the base branch from main to eigenda_v2 February 6, 2025 19:07

ethenotethan requested review from samlaf and litt3 February 6, 2025 19:08

samlaf reviewed Feb 8, 2025

View reviewed changes

ethenotethan added 5 commits February 10, 2025 15:24

feat: Initial V2 scaffolds - add todo around G2 ingestion

01c54de

Merge branch 'epociask--feat-init-v2-scaffolds' of github.com:Layr-La…

d684f1f

…bs/eigenda-proxy into epociask--feat-init-v2-scaffolds

feat: Initial V2 scaffolds - change all usage of directory path to ju…

16a50b0

…st path

feat: Initial V2 scaffolds - refactor manager

c68d7f3

feat: Initial V2 scaffolds - refactor commitments

440493e

litt3 reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Initial V2 E2E Prototype #260

feat: Initial V2 E2E Prototype #260

ethenotethan commented Jan 24, 2025 •

edited

Loading

samlaf left a comment

samlaf Feb 7, 2025

ethenotethan Feb 10, 2025

samlaf Feb 10, 2025

samlaf Feb 8, 2025

samlaf Feb 8, 2025

samlaf Feb 8, 2025

ethenotethan Feb 10, 2025

samlaf Feb 10, 2025

samlaf Feb 8, 2025

samlaf Feb 8, 2025

ethenotethan Feb 10, 2025

samlaf Feb 8, 2025

samlaf Feb 8, 2025

ethenotethan Feb 10, 2025

litt3 Feb 10, 2025 •

edited

Loading

samlaf Feb 8, 2025

ethenotethan Feb 10, 2025

samlaf Feb 8, 2025

ethenotethan Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025 •

edited

Loading

litt3 Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025

litt3 Feb 10, 2025 •

edited

Loading

litt3 Feb 10, 2025 •

edited

Loading

	Usage: "Timeout used when querying a relay for blob contents.",
	Usage: "Timeout used when querying a relay for blobs.",


		var _ common.GeneratedKeyStore = (*Store)(nil)

		func NewStore(log log.Logger, cfg *Config,

feat: Initial V2 E2E Prototype #260

Are you sure you want to change the base?

feat: Initial V2 E2E Prototype #260

Conversation

ethenotethan commented Jan 24, 2025 • edited Loading

Fixes Issue

Changes proposed

Known Issues

Screenshots (Optional)

Note to reviewers

samlaf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

litt3 Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

litt3 Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

litt3 Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

litt3 Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

ethenotethan commented Jan 24, 2025 •

edited

Loading

litt3 Feb 10, 2025 •

edited

Loading

litt3 Feb 10, 2025 •

edited

Loading

litt3 Feb 10, 2025 •

edited

Loading

litt3 Feb 10, 2025 •

edited

Loading