Skip to content

Fail fast in E2E tests when the CLI exits before an expected prompt - #783

Merged
dkijania merged 2 commits into
mainfrom
fix/e2e-fail-fast-on-missing-prompt
Oct 18, 2026
Merged

dkijania merged 2 commits into
mainfrom
fix/e2e-fail-fast-on-missing-prompt

Conversation

@dkijania

Copy link
Copy Markdown
Contributor

Why

The Nightly E2E tests workflow has hit the 185-minute job timeout on all 8 jobs every night since 2026-05-06. Over the last 30 days that is ~37,700 runner-minutes, or ~152,000 billable-equivalent minutes once the macOS (10x) and Windows (2x) multipliers are applied. No nightly run has passed in the 412 runs GitHub still has on record.

Root cause of the hang (verified from the job logs):

  1. In the should deploy zkApp of generated project test, zk deploy stops early with Failed to find the fee payer's account on chain and exits.
  2. executeInteractiveCommand in tests/utils/cli-utils.ts waits for the Are you sure you want to send (yes/no)? prompt via waitForText from @shimkiv/cli-testing-library. That promise never settles when the text does not appear, so the test sits until the Playwright timeout.
  3. playwright.config.ts sets the per-test timeout to 180 minutes, and the job timeout is 185 minutes, so the job is killed before Playwright reports anything.

What

  • executeInteractiveCommand now waits for a prompt via waitForPrompt, which races the prompt against the CLI process exit (polled every 250 ms) and a 10 minute upper bound. On failure it throws with the captured stdout/stderr.
  • Playwright per-test timeout lowered from 180 to 60 minutes so a stuck test fails and reports before the job is killed.
  • zk deploy now prints the underlying transport/GraphQL error when the fee payer lookup request itself fails, so a failed request can be told apart from a missing account.
  • Changelog entry.

Verification

  • npx tsc --noEmit, prettier and eslint pass; src/lib/deploy.test.js passes (67 tests).
  • The should deploy zkApp of generated project E2E test passes locally against the mocked endpoints with o1js 2.15.0 (the version installed by the project template today), including the interactive prompt path.

Open question

The Failed to find the fee payer's account on chain failure in CI is not reproduced locally with the same o1js version, so its cause is still unknown. It started the day after o1js 2.15.0 was published (2026-05-05), but that correlation is not confirmed. With this PR the nightly will fail in ~15 minutes per job instead of 185 and the new diagnostic line will show the actual lookup error. Until the lookup failure is fixed, consider disabling the nightly schedule or trimming the matrix (node 18 is end of life; the mocked macOS and Windows jobs are the bulk of the cost).

🤖 Generated with Claude Code

dkijania and others added 2 commits October 13, 2026 19:59
`waitForText` from the CLI testing library never settles when the text
does not appear. When `zk deploy` failed early (for example, "Failed to
find the fee payer's account on chain"), the E2E helper waited forever
for the "Are you sure you want to send (yes/no)?" prompt. With the
Playwright per-test timeout at 180 minutes and the nightly job timeout at
185 minutes, every nightly job was killed by the job timeout instead of
reporting a test failure. This has happened on all 8 nightly jobs every
night since 2026-05-06.

- Add `waitForPrompt` to `executeInteractiveCommand`: it races the prompt
  against the CLI process exit (polled) and a 10 minute upper bound, and
  fails with the captured stdout/stderr.
- Lower the Playwright per-test timeout from 180 to 60 minutes so that a
  stuck test fails and reports before the job is killed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`zk deploy` reported "Failed to find the fee payer's account on chain" for
both a missing account and a failed GraphQL request. Print the transport
or GraphQL error when the lookup itself failed, so CI logs can tell the
two apart. Also add the changelog entry for the E2E fail-fast change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.93%. Comparing base (3306054) to head (2f32fd3).

Files with missing lines Patch % Lines
src/lib/deploy.js 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##              main     #783      +/-   ##
===========================================
- Coverage   100.00%   99.93%   -0.07%     
===========================================
  Files           11       11              
  Lines         1492     1494       +2     
  Branches       304      305       +1     
===========================================
+ Hits          1492     1493       +1     
- Misses           0        1       +1     
Flag Coverage Δ
unittests 99.93% <50.00%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dkijania
dkijania merged commit 421cf37 into main Oct 18, 2026
16 of 18 checks passed
@dkijania
dkijania deleted the fix/e2e-fail-fast-on-missing-prompt branch October 18, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants