feat: add retryDelay, retryCondition, and retryStrategy options for enhanced test retry control (fix #8482)

[matanshavit]

Summary

This PR implements enhanced retry options for Vitest tests, addressing feature request #8482. The current retry mechanism only retries a fixed number of times on any kind of error. This enhancement introduces a nested retry configuration object that provides greater flexibility and control over test retry behavior while maintaining full backward compatibility.

New Features

The retry option now accepts either a number (for backward compatibility) or an object with the following properties:

1. count - Number of retry attempts

   • Type: number
   • Default: 0
   • Equivalent to the existing retry number option

2. delay - Delay between retry attempts

   • Type: number (milliseconds)
   • Default: 0
   • Useful for tests that interact with rate-limited APIs or need time to recover between attempts

3. condition - Conditional retry based on error type

   • Type: string | ((error: Error) => boolean)
   • Default: undefined (retry on all errors)
   • String: Treated as regex pattern to match against error message (case-insensitive)
   • Function: Called with error object, return true to retry
   • ⚠️ Function form only works in test files (see important notes below)

4. strategy - When to retry failed tests

   • Type: 'immediate' | 'test-file' | 'deferred'
   • Default: 'immediate'
   • immediate: Retry immediately after failure (current behavior, default)
   • test-file: Defer retries until after all tests in the current file complete
   • deferred: Defer retries until after all test files complete

Usage Examples

// Backward compatibility - number syntax still works
it('simple retry', { retry: 3 }, () => {
  // Retries up to 3 times on any error
})

// New nested syntax - retry with delay (useful for rate-limited APIs)
it('calls API', {
  retry: {
    count: 3,
    delay: 1000
  }
}, async () => {
  await callRateLimitedAPI()
})

// Retry only on specific error types (string regex)
it('handles timeout', {
  retry: {
    count: 5,
    condition: 'TimeoutError'
  }
}, () => {
  // Only retries if error message contains "TimeoutError"
})

// Retry only on specific error types (function)
it('handles timeout', {
  retry: {
    count: 5,
    condition: (error) => error.name === 'TimeoutError'
  }
}, () => {
  // Only retries if error name is exactly "TimeoutError"
})

// Defer retries until end of test file
it('flaky test', {
  retry: {
    count: 3,
    strategy: 'test-file'
  }
}, () => {
  // Retries happen after all other tests in this file complete
})

// Defer retries until all test files complete
it('global flaky test', {
  retry: {
    count: 3,
    strategy: 'deferred'
  }
}, () => {
  // Retries happen after all test files have run
})

// Combine all options
it('complex retry scenario', {
  retry: {
    count: 5,
    delay: 500,
    condition: 'NetworkError',
    strategy: 'test-file'
  }
}, () => {
  // Only retries on NetworkError, with 500ms delay between attempts,
  // and defers retries until end of file
})

// Options can be set on describe blocks
describe('flaky suite', {
  retry: {
    count: 3,
    condition: 'timeout'
  }
}, () => {
  it('test 1', () => { /* inherits retry options */ })
  it('test 2', () => { /* inherits retry options */ })
})

Configuration

All options can be configured globally in vitest.config.ts:

export default defineConfig({
  test: {
    // Simple number syntax
    retry: 3,
    
    // Or nested object syntax
    retry: {
      count: 3,
      delay: 1000,
      condition: 'TimeoutError',  // ⚠️ String only in config (see notes)
      strategy: 'test-file'
    }
  }
})

Important Notes

Function Conditions in Config

⚠️ The function form of retry.condition cannot be used in vitest.config.ts because configurations are serialized when passed to worker threads. If you need conditional retry logic, you have two options:

1. Use a string pattern in your config (works everywhere):

   // vitest.config.ts
   export default defineConfig({
     test: {
       retry: {
         count: 3,
         condition: 'TimeoutError|NetworkError'  // ✅ Works
       }
     }
   })

2. Use a function directly in test files (test-level only):

   // test file
   it('test', {
     retry: {
       count: 3,
       condition: (error) => error.name === 'TimeoutError'  // ✅ Works
     }
   }, () => {
     // ...
   })

Implementation Details

Core Changes

• packages/runner/src/run.ts: Implemented retry logic with delay, condition checking, and strategy handling
  • Helper functions to normalize retry options (number vs object)
  • shouldRetryTest(): Helper function to evaluate retry conditions
  • collectDeferredRetryTests(): Collects tests with deferred retry strategies
  • Modified runTest(): Handles immediate strategy with delay and condition
  • Modified runSuite(): Handles test-file strategy retries
  • Modified runFiles(): Handles deferred strategy retries

Type Definitions

• packages/runner/src/types/tasks.ts: Updated TaskBase and TestOptions to support nested retry structure
• packages/runner/src/types/runner.ts: Updated VitestRunnerConfig to support nested retry structure
• packages/vitest/src/node/types/config.ts: Updated InlineConfig with proper documentation and serialization warnings
• packages/vitest/src/runtime/config.ts: Updated serialized config types

Configuration Support

• packages/runner/src/suite.ts: Option inheritance handles both number and object retry syntax
• packages/vitest/src/node/config/serializeConfig.ts: Config serialization with function detection warning
• packages/vitest/src/node/cli/cli-config.ts: CLI support (basic number syntax only)

Test Coverage

• test/core/test/retry-condition.test.ts: Tests for retry.condition (string regex and function)
• test/core/test/retry-delay.test.ts: Tests for retry.delay timing validation
• test/core/test/retry-strategy.test.ts: Tests for all three retry strategies
• test/core/test/retry-nested-syntax.test.ts: Tests for nested syntax and backward compatibility

Backward Compatibility

✅ Fully backward compatible - The existing retry: number syntax continues to work exactly as before:

• retry: 3 is equivalent to retry: { count: 3 }
• All new options default to their current behavior values:
  • delay defaults to 0 (no delay)
  • condition defaults to undefined (retry on all errors)
  • strategy defaults to 'immediate' (retry immediately)

Existing tests with the retry option will continue to work exactly as before with no changes required.

Test Plan

• ✅ Added comprehensive test coverage for all new features
• ✅ All existing tests pass
• ✅ Verified backward compatibility with number syntax
• ✅ Manual testing with various configurations
• ✅ Type safety verified with TypeScript

Related Issues

Closes #8482

Related (future enhancement): #7834

Comparison with Other Test Runners

This implementation provides similar functionality to:

• Jest: retryTimes(numRetries, { waitBeforeRetry: 100 }) → Our retry.delay
• WebdriverIO: specFileRetriesDelay → Our retry.delay
• WebdriverIO: specFileRetriesDeferred → Our retry.strategy: 'deferred'

Our implementation goes beyond existing solutions by:

• Combining all these features in a single, cohesive API
• Adding conditional retry logic via retry.condition
• Providing a nested structure that allows for future enhancements (e.g., issue Run failed tests in a clean environment on retry #7834)
• Maintaining full backward compatibility with existing syntax

[AriPerkkio]

Should we instead have everyting inside retry? It would allow extending it later, like in #7834.

Suggested change

	retry: number
	retryDelay?: number
	retryCondition?: string | ((error: Error) => boolean)
	retryStrategy?: 'immediate' | 'test-file' | 'deferred'
	retry: number | {
	count?: number
	delay?: number
	condition?: string | ((error: Error) => boolean)
	strategy?: 'immediate' | 'test-file' | 'deferred'
	}

[matanshavit]

Sure, done!

[AriPerkkio]

I think errors check should be done before any check that returns true.

Suggested change

	// No condition means always retry
	if (!condition) {
	return true
	}
	// No errors means test passed, shouldn't get here but handle it
	if (!errors || errors.length === 0) {
	return false
	}
	// No errors means test passed, shouldn't get here but handle it
	if (!errors || errors.length === 0) {
	return false
	}
	// No condition means always retry
	if (!condition) {
	return true
	}

[matanshavit]

Yes, that makes sense

[AriPerkkio]

Checking type narrows the typings too:

Suggested change

	if (task.type === 'test') {
	const test = task as Test
	const retry = test.retry ?? 0
	const retryCount = test.result?.retryCount ?? 0
	const testStrategy = test.retryStrategy || 'immediate'
	if (task.type === 'test') {
	const retry = task.retry ?? 0
	const retryCount = task.result?.retryCount ?? 0
	const testStrategy = task.retryStrategy || 'immediate'

[matanshavit]

Ok, thanks!

[AriPerkkio]

Suggested change

	else if (task.type === 'suite') {
	const subSuite = task as Suite
	for (const child of subSuite.tasks) {
	else if (task.type === 'suite') {
	for (const child of task.tasks) {

[matanshavit]

done

[AriPerkkio]

We cannot pass functions from main thread to test runner threads, as everything needs to be serialized. So the vitest.config.ts cannot support functions in retryCondition.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	1e547af
🔍 Latest deploy log	https://app.netlify.com/projects/vitest-dev/deploys/69013de6eeb03300084d6a72
😎 Deploy Preview	https://deploy-preview-8812--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.