[controller] Add DegradedModeRecoveryService, metrics, and E2E test

[mynameborat]

Problem Statement

After a degraded DC is unmarked, stores with PARTIALLY_ONLINE versions need to be recovered (data re-replicated to the recovering DC). Without automation, operators must manually trigger data recovery for each affected store — tedious and error-prone in large clusters. Additionally, there is no observability into degraded mode operations (push auto-conversions, incremental push blocks, recovery progress, DC degradation duration).

This is part of the degraded-mode batch push feature (PRs #2681, #2741, #2732 already merged).

Solution

DegradedModeRecoveryService

• 2-phase recovery orchestrator: Phase 1 initiates data recovery for all PARTIALLY_ONLINE stores in parallel (prepare → poll readiness → initiate). Phase 2 monitors child DC completion and transitions version status from PARTIALLY_ONLINE → ONLINE.
• Orphan detection: Periodic monitor detects PARTIALLY_ONLINE versions with no active recovery (e.g., after controller leader failover) and re-triggers recovery. The recovery flow is idempotent.
• Configurable concurrency: Bounded thread pools for recovery and monitoring, configurable via degraded.mode.recovery.thread.pool.size.
• Retry with exponential backoff: Failed store recoveries are retried up to 3 times.
• Version supersession handling: If a newer version becomes current during recovery polling, the old version is treated as successfully healed.

DegradedModeStats (9 OTel metrics + latency histogram)

• recovery.store_success_count / recovery.store_failure_count
• recovery.version_transitioned_count
• recovery.progress (gauge, 0.0-1.0)
• push.auto_converted_count / push.blocked_incremental_count
• dc.active_count / dc.duration_minutes
• recovery.store_duration_ms (avg/max)

REST API

• GET /get_recovery_progress?cluster=X&datacenter_name=Y — returns recovery status, progress fraction, store counts.

E2E Integration Test

• Full lifecycle: enable degraded mode → mark DC → batch push (auto-converted) → verify data in healthy DCs only → unmark DC → verify recovery.
• Incremental push blocking test.
• Idempotent double-unmark test.
• Feature-disabled rejection test.

Code changes

• Added new code behind a config: degraded.mode.auto.recovery.enabled (default: false), degraded.mode.recovery.thread.pool.size (default: 5)
• Introduced new log lines.
  • Confirmed logs are not in hot paths; recovery is infrequent.

Concurrency-Specific Checks

• Code has no race conditions — activeRecoveries uses ConcurrentHashMap.compute() for atomic check-and-replace.
• Proper synchronization mechanisms — RecoveryProgress uses AtomicInteger and volatile boolean for thread-safe progress tracking. initiatedStores uses Collections.synchronizedList.
• No blocking calls inside critical sections — recovery runs on dedicated daemon thread pools.
• Verified thread-safe collections — VeniceConcurrentHashMap for active recoveries, Collections.synchronizedList for initiated stores.
• Validated proper exception handling — all thread pool tasks catch exceptions to avoid silent termination.

How was this PR tested?

• New unit tests added: TestDegradedModeRecoveryService (17 tests covering happy path, retries, failures, orphan detection, version supersession, leader failover, re-trigger after completion)
• New integration tests added: DegradedModeBatchPushTest (4 E2E tests)
• New OTel metrics tests: DegradedModeStatsOtelTest (12 tests)
• Modified or extended existing tests: existing auto-convert and skipConsumption tests still pass.

Does this PR introduce any user-facing or breaking changes?

• No. Recovery service is opt-in (degraded.mode.auto.recovery.enabled=false by default). New REST endpoint is additive. Metrics are new counters/gauges.

[misyel]

Code review

Found 1 issue:

1. unmarkDatacenterDegraded triggers auto-recovery and updates the active-DC metric even when the underlying call was a no-op. VeniceHelixAdmin.unmarkDatacenterDegraded early-returns with LOGGER.warn(\"...nothing to unmark\") when the DC is not currently in the degraded set, but the parent wrapper unconditionally records the metric and calls degradedModeRecoveryService.triggerRecovery(...) afterward. Calling unmark on an already-non-degraded DC will fire a phantom recovery cycle. Suggest gating both the metric record and the trigger on a real state transition (e.g. precheck getDegradedDcStates(clusterName).isDatacenterDegraded(datacenterName) or have the inner call return a boolean).

venice/services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

Lines 3570 to 3594 in 42cb92c

	degradedModeStats
	.recordDegradedDcActiveCount(getDegradedDcStates(clusterName).getDegradedDatacenterNames().size());
	}
	// Trigger auto-recovery if enabled
	VeniceControllerClusterConfig config = multiClusterConfigs.getControllerConfig(clusterName);
	if (config.isDegradedModeAutoRecoveryEnabled()) {
	LOGGER.info(
	"Auto-recovery enabled. Triggering recovery for datacenter: {} in cluster: {}",
	datacenterName,
	clusterName);
	degradedModeRecoveryService.triggerRecovery(clusterName, datacenterName);
	} else {
	LOGGER.info(
	"Auto-recovery disabled for cluster: {}. Skipping recovery for datacenter: {}",
	clusterName,
	datacenterName);
	}
	}
	@Override
	public boolean isDegradedModeEnabled(String clusterName) {
	return getVeniceHelixAdmin().isDegradedModeEnabled(clusterName);
	}
	@Override

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[Copilot]

Pull request overview

Adds automated post–degraded-DC recovery orchestration on the parent controller, plus new degraded-mode observability (OTel/Tehuti metrics + a recovery-progress REST endpoint) and expanded test coverage (unit + E2E).

Changes:

• Introduces DegradedModeRecoveryService + StoreRecoveryExecutor + DegradedDcMonitor to trigger and monitor recovery of PARTIALLY_ONLINE versions after a DC is unmarked.
• Adds DegradedModeStats (OTel + Tehuti) and wires new route GET /get_recovery_progress.
• Migrates degraded DC state storage from DegradedDcStates ZK node to LiveClusterConfig (degraded.datacenters), updating controller logic and tests accordingly.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java	Updates parent-admin tests for new degraded-DC API + targeted-region intersection behavior.
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestDegradedModeRecoveryService.java	New unit tests for recovery orchestration + orphan detection + retries.
services/venice-controller/src/test/java/com/linkedin/venice/controller/stats/DegradedModeStatsOtelTest.java	New OTel metrics tests for degraded-mode stats.
services/venice-controller/src/test/java/com/linkedin/venice/controller/server/CreateVersionTest.java	Updates degradedDatacenters response population tests for new API.
services/venice-controller/src/test/java/com/linkedin/venice/controller/server/ClusterRoutesTest.java	Updates degraded-DC routes test for new API response type.
services/venice-controller/src/test/java/com/linkedin/venice/controller/kafka/consumer/AdminExecutionTaskTest.java	Updates skipConsumption tests to use AddVersion’s embedded degradedDatacenters.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java	Wires metrics, recovery service trigger on unmark, targeted-region intersection, and AddVersion degradedDatacenters snapshotting.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java	Moves degraded DC state to LiveClusterConfig and adds version-status update API.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java	Adds per-cluster config for auto-recovery enablement and recovery thread pool size.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceController.java	Registers degraded-mode metric entity for controller service metrics.
services/venice-controller/src/main/java/com/linkedin/venice/controller/UserSystemStoreLifeCycleHelper.java	Updates addVersion call signature with new isDegradedPush parameter.
services/venice-controller/src/main/java/com/linkedin/venice/controller/StoreRecoveryExecutor.java	New per-store recovery execution + polling/transition logic with retry/backoff and metrics.
services/venice-controller/src/main/java/com/linkedin/venice/controller/stats/DegradedModeStats.java	New degraded-mode metrics (recovery/push/duration/active-count).
services/venice-controller/src/main/java/com/linkedin/venice/controller/server/CreateVersion.java	Updates degradedDatacenters population to use new degradedDatacenters map API.
services/venice-controller/src/main/java/com/linkedin/venice/controller/server/ClusterRoutes.java	Adds GET recovery progress route + updates degraded-DC route to new API.
services/venice-controller/src/main/java/com/linkedin/venice/controller/server/AdminSparkServer.java	Registers GET /get_recovery_progress endpoint.
services/venice-controller/src/main/java/com/linkedin/venice/controller/RecoveryProgress.java	New in-memory recovery progress model shared by service + API response.
services/venice-controller/src/main/java/com/linkedin/venice/controller/kafka/consumer/AdminExecutionTask.java	Uses AddVersion.degradedDatacenters to decide degraded-region skipConsumption locally.
services/venice-controller/src/main/java/com/linkedin/venice/controller/DegradedModeRecoveryService.java	New orchestrator for recovery and phase-2 completion polling + monitor wiring.
services/venice-controller/src/main/java/com/linkedin/venice/controller/DegradedDcMonitor.java	New scheduled monitor for degraded-DC duration metrics + orphan recovery detection.
services/venice-controller/src/main/java/com/linkedin/venice/controller/Admin.java	Adds new Admin APIs: getDegradedDatacenters/getRecoveryProgress/getCurrentVersionInRegion/updateStoreVersionStatus.
internal/venice-test-common/src/integrationTest/java/com/linkedin/venice/endToEnd/DegradedModeBatchPushTest.java	New multi-region E2E tests for degraded-mode lifecycle + incremental push blocking + idempotent unmark.
internal/venice-test-common/src/integrationTest/java/com/linkedin/venice/controller/TestVeniceHelixAdminWithSharedEnvironment.java	Updates addVersion call signature with new isDegradedPush parameter.
internal/venice-common/src/test/java/com/linkedin/venice/meta/TestLiveClusterConfig.java	Adds tests for new degraded.datacenters LiveClusterConfig field + backwards compat deserialization.
internal/venice-common/src/test/java/com/linkedin/venice/meta/TestDegradedDcStates.java	Removes tests for deleted DegradedDcStates model.
internal/venice-common/src/test/java/com/linkedin/venice/helix/TestHelixDegradedDcStatesRepository.java	Removes tests for deleted degraded-DC ZK repositories.
internal/venice-common/src/test/java/com/linkedin/venice/controllerapi/ControllerRouteDimensionTest.java	Adds route dimension coverage for GET_RECOVERY_PROGRESS.
internal/venice-common/src/main/java/com/linkedin/venice/zk/VeniceZkPaths.java	Removes ZK path constant for deleted degraded-DC states node.
internal/venice-common/src/main/java/com/linkedin/venice/serialization/avro/AvroProtocolDefinition.java	Bumps protocol IDs for new AdminOperation and StoreMetaValue schemas.
internal/venice-common/src/main/java/com/linkedin/venice/meta/VersionImpl.java	Implements isDegradedPush field propagation (including clone).
internal/venice-common/src/main/java/com/linkedin/venice/meta/Version.java	Adds isDegradedPush API to Version contract.
internal/venice-common/src/main/java/com/linkedin/venice/meta/ReadOnlyStore.java	Delegates isDegradedPush; disallows mutation.
internal/venice-common/src/main/java/com/linkedin/venice/meta/LiveClusterConfig.java	Adds degraded.datacenters map + helpers (add/remove/isDegraded).
internal/venice-common/src/main/java/com/linkedin/venice/meta/DegradedDcStates.java	Deletes legacy DegradedDcStates model.
internal/venice-common/src/main/java/com/linkedin/venice/meta/DegradedDcInfo.java	Updates documentation for new storage location (LiveClusterConfig).
internal/venice-common/src/main/java/com/linkedin/venice/helix/HelixReadWriteDegradedDcStatesRepository.java	Deletes legacy degraded-DC ZK repository.
internal/venice-common/src/main/java/com/linkedin/venice/helix/HelixReadOnlyDegradedDcStatesRepository.java	Deletes legacy degraded-DC ZK repository.
internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/UpdateClusterConfigQueryParams.java	Adds updateClusterConfig support for degraded.mode.enabled.
internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/RecoveryProgressResponse.java	New controller API response type for recovery progress endpoint.
internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/ControllerRoute.java	Adds GET_RECOVERY_PROGRESS route definition.
internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java	Adds degraded-mode auto-recovery configs.
build.gradle	Removes Avro codegen versionOverrides now that new protocol versions are active.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.