Skip to content

Fix unit metric reporting #403

Fix unit metric reporting

Fix unit metric reporting #403

GitHub Actions / IntegrationTests_1050 Test Reports succeeded Feb 21, 2025 in 0s

30 passed, 6 failed and 0 skipped

Tests failed

Report Passed Failed Skipped Time
IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestion.xml 5✅ 157s
IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml 6✅ 1❌ 960s
IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml 3✅ 2❌ 868s
IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion.xml 1✅ 43s
internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestion.xml 5✅ 157s
internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml 6✅ 1❌ 960s
internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml 3✅ 2❌ 868s
internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion.xml 1✅ 43s

✅ IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestion.xml

5 tests were completed in 157s with 5 passed, 0 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveIngestion 5✅ 157s

✅ com.linkedin.venice.endToEnd.TestActiveActiveIngestion

✅ testActiveActiveStoreRestart
✅ testBatchPushWithSeparateDrainer
✅ testKIFRepushActiveActiveStore[0](false)
✅ testKIFRepushActiveActiveStore[2](true)
✅ testLeaderLagWithIgnoredData

❌ IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml

7 tests were completed in 960s with 6 passed, 1 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing 6✅ 1❌ 960s

❌ com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing

✅ testActiveActiveStoreRestart
✅ testBatchPushWithSeparateDrainer
✅ testKIFRepushActiveActiveStore[0](false)
❌ testKIFRepushActiveActiveStore[2](true)
	java.lang.AssertionError: java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
✅ testLeaderLagWithIgnoredData
✅ testKIFRepushActiveActiveStore[0](false)
✅ testKIFRepushActiveActiveStore[2](true)

❌ IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml

5 tests were completed in 868s with 3 passed, 2 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush 3✅ 2❌ 868s

❌ com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush

❌ testAAReplicationForIncPush
	org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush() didn't finish within the time-out 180000
❌ testAAReplicationForIncPushWithSeparateRT
	org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPushWithSeparateRT() didn't finish within the time-out 180000
✅ testRealTimeTopicPartitionCountMatchesHybridVersion
✅ testAAReplicationForIncPush
✅ testAAReplicationForIncPushWithSeparateRT

✅ IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion.xml

1 tests were completed in 43s with 1 passed, 0 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion 1✅ 43s

✅ com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion

✅ testLeaderShouldCalculateRewindDuringPromotion

✅ internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestion.xml

5 tests were completed in 157s with 5 passed, 0 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveIngestion 5✅ 157s

✅ com.linkedin.venice.endToEnd.TestActiveActiveIngestion

✅ testActiveActiveStoreRestart
✅ testBatchPushWithSeparateDrainer
✅ testKIFRepushActiveActiveStore[0](false)
✅ testKIFRepushActiveActiveStore[2](true)
✅ testLeaderLagWithIgnoredData

❌ internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml

7 tests were completed in 960s with 6 passed, 1 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing 6✅ 1❌ 960s

❌ com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing

✅ testActiveActiveStoreRestart
✅ testBatchPushWithSeparateDrainer
✅ testKIFRepushActiveActiveStore[0](false)
❌ testKIFRepushActiveActiveStore[2](true)
	java.lang.AssertionError: java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
✅ testLeaderLagWithIgnoredData
✅ testKIFRepushActiveActiveStore[0](false)
✅ testKIFRepushActiveActiveStore[2](true)

❌ internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml

5 tests were completed in 868s with 3 passed, 2 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush 3✅ 2❌ 868s

❌ com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush

❌ testAAReplicationForIncPush
	org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush() didn't finish within the time-out 180000
❌ testAAReplicationForIncPushWithSeparateRT
	org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPushWithSeparateRT() didn't finish within the time-out 180000
✅ testRealTimeTopicPartitionCountMatchesHybridVersion
✅ testAAReplicationForIncPush
✅ testAAReplicationForIncPushWithSeparateRT

✅ internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion.xml

1 tests were completed in 43s with 1 passed, 0 failed and 0 skipped.

Test suite Passed Failed Skipped Time
com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion 1✅ 43s

✅ com.linkedin.venice.endToEnd.TestActiveActiveReplicationWithDelayedLeaderPromotion

✅ testLeaderShouldCalculateRewindDuringPromotion

Annotations

Check failure on line 271 in internal/venice-test-common/src/main/java/com/linkedin/venice/utils/TestUtils.java

See this annotation in the file changed.

@github-actions github-actions / IntegrationTests_1050 Test Reports

com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing ► testKIFRepushActiveActiveStore[2](true)

Failed test found in:
  IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml
  internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveIngestionWithReadWriteLeaderCompactionTuningAndParallelProcessing.xml
Error:
  java.lang.AssertionError: java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
Raw output
java.lang.AssertionError: java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
	at com.linkedin.venice.utils.TestUtils.waitForNonDeterministicAssertion(TestUtils.java:271)
	at com.linkedin.venice.utils.TestUtils.waitForNonDeterministicAssertion(TestUtils.java:251)
	at com.linkedin.venice.endToEnd.TestActiveActiveIngestion.testKIFRepushActiveActiveStore(TestActiveActiveIngestion.java:295)
Caused by: java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at com.linkedin.venice.endToEnd.TestActiveActiveIngestion.lambda$testKIFRepushActiveActiveStore$3(TestActiveActiveIngestion.java:296)
	at com.linkedin.venice.utils.TestUtils.waitForNonDeterministicAssertion(TestUtils.java:266)
	at com.linkedin.venice.utils.TestUtils.waitForNonDeterministicAssertion(TestUtils.java:251)
	at com.linkedin.venice.endToEnd.TestActiveActiveIngestion.testKIFRepushActiveActiveStore(TestActiveActiveIngestion.java:295)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.linkedin.venice.client.exceptions.VeniceClientException: java.net.SocketTimeoutException: 1000 MILLISECONDS
	at app//com.linkedin.venice.client.store.transport.HttpTransportClient$HttpTransportClientCallback.failed(HttpTransportClient.java:197)
	at app//org.apache.hc.core5.concurrent.BasicFuture.failed(BasicFuture.java:138)
	at app//org.apache.hc.core5.concurrent.ComplexFuture.failed(ComplexFuture.java:79)
	at app//org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2.failed(InternalAbstractHttpAsyncClient.java:350)
	at app//org.apache.hc.client5.http.impl.async.AsyncRedirectExec$1.failed(AsyncRedirectExec.java:248)
	at app//org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed(AsyncHttpRequestRetryExec.java:197)
	at app//org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed(AsyncProtocolExec.java:295)
	at app//org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed(HttpAsyncMainClientExec.java:131)
	at app//org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed(ClientHttp1StreamHandler.java:285)
	at app//org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.terminate(ClientHttp1StreamDuplexer.java:193)
	at app//org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.shutdownSession(AbstractHttp1StreamDuplexer.java:164)
	at app//org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onException(AbstractHttp1StreamDuplexer.java:404)
	at app//org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onTimeout(AbstractHttp1StreamDuplexer.java:399)
	at app//org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.timeout(AbstractHttp1IOEventHandler.java:82)
	at app//org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.timeout(ClientHttp1IOEventHandler.java:41)
	at app//org.apache.hc.core5.reactor.InternalDataChannel.onTimeout(InternalDataChannel.java:169)
	at app//org.apache.hc.core5.reactor.InternalChannel.checkTimeout(InternalChannel.java:67)
	at app//org.apache.hc.core5.reactor.SingleCoreIOReactor.checkTimeout(SingleCoreIOReactor.java:241)
	at app//org.apache.hc.core5.reactor.SingleCoreIOReactor.validateActiveChannels(SingleCoreIOReactor.java:168)
	at app//org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:130)
	at app//org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86)
	at app//org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	... 1 more
Caused by: java.net.SocketTimeoutException: 1000 MILLISECONDS
	at org.apache.hc.core5.io.SocketTimeoutExceptionFactory.create(SocketTimeoutExceptionFactory.java:50)
	... 11 more

Check failure on line 1511 in internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/ControllerClient.java

See this annotation in the file changed.

@github-actions github-actions / IntegrationTests_1050 Test Reports

com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush ► testAAReplicationForIncPush

Failed test found in:
  IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml
  internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml
Error:
  org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush() didn't finish within the time-out 180000
Raw output
org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush() didn't finish within the time-out 180000
	at [email protected]/java.lang.SecurityManager.getClassContext(Native Method)
	at app//org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil$PrivateSecurityManager.getClassContext(PrivateSecurityManagerStackTraceUtil.java:67)
	at app//org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil.getCurrentStackTrace(PrivateSecurityManagerStackTraceUtil.java:54)
	at app//org.apache.logging.log4j.util.StackLocator.getCurrentStackTrace(StackLocator.java:87)
	at app//org.apache.logging.log4j.util.StackLocatorUtil.getCurrentStackTrace(StackLocatorUtil.java:94)
	at app//org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:111)
	at app//org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:96)
	at app//org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:629)
	at app//org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:63)
	at app//org.apache.logging.log4j.core.layout.PatternLayout$NoFormatPatternSerializer.toSerializable(PatternLayout.java:343)
	at app//org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:241)
	at app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:226)
	at app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:60)
	at app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
	at app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
	at app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
	at app//org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
	at app//org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
	at app//org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
	at app//org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
	at app//org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
	at app//org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
	at app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
	at app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
	at app//org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
	at app//org.apache.logging.log4j.core.Logger.log(Logger.java:161)
	at app//org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
	at app//org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
	at app//org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
	at app//org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2071)
	at app//org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1948)
	at app//org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1477)
	at app//com.linkedin.venice.controllerapi.ControllerClient.request(ControllerClient.java:1511)
	at app//com.linkedin.venice.controllerapi.ControllerClient.request(ControllerClient.java:1468)
	at app//com.linkedin.venice.controllerapi.ControllerClient.createNewStore(ControllerClient.java:558)
	at app//com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush(TestActiveActiveReplicationForIncPush.java:320)
	at app//com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush(TestActiveActiveReplicationForIncPush.java:143)

Check failure on line 18 in internal/venice-client-common/src/main/java/com/linkedin/venice/exceptions/VeniceException.java

See this annotation in the file changed.

@github-actions github-actions / IntegrationTests_1050 Test Reports

com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush ► testAAReplicationForIncPushWithSeparateRT

Failed test found in:
  IntegrationTests_1050-artifacts/internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml
  internal/venice-test-common/build/test-results/integrationTests_1050/TEST-com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.xml
Error:
  org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPushWithSeparateRT() didn't finish within the time-out 180000
Raw output
org.testng.internal.thread.ThreadTimeoutException: Method com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPushWithSeparateRT() didn't finish within the time-out 180000
	at [email protected]/java.lang.Throwable.fillInStackTrace(Native Method)
	at [email protected]/java.lang.Throwable.fillInStackTrace(Throwable.java:798)
	at [email protected]/java.lang.Throwable.<init>(Throwable.java:271)
	at [email protected]/java.lang.Exception.<init>(Exception.java:67)
	at [email protected]/java.lang.RuntimeException.<init>(RuntimeException.java:63)
	at app//com.linkedin.venice.exceptions.VeniceException.<init>(VeniceException.java:18)
	at app//com.linkedin.venice.hadoop.VenicePushJob.pollStatusUntilComplete(VenicePushJob.java:2467)
	at app//com.linkedin.venice.hadoop.VenicePushJob.run(VenicePushJob.java:864)
	at app//com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPush(TestActiveActiveReplicationForIncPush.java:371)
	at app//com.linkedin.venice.endToEnd.TestActiveActiveReplicationForIncPush.testAAReplicationForIncPushWithSeparateRT(TestActiveActiveReplicationForIncPush.java:137)