[WIP] [Not Ready For Review] Adding poll rate to check how slow partition consuming. #4320
Annotations
9 errors
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseWithoutBufferAfterLeaderTest.java#L1
Wanted but not invoked:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:499)
However, there were exactly 22 interactions with this mock:
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.122307d,
1738967351957L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.069169d,
1738967351958L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
7.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.008296d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.002304d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.08766299999999999d,
1738967351970L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.002094d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.001694d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestio
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseWithoutBufferAfterLeaderTest.java#L1
Wanted but not invoked:
leaderFollowerStoreIngestionTask.reportError(
<any string>,
1,
<Capturing argument>
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.reportError(StoreIngestionTask.java:4353)
However, there were exactly 54 interactions with this mock:
leaderFollowerStoreIngestionTask.subscribePartition(
TestTopic_6141b4200f_4b00ced1_v1-1
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTaskTest.runTest(StoreIngestionTaskTest.java:887)
leaderFollowerStoreIngestionTask.subscribePartition(
TestTopic_6141b4200f_4b00ced1_v1-1,
true
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:644)
leaderFollowerStoreIngestionTask.throwIfNotRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:658)
leaderFollowerStoreIngestionTask.isRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.throwIfNotRunning(StoreIngestionTask.java:605)
leaderFollowerStoreIngestionTask.getIsRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.isRunning(StoreIngestionTask.java:4186)
leaderFollowerStoreIngestionTask.nextSeqNum();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:667)
leaderFollowerStoreIngestionTask.run();
-> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
leaderFollowerStoreIngestionTask.isRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1657)
leaderFollowerStoreIngestionTask.getIsRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.isRunning(StoreIngestionTask.java:4186)
leaderFollowerStoreIngestionTask.updateIngestionRoleIfStoreChanged(
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1659)
leaderFollowerStoreIngestionTask.isHybridMode();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.updateIngestionRoleIfStoreChanged(StoreIngestionTask.java:1575)
leaderFollowerStoreIngestionTask.processConsumerActions(
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1660)
leaderFollowerStoreIngestionTask.processConsumerAction(
KafkaTaskMessage{type=SUBSCRIBE, topicPartition=TestTopic_6141b4200f_4b00ced1_v1-1, attempts=3, sequenceNumber=1, createdTimestampInMs=1738967345937},
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1949)
leaderFollowerStoreIngestionTask.processCommonConsumerAction(
KafkaTaskMessage{type=SUBSCRIBE, topicPartition=TestTopic_6141b4200f_4b00ced1_v1-1, attempts=3, sequenceNumber=1, createdTimestampInMs=1738967345937}
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.processConsumerAction(LeaderFollowerStoreIngestionTask.java:572)
leaderFollowerStoreIngestionTask.reportIfCatchUpVersionTopicOffset(
PCS{replicaId=TestTopic_6141b4200f_4b00ced1_v1-1, hybrid=false, latestProcessedLocalVersionTopicOffset=-1, latestProcessedUpstreamVersionTopicOffset=-1, latestProcessedUpstreamRTOffsetMap={}, latestIgnoredUpstreamRTOffsetMap={}, latestRTOffsetTriedToProduceToVTMap{}, offsetRecord=OffsetRecord{localVersionTopicOffset=-1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=-1, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}, recordTransformerClassHash=null}, errorReported=false, started=false, lagCaughtUp=false, processedRecordSizeSinceLastSync=0, leaderFollowerState=STANDBY}
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processCommonConsumerAction(StoreIngestionTask.java:2199)
leaderFollowerStoreIngestionTask.updateLeaderTopicOnFollower(
PCS{replicaId=TestTopic_6141b4200f_4b00ced1_v1-1, hybrid=false, latestProcessedLocalVersionTopicOffset=-
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseWithoutBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.TooFewActualInvocations:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
Wanted 2 times:
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:499)
But was 1 time:
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2629)
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/repository/NativeMetadataRepositoryTest.java#L133
java.lang.AssertionError: expected [2000.0] but found [1000.0]
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithPWiseWithoutBufferAfterLeaderTest.java#L1
Wanted but not invoked:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:499)
However, there were exactly 22 interactions with this mock:
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.122307d,
1738967351957L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.069169d,
1738967351958L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
7.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.008296d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.002304d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordConsumerRecordsQueuePutLatency(
0.08766299999999999d,
1738967351970L
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1302)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.produceToStoreBufferServiceOrKafka(StoreIngestionTask.java:1307)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordTotalRecordsConsumed();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2621)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.002094d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.checkLongRunningTaskState(LeaderFollowerStoreIngestionTask.java:756)
hostLevelIngestionStats.recordStorageQuotaUsed(
NaNd
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.recordQuotaMetrics(StoreIngestionTask.java:1635)
hostLevelIngestionStats.recordProcessConsumerActionLatency(
0.0d
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1995)
hostLevelIngestionStats.recordCheckLongRunningTasksLatency(
0.001694d
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestio
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseWithoutBufferAfterLeaderTest.java#L1
Wanted but not invoked:
leaderFollowerStoreIngestionTask.reportError(
<any string>,
1,
<Capturing argument>
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.reportError(StoreIngestionTask.java:4353)
However, there were exactly 54 interactions with this mock:
leaderFollowerStoreIngestionTask.subscribePartition(
TestTopic_6141b4200f_4b00ced1_v1-1
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTaskTest.runTest(StoreIngestionTaskTest.java:887)
leaderFollowerStoreIngestionTask.subscribePartition(
TestTopic_6141b4200f_4b00ced1_v1-1,
true
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:644)
leaderFollowerStoreIngestionTask.throwIfNotRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:658)
leaderFollowerStoreIngestionTask.isRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.throwIfNotRunning(StoreIngestionTask.java:605)
leaderFollowerStoreIngestionTask.getIsRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.isRunning(StoreIngestionTask.java:4186)
leaderFollowerStoreIngestionTask.nextSeqNum();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.subscribePartition(StoreIngestionTask.java:667)
leaderFollowerStoreIngestionTask.run();
-> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
leaderFollowerStoreIngestionTask.isRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1657)
leaderFollowerStoreIngestionTask.getIsRunning();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.isRunning(StoreIngestionTask.java:4186)
leaderFollowerStoreIngestionTask.updateIngestionRoleIfStoreChanged(
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1659)
leaderFollowerStoreIngestionTask.isHybridMode();
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.updateIngestionRoleIfStoreChanged(StoreIngestionTask.java:1575)
leaderFollowerStoreIngestionTask.processConsumerActions(
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.run(StoreIngestionTask.java:1660)
leaderFollowerStoreIngestionTask.processConsumerAction(
KafkaTaskMessage{type=SUBSCRIBE, topicPartition=TestTopic_6141b4200f_4b00ced1_v1-1, attempts=3, sequenceNumber=1, createdTimestampInMs=1738967345937},
Mock for Store, hashCode: 842912743
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerActions(StoreIngestionTask.java:1949)
leaderFollowerStoreIngestionTask.processCommonConsumerAction(
KafkaTaskMessage{type=SUBSCRIBE, topicPartition=TestTopic_6141b4200f_4b00ced1_v1-1, attempts=3, sequenceNumber=1, createdTimestampInMs=1738967345937}
);
-> at com.linkedin.davinci.kafka.consumer.LeaderFollowerStoreIngestionTask.processConsumerAction(LeaderFollowerStoreIngestionTask.java:572)
leaderFollowerStoreIngestionTask.reportIfCatchUpVersionTopicOffset(
PCS{replicaId=TestTopic_6141b4200f_4b00ced1_v1-1, hybrid=false, latestProcessedLocalVersionTopicOffset=-1, latestProcessedUpstreamVersionTopicOffset=-1, latestProcessedUpstreamRTOffsetMap={}, latestIgnoredUpstreamRTOffsetMap={}, latestRTOffsetTriedToProduceToVTMap{}, offsetRecord=OffsetRecord{localVersionTopicOffset=-1, upstreamOffset=-1, leaderTopic=null, offsetLag=9223372036854775807, eventTimeEpochMs=-1, latestProducerProcessingTimeInMs=0, isEndOfPushReceived=false, databaseInfo={}, realTimeProducerState={}, recordTransformerClassHash=null}, errorReported=false, started=false, lagCaughtUp=false, processedRecordSizeSinceLastSync=0, leaderFollowerState=STANDBY}
);
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processCommonConsumerAction(StoreIngestionTask.java:2199)
leaderFollowerStoreIngestionTask.updateLeaderTopicOnFollower(
PCS{replicaId=TestTopic_6141b4200f_4b00ced1_v1-1, hybrid=false, latestProcessedLocalVersionTopicOffset=-
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/kafka/consumer/SITWithTWiseWithoutBufferAfterLeaderTest.java#L1
org.mockito.exceptions.verification.TooFewActualInvocations:
hostLevelIngestionStats.recordTotalBytesConsumed(
<any long>
);
Wanted 2 times:
-> at com.linkedin.davinci.stats.HostLevelIngestionStats.recordTotalBytesConsumed(HostLevelIngestionStats.java:499)
But was 1 time:
-> at com.linkedin.davinci.kafka.consumer.StoreIngestionTask.processConsumerRecord(StoreIngestionTask.java:2629)
|
|
Publish Test Report:
clients/da-vinci-client/src/test/java/com/linkedin/davinci/repository/NativeMetadataRepositoryTest.java#L133
java.lang.AssertionError: expected [2000.0] but found [1000.0]
|
|
Run Unit Tests with Code Coverage
Process completed with exit code 1.
|
Loading