Skip to content

Commit

Permalink
Merge pull request #99 from crawler-commons/87
Browse files Browse the repository at this point in the history
Fix #87, similar behaviour re-blocking whether specitying a key or not
  • Loading branch information
jnioche authored Sep 14, 2024
2 parents bd4e209 + 6daf49e commit 1d99fab
Showing 1 changed file with 6 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,12 @@ public void getURLs(GetParams request, StreamObserver<URLInfo> responseObserver)
return;
}

// already has its fill of URLs in process
if (queue.getInProcess(now) >= maxURLsPerQueue) {
responseObserver.onCompleted();
return;
}

int totalSent =
sendURLsForQueue(
queue, qwc, maxURLsPerQueue, secsUntilRequestable, now, synchStreamObs);
Expand Down

0 comments on commit 1d99fab

Please sign in to comment.