Waiting for new requests in queue

[Cit1zeN4]

Can the crawler wait for new requests in the queue if I run it with the keepAlive: true parameter and add a message to the queue asynchronously?

const run = async (req: Request, res: Response, next: NextFunction) => {
    try {
        const parts: Part[] = req.body;
        const splited = distributeEvenly(parts, credentials.length);

        await splited.forEach(async (subArray, index) => {
            const queue = await getOrInitQueue(index.toString());
            const requests = getParsingRequests(subArray);
            await queue.addRequests(requests);
        });

        res.status(200).json({ added: parts.length });
    } catch (error) {
        next(error);
    }
};

import { RequestQueueV2 } from 'crawlee';

// Create the request queue that also supports parallelization
let queue: RequestQueueV2;

/**
 * @param {boolean} makeFresh Whether the queue should be cleared before returning it
 * @returns The queue
 */
export async function getOrInitQueue(workerIndex: string, makeFresh = false) {
    if (queue) {
        return queue;
    }

    queue = await RequestQueueV2.open(`parser-${workerIndex}`);

    if (makeFresh) {
        await queue.drop();
        queue = await RequestQueueV2.open(`parser-${workerIndex}`);
    }

    return queue;
}

[janbuchar]

Hi @Cit1zeN4! By default, the BasicCrawler checks for new requests in the queue every 0.5 seconds. You can adjust it like this:

const crawler = new BasicCrawler({
  autoscaledPoolOptions: {
    maybeRunIntervalSecs: 0.1
  }
})

Unfortunately, there is no "passive wait" implemented.

[Cit1zeN4]

I think the problem is related to the getOrInitQueue function. Before creating the crawlers, the queue is initialized, and when I call the getOrInitQueue function again, I get the same queue object. However, when the http request is executed, the let queue: RequestQueueV2; variable is undefined, and the await RequestQueueV2.open(parser-${workerIndex}); function is called again. Can the repeated call of RequestQueueV2.open with the same parameters somehow break the connection between the queue and the specific crawler?

[janbuchar]

No, I don't think that can happen. Did you make sure that both the express app and the workers are looking for storage in the same directory?

[Cit1zeN4]

Yes, everything was working fine until I started implementing the express.js server

[Cit1zeN4]

Looks like I figured out what the problem was. During the experiments, I stopped using a shared request queue and sent requests directly to crawlers. But in this case, I ran into the problem that after adding the first requests to the queue, crawling started, but re-adding the same request did not start a new crawling. The problem turned out to be that these requests were marked as wasAlreadyPresent and were not added to the queue.