Hello Everyone.
I am currently working on a project whose goal is to create scrapers in a dynamic way and then being able to process a huge amount of urls at scale.
For example, I have in DDS two websites: www.xxx.it associated with Scraper IT and www.xxx.ca associated with Scraper CA.
I want to send an infinite amount of urls to DDS, and I want all the urls in the form xxx.it/* to be scraped by Scraper IT, and all the urls in the form xxx.ca/* to be scraped by Scraper CA. I want this to be automatically inferred and done by DDS.
Is there a way to achieve this with the current implementation ?
Hello Everyone.
I am currently working on a project whose goal is to create scrapers in a dynamic way and then being able to process a huge amount of urls at scale.
For example, I have in DDS two websites: www.xxx.it associated with Scraper IT and www.xxx.ca associated with Scraper CA.
I want to send an infinite amount of urls to DDS, and I want all the urls in the form xxx.it/* to be scraped by Scraper IT, and all the urls in the form xxx.ca/* to be scraped by Scraper CA. I want this to be automatically inferred and done by DDS.
Is there a way to achieve this with the current implementation ?