Testing Non-Test Oriented Websites

[G-markier]

Hi
Is there a way to see what anti-bot checks are being done by a specific website (e.g. Amazon), so as to tailor UC Mode a bit more for such websites while scraping?

Thanks

[mdmintz]

If you run a python requests.get(url) call and the status code starts with a 4, then there's a good change that the website is blocking bots.

In UC Mode, driver.get(url) automatically disconnects chromedriver from Chrome briefly if that is the case.

This was covered in the video:

https://www.youtube.com/watch?v=5dMFI3e85ig

[G-markier]

Thanks for that answer.

QUESTION: If I understand how the disconnect works, it means our code stays locked for the duration of the disconnect - which will add time to scrapes with this disconnect procedure. So if we set reconnect time to 20 seconds, then our code stays locked for 20 seconds. To optimize this and get the minimum reconnection time, especially if we want to call this disconnect for clicks, is there a way to estimate the minimum amount of time to all for the duration?

[mdmintz]

I'm not sure if there's a way to estimate how long of a disconnect you need. It's based on your internet connection speed and if/when the website continues to check for bots after the page has loaded.

[G-markier]

That gives me the idea of setting the reconnect time to be equal to end of page load, plus another bunch of time. For example page_load+5 seconds. Question is, how would we be able to implement such a reconnect time in UC Mode, since the reconnect time requires an integer? Any suggestions?

[mdmintz]

There's no way to guess in advance how long the page will take to load. And the value doesn't need to be an integer.

[G-markier]

So if you created a function, the value could be that function?