All I had to do was:
my_options = webdriver.ChromeOptions() my_options.add_argument( '--disable-blink-features=AutomationControlled' ) Some more information to this: This relates to website skyscanner.com. In the past I have been able to scrape it. Yes, it did detect the browser automation and it gave me a captcha to press and hold a button. I used to be able to complete the captcha manually, then search flights and then scrape. But this time around after completing the captcha I get the same captcha again and again, just can't seem to escape from it. I tried some of the most popular suggestions to avoid automation being detected, but they didn't work. Then I found this article which did work, and by process of elimination I found out it only took the option above to get around their browser automation detection. Now I don't even get the captcha and everything else seems to be working normally.
Versions I am running currently:
- OS: Windows 7 64 bit
- Python 3.8.0 (tags/v3.8.0:fa919fd, 2019-10-14) (MSC v.1916 64 bit (AMD64)) on win32
- Browser: Chrome Version 100.0.4896.60 (Official Build) (64-bit)
- Selenium 4.1.3
- ChromeDriver 100.0.4896.60 chromedriver_win32.zip 930ff33ae8babeaa74e0dd1ce1dae7ff
