How does reCAPTCHA 3 know I'm using Selenium/chromedriver?

reCaptcha

Websites can easily detect the network traffic and identify your program as a BOT. Google have already released 5(five) reCAPTCHA to choose from when creating a new site. While four of them are active and reCAPTCHA v1 being shutdown.

reCAPTCHA versions and types

• reCAPTCHA v3 (verify requests with a score): reCAPTCHA v3 allows you to verify if an interaction is legitimate without any user interaction. It is a pure JavaScript API returning a score, giving you the ability to take action in the context of your site: for instance requiring additional factors of authentication, sending a post to moderation, or throttling bots that may be scraping content.
• reCAPTCHA v2 - "I'm not a robot" Checkbox: The "I'm not a robot" Checkbox requires the user to click a checkbox indicating the user is not a robot. This will either pass the user immediately (with No CAPTCHA) or challenge them to validate whether or not they are human. This is the simplest option to integrate with and only requires two lines of HTML to render the checkbox.

• reCAPTCHA v2 - Invisible reCAPTCHA badge: The invisible reCAPTCHA badge does not require the user to click on a checkbox, instead it is invoked directly when the user clicks on an existing button on your site or can be invoked via a JavaScript API call. The integration requires a JavaScript callback when reCAPTCHA verification is complete. By default only the most suspicious traffic will be prompted to solve a captcha. To alter this behavior edit your site security preference under advanced settings.

• reCAPTCHA v2 - Android: The reCAPTCHA Android library is part of the Google Play services SafetyNet APIs. This library provides native Android APIs that you can integrate directly into an app. You should set up Google Play services in your app and connect to the GoogleApiClient before invoking the reCAPTCHA API. This will either pass the user through immediately (without a CAPTCHA prompt) or challenge them to validate whether they are human.
• reCAPTCHA v1: reCAPTCHA v1 has been shut down since March 2018.

Solution

However there are some generic approaches to avoid getting detected while web-scraping:

• The first and foremost attribute a website can determine your script/program is through your monitor size. So it is recommended not to use the conventional Viewport.
• If you need to send multiple requests to a website keep on changing the User Agent on each request. Here you can find a detailed discussion on Way to change Google Chrome user agent in Selenium?
• To simulate human like behavior you may require to slow down the script execution even beyond WebDriverWait and expected_conditions inducing time.sleep(secs). Here you can find a detailed discussion on How to sleep webdriver in python for milliseconds

Outro

Some food for thought:

• Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
• Unable to use Selenium to automate Chase site login
• Confidence Score of the request using reCAPTCHA v3 API

Selenium and Puppeteer have some browser configurations that is different from a non-automated browser. Also, since some JavaScript functions are injected into browser to manipulate elements, you need to create some override to avoid detections.

There are some good articles explaining some points about Selenium and Puppeteer detection while it runs on a site with detection mechanisms:

Detecting Chrome headless, new techniques - You can use it to write defensive code for your bot.

It is not possible to detect and block Google Chrome headless - it explains in a clear and sound way the differences that JavaScript code can detect between a browser launched by automated software and a real one, and also how to fake it.

GitHub - headless-cat-n-mouse - Example using Puppeteer + Python to avoid detection

I'm one of the authors mentioned in the answers of this thread. As discussed, reCAPTCHA v3, and anti-bot detection in general, tend to rely on browser fingerprinting challenges to detect side effects linked to headless browsers and automated browsers.

There exists different tests to detect Selenium. Since it tends to evolve frequently, I wrote an updated article to explain how (headless) Chrome (even when modified) instrumented with Selenium can be detected as of June 2024.

Testing the presence of the HeadlessChrome substring in the user agent and verifying the value of navigator.webdriver is still helpful against bots that don't modify too much their fingerprint.

Otherwise, there is a new detection techniques that aims to detect CDP automation (Chrome devtool protocol) used by instrumentation frameworks like Selenium.

The new test looks as follows:

var cdpDetected = false; var e = new Error(); Object.defineProperty(e, 'stack', { get() { cdpDetected = true; } }); // This is part of the detection, the console.log shouldn't be removed! console.log(e); if (cdpDetected) { isBot = true; }