1

I am using Selenium to open different pages of a site. Have tried multiple times but the browser does not open a second webpage after the initial GET call. Have tried on both Chrome and Safari. Here is my code:

driver = webdriver.Chrome() driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2") driver.set_page_load_timeout(30) driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

Here is the error I get for the second call:

enter image description here

The info from Network logs is Error 504, but I have verified that it works perfectly when done on another window of the browser, without automation

Improve this question
4
  • It looks like the page DOES open it's just opened with an errror from the webserver. Commented Feb 1, 2021 at 15:20
  • Yes, it is a server error, but no more clues to help resolve the same. The same is fine for some other 2 sites Commented Feb 1, 2021 at 15:28
  • I wonder if there is more to the URL action than a simple GET is providing. Are you sure that is the correct and full URL? Can you execute this manually? What does the network tab in the browser say is happening when run manually? Commented Feb 1, 2021 at 15:30
  • The code worked perfectly fine for me. I used Chrome too. Commented Feb 1, 2021 at 16:36

2 Answers 2

Reset to default
3 
options.add_experimental_option( "excludeSwitches", ['enable-automation']) options.add_argument( "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36") options.add_argument("--remote-debugging-port=9222") driver = webdriver.Chrome(options=options) driver.get( "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2") driver.set_page_load_timeout(30) driver.get( "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

the website is detecting automation use above code :)

You can also do this in single line

Just add below argument:

options.add_argument('--disable-blink-features=AutomationControlled')

disabling enable-automation , or disabling automation controller disables webdriver.navigator which few websites uses to detect automation scripts

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

A bit of more information about your usecase would have helped to construct a more canonical answer. However I was able to access the Page 2 of justdial.com/Chennai/Hr-Consultancy-Services with a minimized code block as follows:

  • Code Block:

    from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument("start-maximized") driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")

  • Browser Snapshot:

justdial

But while sending multiple get() one after another:

driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2") driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

It seems ChromeDriver initiated Chrome Browser gets detected and the following error is shown:

An error occurred while processing your request. Reference #97.e5732c31.1612205693.6fd2708

Solution

To avoid the detection you can add the following option:

--disable-blink-features=AutomationControlled

Example

from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_argument('--disable-blink-features=AutomationControlled') driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2") driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")
Improve this answer

1 Comment

it seems that nowadays (2023) use more complex detection and start-maximized and --disabled-blink-features=AutomationControlled alone it's not enough.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.