30

I've been trying to scrape some information of this real estate website with selenium. However when I access the website I need to accept cookies to continue. This only happens when the bot accesses the website, not when I do it manually. When I try to find the corresponding element either by xpath or id, as I find it when I inspect the page manually I get following error.

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="uc-btn-accept-banner"]"}

from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC PATH = "/usr/bin/chromedriver" driver = webdriver.Chrome(PATH) driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance") driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

Does anyone know how to get around this? Why can't I find the element?

Below is an image of the accept cookies popup.

enter image description here

This is the HTML corresponding to the button "Keep browsing". The XPATH is as above.

<button aria-label="Keep browsing" id="uc-btn-accept-banner" class="uc-btn-new uc-btn-accept">Keep browsing <span id="uc-optin-timer-display"></span></button>
Improve this question
6
  • Is the popup really an alert? Or is it elements in the DOM that are made visible and positioned withe some combination of JavaScript and CSS? Commented Sep 23, 2020 at 16:25
  • 1
    Interesting point, I think you could be onto something, however when I look for the matching elements by id within my window I can't find them. As a result I thought they must be in a different window. How would I then approach this issue? Commented Sep 23, 2020 at 16:32
  • 1
    Can you show the popup? I opened the site and I didn't get any popups (I waited for more than 1 minute) Commented Sep 23, 2020 at 16:34
  • My suggestion to debug this is to first be sure your selenium driver is not running headless. Then use your favorite debugger to set a breakpoint after your driver.get() call. Now you can inspect the page directly in Chrome to discover how to access the popup. Commented Sep 23, 2020 at 16:53
  • You don't need to switch. What you see is a modal. Just click the button. If you remove the alert = driver.switch_to.alert part, your code works. Commented Sep 23, 2020 at 19:34

1 Answer 1

Reset to default
22

You were very close!

If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears.

The default wait strategy in selenium is just that the page is loaded. That draw delay between page loaded and display appearing is causing your scripts to fail.

You have two good synchronisation options.

1/ Include an implicit wait for your driver. This is done once per script and affects all objects. This waits 10 seconds before throwing any error, or continues when it's ready:

PATH = "/usr/bin/chromedriver" driver = webdriver.Chrome(PATH) driver.implicitly_wait(10) driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance") driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

2/ Do a explicit wait on your object only:

PATH = "/usr/bin/chromedriver" driver = webdriver.Chrome(PATH) driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance") WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="uc-btn-accept-banner"]'))).click()

More info on the wait strategies is here

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.