python selenium wait for page to load

Question

I've written a script that gets data from a page, but sometimes the page takes time to load and so when it pull the html into a soup object sometimes It pulls nothing as the page still needs to finish.

I wrote the following code for waiting the page to finish.

def scrape_page(url): browser.get(url) try: WebDriverWait(browser, 10).until(EC.presence_of_element_located(browser.find_element_by_id ("selection-box"))) #Extract Source Code html = browser.page_source; soup = BeautifulSoup(html)

It works

But I'm getting the following error when I call the function;

TypeError: find_element() argument after * must be a sequence, not WebElement

WKPlus · Accepted Answer · 2017-05-11 06:50:56Z

14

I think you should use presence_of_element_located like this:

element = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "myDynamicElement")) )

as described in the manual.

edited May 11, 2017 at 6:50

answered May 20, 2015 at 3:45

WKPlus

7,3352 gold badges38 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Grant McKinnon Over a year ago

I tried this originallly, , but get the error global name By is not defined. Only thing that is different is that Im using Chrome Driver instead of firefox

sobolevn Over a year ago

selenium-python.readthedocs.org/en/latest/…

WKPlus Over a year ago

@GrantMcKinnonG import By before use it: from selenium.webdriver.common.by import By

Learner Over a year ago

@GrantMcKinnon- You have to import as from selenium.webdriver.common.by import By

Andrew Over a year ago

The link in this answer has died. RIP link.

B. Okba · Accepted Answer · 2022-03-13 08:09:36Z

After 6 years from this question, all the solutions I looked for was not good for my situation, I found the solution in other programming language and I did implementation for Python, so this is the best way to wait a page to fully load.

WebDriverWait(self.driver, self.SERVER_TIMEOUT).until( lambda wd: self.driver.execute_script("return document.readyState") == 'complete', "Page taking too long to load" )

Dharman · Accepted Answer · 2021-04-20 14:38:42Z

I apply this func to every WebElement I need to use.

from selenium import webdriver def FindElem(Driver: webdriver, XPath: str, Timeout: int = 300): while Timeout > 0: try: return Driver.find_element_by_xpath(XPath) except: # if element isn't already loaded or doesn't exist time.sleep(1) Timeout -= 1 raise RuntimeError(f"Page loading timeout") # or whatever the hell you want

Usage:

Driver = webdriver.Firefox() webdriver.get("http://somewhere.com/somepage.html") MyWebElement = FindElem(Driver, "//input[@name='email']") # raise exception if timeout

phpjunkie · Accepted Answer · 2024-06-11 12:43:54Z

Load the class for the browser your using and use implicitly_wait. implicitly_wait will wait for whatever element your trying to find. This is built into selenium and works perfectly for finding WebElements on a page that hasn't finished loading.

from datetime import datetime from selenium.webdriver import Firefox from selenium.webdriver.common.by import By from selenium.common.exceptions import NoSuchElementException firefox = Firefox() firefox.implicitly_wait(time_to_wait = 120) # in seconds: 2 minutes firefox.get('https://www.google.com/') timestamp = datetime.now().timestamp() try: firefox.find_element(by = By.LINK_TEXT, value = 'aedgsf') except NoSuchElementException: pass time = datetime.fromtimestamp(datetime.now().timestamp() - timestamp).strftime('%M:%S:%f') print(time)

>>> 02:00:910277

Collectives™ on Stack Overflow

python selenium wait for page to load

4 Answers 4

5 Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

Comments

Comments

Comments

Linked

Related