Select element using XPATH with Python?

Question

I am trying to determine the number of pages of data generated by the Indian Central Pollution Controal Board. Here is an example of output. Following https://github.com/RachitKamdar/Python-Scraper, I used selenium/python

maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

but this produces an empty array. I am really not sure what I am doing wrong. Any help would be greatly appreciated. Thanks

Prophet · Accepted Answer · 2021-06-15 21:14:11Z

You have to add expected condition to wait until the page loaded the data.
You can wait for visibility of element you are using and after that get it's text, like this:

from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC wait = WebDriverWait(driver, 20) wait.until(EC.visibility_of_element_located((By.XPATH, "//*[@id='DataTables_Table_0_paginate']/span/a"))) maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

Rohan · Accepted Answer · 2021-06-16 10:53:36Z

You might want to try getattribute('textContent')

In your case:

maxpage=browser.find_element_by_xpath("(//*[@id='DataTables_Table_0_paginate']/span/a)[last()]").getattribute('textContent')

Collectives™ on Stack Overflow

Select element using XPATH with Python?

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related