Scroll in Selenium Webdriver (Python)

Question

Prerequisites.
You need an account at Instagram to use this script.
Setup a test environment:

Log in, open the needed list(works correctly):

from selenium import webdriver from selenium.webdriver.common.keys import Keys from time import sleep driver = webdriver.Chrome( # driver = webdriver.Firefox( # driver = webdriver.PhantomJS( service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any']) driver.get("https://instagram.com/accounts/login") username = driver.find_element_by_name("username") password = driver.find_element_by_name("password") username1 = 'instagram' # change it! password1 = 'instagrampassword1' # change it! username.send_keys(username1) password.send_keys(password1) submit_button = driver.find_element_by_css_selector( '#react-root > div > article > div > div:nth-child(1) > div > form > span > button') submit_button.click() sleep(2) link = 'https://www.instagram.com/youtube/' driver.get(link) driver.implicitly_wait(2) driver.find_elements_by_class_name("_218yx")[2].click()

Wrong scroll. How to fix this block?

How to focus and scroll correctly on this page?

My attempts:

driver.find_element_by_class_name("_cx1ua").send_keys(Keys.NULL) # focus #The element has been deleted entirely or #The element is no longer attached to the DOM. driver.find_element_by_class_name("_q44m8").send_keys(Keys.NULL) # cannot focus element driver.find_element_by_class_name("_qjr85").send_keys(Keys.NULL) # cannot focus element for i in range(5): driver.find_element_by_class_name("_cx1ua").send_keys(Keys.END)

=============================================================
to @Moshisho :

We need to focus on some element to activate it.
The question is what the element we need to choose to focus and how?
This is not a "body":
something like that, but not this:

background = driver.find_element_by_css_selector("body") # background = driver.find_element_by_css_selector("div._2uju6") for i in range(5): background.send_keys(Keys.SPACE) time.sleep(1)

Without it this command do not work.

to @Naveen :

print(driver.find_element_by_css_selector("div._a1rcs").location_once_scrolled_into_view) # {'x': 0, 'y': 0} print(driver.find_element_by_class_name("_cx1ua").location_once_scrolled_into_view) # {'x': 376, 'y': 229} print(driver.find_element_by_class_name("_q44m8").location_once_scrolled_into_view) # {'x': 376, 'y': 180} print(driver.find_element_by_class_name("_qjr85").location_once_scrolled_into_view) # {'x': 376, 'y': 180}

And what's next?

driver.execute_script("window.scrollTo(0, 3000);") # do not working

Your question should be closed as off-topic. Give some details and description to make your issue more clear... not everyone has account on Instagram :) — Andersson
– Andersson, Commented Dec 25, 2016 at 17:58
Your attempts are for focusing or scrolling a specific element? — Moshisho
– Moshisho, Commented Dec 25, 2016 at 18:40
Yes, I want to scroll all the list of followers, to see > 10 followers per one visit. — lvcpp
– lvcpp, Commented Dec 25, 2016 at 18:42

lvcpp · Accepted Answer · 2016-12-26 15:39:09Z

Try the following code:

from selenium import webdriver from selenium.webdriver.common.keys import Keys from time import sleep from selenium.webdriver.support.ui import Select driver = webdriver.Chrome( # driver = webdriver.Firefox( # driver = webdriver.PhantomJS( service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any']) driver.maximize_window() driver.get("https://instagram.com/accounts/login") username = driver.find_element_by_name("username") password = driver.find_element_by_name("password") username1 = 'instagramlogin1' # change it! password1 = 'instagrampassword1' # change it! username.send_keys(username1) password.send_keys(password1) submit_button = driver.find_element_by_css_selector( '#react-root > div > article > div > div:nth-child(1) > div > form > span > button') submit_button.click() sleep(2) link = 'https://www.instagram.com/youtube/' driver.get(link) driver.implicitly_wait(2) following = driver.find_element_by_xpath("//a[@href='/youtube/following/']/span") total_following = int(following.text) print "total no. of users following: ", total_following # click on 239 following, displays 10 users following.click() loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li") loaded_till_now = len(loaded_following) while(loaded_till_now<total_following): print "following users loaded till now: ", loaded_till_now print loaded_following[loaded_till_now-1] loaded_following[loaded_till_now-1].location_once_scrolled_into_view # driver.execute_script("arguments[0].focus();", loaded_following[loaded_till_now-1]) driver.find_element_by_tag_name('body').send_keys(Keys.END) # triggers AJAX request to load more users. observed that loading 10 users at a time. sleep(1) # tried wihtout sleep but throws StaleElementReferenceException. As it takes time to get the resposne and update the DOM loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li") loaded_till_now = len(loaded_following) # All 239 users are loaded. driver.quit()

Observed that browser is sending AJAX request to load more users. this action is triggered when you scroll using mouse or enter Space or End keys

thanks, interesting property, but how I can use it to do the needed things? see the explained message to you above.
I am not sure which element you are talking about (the class names are dynamic, so not working for me). Can you just share the screenshot of the screen, highlighting the element and what you want to do with it? either click or send keys?
At this case the class names are static, only looks like dynamic. You can check it with different browsers.
without any manipulation self.driver.find_elements_by_class_name("_6jvgy") I'll get only 10 elements,
but I want to scroll down, ideally to the end of list, to get 239 elements(for youtube).

Moshisho · Accepted Answer · 2016-12-26 07:24:18Z

4

In order to scroll in the window, you need to execute JavaScript, try this:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

EDIT: in order to focus an element (it needs to be able to get the focus e.g. an anchor, input, button etc...) you also need to use JavaScript executor:

elementToFocus = driver.find_element_by_id("yourID") driver.execute_script("arguments[0].focus();", elementToFocus)

edited Dec 26, 2016 at 7:24

answered Dec 25, 2016 at 18:46

Moshisho

3,0511 gold badge26 silver badges43 bronze badges

2 Comments

lvcpp Over a year ago

thanks, I saw this code snippet in a few places earlier. We need to focus on some element to activate it. The question is what the element we need to choose to focus and how? This is not a "body": background = browser.find_element_by_css_selector("body") # background = driver.find_element_by_css_selector("div._2uju6") for i in range(5): background.send_keys(Keys.SPACE) time.sleep(1) Without it this command do not work.

lvcpp Over a year ago

there no any element with id.<br/> I've tested the second line, but it dont work yet.<br/> So I've selected by class, but maybe need to select 2nd or 3rd element of the class<br/> (I know how to do it, but there a lot of elements, and it looks like the majority of them should be visible and acceptable).

Sida Zhou · Accepted Answer · 2021-03-24 04:03:20Z

I'm working with a dynamic React app, I need to scroll to the pages bottom to make react render all the data.

For unknown reasons, solutions based on JS execute_script didn't work. However I got send_keys solution working:

# scroll to bottom to load all WebDriverWait(driver, 5).until( EC.presence_of_element_located((By.XPATH, "//body")) ) attempt_num = 2 while attempt_num > 0: try: elem = driver.find_element_by_xpath("//body") elem.click() elem.send_keys(Keys.END) except StaleElementReferenceException as e: print(e) attempt_num = attempt_num - 1

The click() on body and the retry for StaleElementReferenceException are crucial. I haven't found a more elegant way than to retry.

See top answer of How to avoid "StaleElementReferenceException" in Selenium?

Collectives™ on Stack Overflow

Scroll in Selenium Webdriver (Python)

3 Answers 3

6 Comments

2 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

2 Comments

Comments

Linked

Related