selenium
You can:
- execute .js commands, ...
- interact with websites,
- ...
Selenium is availible in lots of different programming languages - not only python.
pip3 install seleniumThen you normally need download a driver for your desired browser: URL: http://www.seleniumhq.org/download/ refer to Third Party Drivers, Bindings, and Plugins
URL: https://www.selenium.dev/downloads/ go to Platforms Supported by Selenium
Direct links:
- Chromium: https://sites.google.com/chromium.org/driver/
- Firefox/Gecko: https://github.com/mozilla/geckodriver/releases
Then copy the files to the right place & adapt permissions.
sudo mv geckodriver /usr/bin/ chmod a+x /usr/bin/geckodriverJust add your executable of the Gecko driver to your Path variables. For example:
# local (per `cmd.exe` session) set PATH=%PATH%;C:\Users\usr\AppData\Local\Programs\geckoDriver # global setx PATH C:\Users\usr\AppData\Local\Programs\geckoDriver /m # Or just add path via GUIIf path is not set correctly you probably get an error message like Exception AttributeError: "'Service' object has no attribute 'process'" in <bound method Service.__del__ of <selenium.webdriver.firefox.service.Service object at 0x1006890d0>> ignored
Now we are ready to go! :D
import time from selenium import webdriver from pyvirtualdisplay import Display # Optional: use virtual display for use in headless env (servers without X) display = Display(visible=0, size=(800, 600)) # Not necessarily needed display.start() # Not necessarily needed browser= webdriver.Chrome() # For Firefox: browser = webdriver.Firefox() browser.get("https://github.com/TeamFlowerPower") # Access page wait = 5 print(f"Waiting for {wait} seconds that page is completely loaded...") time.sleep(wait) print(browser.current_url) # Print URL print(browser.page_source.encode('utf-8')) contentXPattern = r'//*[@id="MainPart_lbUsersInLineAheadOfYou"]' result = None try: someVarTmp = browser.find_element_by_xpath(contentXPattern) # Raises an NoSuchElementException when pattern not found someVar = someVarTmp.text # Convert to text print(f"Users ahead of me: {someVar}") except Exception as ex: exName = type(ex).__name__ # Get exception name for printing print(f"XPath not found ({exName})") #browser.quit() result = None else: result = {someVar: browser} browser.quit() display.stop() # Not necessarily neededNote: quit() may raise a warning/exception in Firefox; there is also close() where this message does not show up using Firefox but it does not close the window either.
browser.execute_script("document.getElementById('email').setAttribute('value', 'recipient@gmail.com')");TODO
Most in modern Selenium with JavaScript is based on Promises: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises
Here also arrow-functions become handy.
// Traditional anonymous function function (name){ return `hello ${name}`; } # becomes: (name) => { return `hello ${name}`; } namedArrowFunction = (name) => `hello ${name}`; // Only omit the `return` for one-line functions (no curly braces)About template literals -> `${}`
Allows sequentiality of promises:
A common need is to execute two or more asynchronous operations back to back, where each subsequent operation starts when the previous operation succeeds, with the result from the previous step. We accomplish this by creating a promise chain.
Here's the magic: the then() function returns a new promise, different from the original
(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises#chaining)
XPath, CSS, ...
# XPath //div[starts-with(@class,'class-name-fragment-start')] //*[contains(@class,'class-name-fragment')] //div[ends-with(@class,'class-name-fragment-start')] # `*`: any element # CSS div[class^='class-name-fragment-startswith'] div[class*='class-name-fragment-contains'] div[class$='class-name-fragment-endswith']The Attribute is role (with the name button) in this example:
# XPath //div[@role = 'button'] # CSS div[role = 'button']
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License *.
Code (snippets) are licensed under a MIT License *.
* Unless stated otherwise