3

Why does this happen using selenium with python and firefox? Selenium cant scale to a simple loop. Why cant firefox scale? Over time it decides to thow an error and quit.

Mozilla Firefox 45.0 selenium.version '2.53.2'

from pyvirtualdisplay import Display display = Display(visible=0, size=(800, 600)) display.start() mylist = ['page1','page1',...,'pageN'] while True: for data in mylist: browser = webdriver.Firefox() try: myfunc(data) # Code that uses selenum to fetch pages except: pass browser.quit() time.sleep(60) No handlers could be found for logger "sentry.errors.serializer" Traceback (most recent call last): File "platrieveerp.py", line 231, in <module> browser = webdriver.Firefox() File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/webdriver.py", line 103, in __init__ self.binary, timeout) File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__ self.binary.launch_browser(self.profile, timeout=timeout) File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser self._wait_until_connectable(timeout=timeout) File "/usr/local/lib/python2.7/dist-packages/selenium-2.53.2-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable % (self.profile.path)) selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpOgsKOx If you specified a log_file in the FirefoxBinary constructor, check it for details.
Improve this question
1
  • Quick check - would the same error be thrown in case of driver.close() instead of driver.quit()? Thanks. Commented May 22, 2016 at 23:52

1 Answer 1

Reset to default
1

Selenium is not failing while running the loop, it is failing at webdriver.Firefox(). The error is recognized when the connection times out at a certain time after line 231 when the script still cant recognize an open firefox application.

It is just a bug in selenium when running with a headless browser. To get a working version of selenium, run 

sudo pip install -U selenium

This should be a similar issue: Selenium: FirefoxProfile exception Can't load the profile

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

I actually had the same issue earlier today and thought I resolved it with the above statement. It turns out that it was only working when I run the script through a crontab instead of in the terminal. Not that running your script with a crontab is the solution, but just sharing my same problem
I ran the script twice today, successfully and then started falling to this trap, cleared temp files, did not work. Worst part is, it is still working fin in my local, just stopped working on the server!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.