9

I am using selenium with Firefox to automate some tasks on Instagram. It basically goes back and forth between user profiles and notifications page and does tasks based on what it finds.

It has one infinite loop that makes sure that the task keeps on going. I have sleep() function every few steps but the memory usage keeps increasing. I have something like this in Python:

while(True): expected_conditions() ...doTask() driver.back() expected_conditions() ...doAnotherTask() driver.forward() expected_conditions()

I never close the driver because that will slow down the program by a lot as it has a lot of queries to process. Is there any way to keep the memory usage from increasing overtime without closing or quitting the driver?

EDIT: Added explicit conditions but that did not help either. I am using headless mode of Firefox.

Improve this question
3
  • Try to avoid using sleep method, use Explicit Waits or Implicit Waits methods see this for more information selenium-python.readthedocs.io/waits.html Commented Mar 9, 2019 at 0:27
  • Alright. I will try to do that tomorrow and post the discovery here. Commented Mar 9, 2019 at 0:32
  • A change from sleep() to waits won't change anything re: mem usage. Te reason for not using sleep is because this hardcodes an interval where your script is not doing anything, in the hope the page will change state; and with the *Wait the interval is not hardcoded, but "as soon as the change happened ". In your usecase it does sound like you use the sleep() as delimiter between executions, and running in an infinite loop you don't sound like you'd care for shaving a few seconds off. Commented Mar 9, 2019 at 4:12

5 Answers 5

Reset to default
12

Well, This the serious problem I've been going through for some days. But I have found the solution. You can add some flags to optimize your memory usage.

options = Options() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") options.add_argument('--no-sandbox') options.add_argument('--disable-application-cache') options.add_argument('--disable-gpu') options.add_argument("--disable-dev-shm-usage")

These are the flags I added. Before I added the flags RAM usage kept increasing after it crosses 4GB (8GB my machine) my machine stuck. after I added these flags memory usage didn't cross 500MB. And as DebanjanB answers, if you running for loop or while loop tries to put some seconds sleep after each execution it will give some time to kill the unused thread.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

I am pretty sure this was helpful, deserves more upvotes. One question. You said you kept the memory below 50 MB?? For me its constantly below 500 MB. Was yours a typo or how is it possible to have a 10 times higher memory usage?
yes sorry that's typo i changed it Thanks
When I run this code without setting options.headless, Firefox complains that it cannot reach the website "disable-infobars". You may wish to add two dashes before "start-maximized" and "disable-infobars". Using Firefox 113.0.2, if it matters.
5

To start with Selenium have very little control over the amount of RAM used by Firefox. As you mentioned the Browser Client i.e. Mozilla goes back and forth between user profiles and notifications page on Instagram and does tasks based on what it finds is too broad as a single usecase. So, the first and foremost task would be to break up the infinite loop pertaining to your usecase into smaller Tests.

time.sleep()

Inducing time.sleep() virtually puts a blanket over the underlying issue. However while using Selenium and WebDriver to execute tests through your Automation Framework, using time.sleep() without any specific condition defeats the purpose of automation and should be avoided at any cost. As per the documentation:

time.sleep(secs) suspends the execution of the current thread for the given number of seconds. The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. Also, the suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.

You can find a detailed discussion in How to sleep webdriver in python for milliseconds

Analysis

There were previous instances when Firefox consumed about 80% of the RAM.

Firefox_RAM

However as per this discussion some of the users feels that the more memory is used the better because it means you don't have RAM wasted. Firefox uses RAM to make its processes faster since application data is transferred much faster in RAM.

Solution

You can implement either/all of the generic/specific steps as follows:

  • Upgrade Selenium to current levels Version 3.141.59.
  • Upgrade GeckoDriver to GeckoDriver v0.24.0 level.
  • Upgrade Firefox version to Firefox v65.0.2 levels.
  • Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
  • If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.

  • Some extensions allow you to block such unnecessary content, as an example:

    • uBlock Origin allows you to hide ads on websites.
    • NoScript allows you to selectively enable and disable all scripts running on websites.

    • To open the Firefox client with an extension you can download the extension i.e. the XPI file from https://addons.mozilla.org and use the add_extension(extension='webdriver.xpi') method to add the extension in a FirefoxProfile as follows:

      from selenium import webdriver profile = webdriver.FirefoxProfile() profile.add_extension(extension='extension_name.xpi') driver = webdriver.Firefox(firefox_profile=profile, executable_path=r'C:\path\to\geckodriver.exe')

  • If your Tests doesn't requires the CSS you can disable the CSS following the this discussion.

Improve this answer

7 Comments

Hi. The driver, Firefox and Selenium are all up-to-date. My workspace was new but I cleaned and rebuilt it again. It didn't seem to affect the performance in any way. I have also gotten rid of time.sleep and using explicit conditions provided in Selenium I'm using headless mode for firefox, would css/scripts still matter for that?
@NaeemKhan Of coarse it matters. Check out the discussion for disabling CSS within my answer.
There doesn't seem to be an add_experimental_option() function in FirefoxOptions(). How would I disable it for Firefox?
Ok so I have disabled css by firefoxProfile.set_preference('permissions.default.stylesheet', 2) and images as well. It still did not make a difference. The Ram usage is still increasing. I think the problem could be Instagram's heavy usage of javascript and AJAX usage?
It still didn't solve the problem but it was the closest to my problem, so I will accept the answer.
|
3
  • Use Explicit Waits or Implicit Waits.
  • Use driver.quit() to close all the browser windows and terminate the WebDriver session because if you do not use quit() at the end of the program, the WebDriver session will not be closed properly and the files will not be cleared off memory. And this may result in memory leak errors.
Improve this answer

3 Comments

Adding Explicit waits did not help, it's still the same story. The memory usage keeps increasing. I did find a temporary work around by adding an inner loop to run X number of times before closing/quitting the driver, and the infinite loop starts the driver again and goes into that loop again. It does prevent it from eating up all the memory but that is not a proper solution.
are you used driver.quit()?
Yeah. Did did driver.close() then driver.quit()
2

Creating new firefox profile and use it every time while running test cases in Firefox shall eventually increase the performance of execution as without doing so always new profile would be created and caching information would be done there and if driver.quit does not get called somehow before failure then in this case, every time we end up having new profiles created with some cached information which would be consuming memory.

// ------------ Creating a new firefox profile -------------------

1. If Firefox is open, close Firefox. 2. Press Windows +R on the keyboard. A Run dialog will open. 3. In the Run dialog box, type in firefox.exe -P Note: You can use -P or -ProfileManager(either one should work). 4. Click OK. 5. Create a new profile and sets its location to the RAM Drive.

// ----------- Associating Firefox profile ------------------- 

ProfilesIni profile = new ProfilesIni(); FirefoxProfile myprofile = profile.getProfile("automation_profile"); WebDriver driver = new FirefoxDriver(myprofile);

Please share execution performance with community if you plan to implement this way.

Improve this answer

2 Comments

I'm using headless mode though. Would using a different profile even matter?
I would still say yes you shall give a try. because caching information shall be part of profile management in headless mode too. Apart from this, i would like to share, The headless mode of Mozilla Firefox performs 3.68% better than the UI version. This is a disappointment since the Chrome's headless mode achieves > 30% better time than the UI one.
2

There is no fix for that as of now. I suggest you use driver.close() approach. I was also struggling with the RAM issue and what i did was i counted the number of loops and when the loop count reached to a certain number( for me it was 200) i called driver.close() and then start the driver back again and also reset the count. This way i did not need to close the driver every time the loop is executed and has less effect on the performance too. Try this. Maybe it will help in your case too.

Improve this answer

1 Comment

This fixed it for me. Sadly, the number for me was WAY smaller (~25) but I assume this is highly dependent on the site and your local machine.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.