0

I am trying to run a Selenium WebDriver script that takes quite a long time to complete. The thing is that while executing the said script on my server, after about an hour and a half, actions that before took 10 seconds to complete now take about 2 minutes, which makes the execution extremly slow. Analyzing what the problem could be, I have seen that several crhome processes are created during the execution. For example, when I create the execution at first, I have 4 chrome processes, while, when the execution has spent some hours executing, 24 processes have been created.

I have the following script that tries to rerun the script if an exception has been thrown during the execution and also try to remove unnecessary chrome processes to free resources up every hour, but it seems this is not working, because instead of what I wanted to do, this is killing the main chrome process, meaning it causes the script to stop and having to rerun it from the shell (so only 10 hours of execution would be performed, one per attempt). Is there any way to only get rid of the unnecessary chrome processes in this script?

max_attempts=10 attempt=1 log_file="../logs/execution_log_$(date +"%d%m%Y").txt" while [ $attempt -le $max_attempts ]; do # Kill any running instances of python3, chromedriver, chrome and google-chrome-stable pkill -f "python3" pkill -f "chromedriver" pkill -f "chrome" pkill -f "google-chrome-stable" # Run script with nohup nohup python3 main.py >> "$log_file" 2>&1 & # Store the process ID of the script script_pid=$! start_time=$(date +%s) # Wait for the script to finish or check if it is still running while ps -p $script_pid > /dev/null; do sleep 5 # Check if the elapsed time exceeds 1 hour (3600 seconds) current_time=$(date +%s) elapsed_time=$((current_time - start_time)) if [ $elapsed_time -ge 3600 ]; then echo "$(date) Killing zombie Chrome processes..." # Kill all chrome processes except for the oldest pkill -f "chrome --type=zygote" pkill -f "chrome --type=renderer" pkill -f "chrome --type=gpu-process" pkill -f "chrome --type=utility" break fi done echo "$(date) The command has stopped" exit_code=$? # Check if exit code is different from 0 and if the execution log file's last line is equal to the literal String to break the loop if [ $exit_code -ne 0 ] && [ "$(tail -n 1 "$log_file")" = "ENDOF SCRIPT" ]; then echo "$(date) Script completed successfully with exit code 0." break else echo "$(date) Script exited with code $exit_code." # If it's the last attempt, print a failure message and exit if [ $attempt -eq $max_attempts ]; then echo "$(date) Maximum attempts reached. Script failed." # Kill any running instances of python3, chromedriver, chrome and google-chrome-stable pkill -f "python3" pkill -f "chromedriver" pkill -f "chrome" pkill -f "google-chrome-stable" exit 1 fi echo "$(date) Restarting command..." attempt=$((attempt + 1)) fi done
Improve this question
1
  • modern browsers isolate domains by keeping them in separate threads... (it's a security thing...) You might just check for any extra windows/tabs. It sounds like that may be happening, but there are all sorts of outside-domain resources that sites will load... (ads, trackers, js, css, 3rd party logins, etc...) It's hard to say what kind of options you could set in the driver to help here without know why so many domains/threads are being accessed. One thing I usually set is to turn off DNS-Prefetch, but not sure it'll help here. Commented Jul 18, 2023 at 22:47

1 Answer 1

Reset to default
0

I am not sure if you are already doing this, but if not, you could combine your Selenium script with a unit test runner. When your script is combined with a test runner you can execute useful setUp and tearDown actions before and after the test has run. As an example even after an error the Selenium resources should be correctly closed / disposed of. (Also I am not sure in which language your Selenium script is written in so I just added a Python example...)

import unittest from selenium import webdriver class Tests(unittest.TestCase): def setUp(self): # create a new Chrome session self.driver = webdriver.Chrome() self.driver.implicitly_wait(30) self.driver.maximize_window() # navigate to the application home page self.driver.get("https://www.google.com/") def test(self): # your test code def tearDown(self): # close the browser window self.driver.quit()
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments