Is there a rule of order for joining processes when using multiprocessing?

Question

I have three Python script files (script1.py, script2.py and script3.py) and I want script1.py and script2.py to run simultaneously and script3.py to start only when script2.py is finished (script1.py doesn't have to be finished yet when script3 starts).

I have a code like this to work it out but I'm not sure if I'm doing the right stuff:

import subprocess import multiprocessing def func1(command): subprocess.run(command, shell=True) def func2(command): subprocess.run(command, shell=True) def func3(command): subprocess.run(command, shell=True) if __name__ =='__main__': p1 = multiprocessing.Process(target=func1,args=('python3 script1.py',)) p2 = multiprocessing.Process(target=func2,args=('python3 script2.py',)) p3 = multiprocessing.Process(target=func3,args=('python3 script3.py',)) p1.start() p2.start() p2.join() p3.start() p3.join() p1.join()

Is it gonna work? Is there a better way to do this? Is there a rule that I have to join them in the order as they were started?

I think you wanted script3.py in the p3 line

bigh_29
– bigh_29

2019-12-09 16:16:43 +00:00
Commented Dec 9, 2019 at 16:16 — bigh_29
– bigh_29, Commented Dec 9, 2019 at 16:16
Yes, thankyou. I'll edit.

Barna96
– Barna96

2019-12-09 16:24:24 +00:00
Commented Dec 9, 2019 at 16:24 — Barna96
– Barna96, Commented Dec 9, 2019 at 16:24

Marat · Accepted Answer · 2019-12-09 16:12:06Z

Yes, it is going to work and there is no rule of order.

Also, you could use the same function for all three processes as they're identical, and you'll probably need to convert command to a list (args=(['python3', 'script1.py'],))

So the script3 won't start until the script2 is finished, right? Thank you for the quick answer! - yes, the the same function should be used!

bigh_29 · Accepted Answer · 2019-12-09 16:31:53Z

Yes, the code will function like you want. p2.join blocks until that process is completely finished.

IMO, this is a case where multithreading makes more sense than multiprocessing. Your subprocesses aren't really doing any work. They are just spawning another process and sleeping until it returns.

Consider this approach instead

import threading import subprocess def run_script(script_filename): command = f'python3 {script_filename}' subprocess.run(command, shell=True) if __name__ =='__main__': p1 = threading.Thread(target=run_script, args=['script1.py']) p2 = threading.Thread(target=run_script, args=['script2.py']) p3 = threading.Thread(target=run_script, args=['script3.py']) p1.start() p2.start() p2.join() p3.start() p3.join() p1.join()

Collectives™ on Stack Overflow

Is there a rule of order for joining processes when using multiprocessing?

2 Answers 2

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Related