python multiprocessing: processes don't work

Question

I want to assign my computing jobs to more than one cpu, so I choose multiprocessing. However, the result is not what I want.

import numpy as np from multiprocessing import Process def func(begin,end): print('*'*5) print('begin=%d' %(begin)) for i in range(begin,end): for j in range(10): myarray[i][j]=1 myarray=np.zeros((12,10)) print(myarray) for i in range(4): begin=i*3 end=(i+1)*3 p=Process(target=func,args=(begin,end,)) p.start() print('*'*5) print(myarray)

I think that myarray should be all ones. But it doesn't change at all. Why? The func function does not change elements of myarray? I tried the example from this linkenter link description here

from multiprocessing import Process def f(name): print('hello',name) p=Process(target=f,args=('bob',)) p.start()

It shows nothing on the screen. Why? How should I finish my computation with python? Can anyone give a way of fully take advantage of multi-cpus?

@alfasin Seems to me it should. But I test that code using python 3.4.3 on win7 and corresponding code(the print function to statement) with python 2.6.6 on centos6.5 in virtual box. Neither show anything. — dudu
– dudu, Commented Aug 31, 2015 at 4:15
@alfasin Sorry. The second didn't work on win7 with python3.4.3. This is the platform I use to test simple programs. But it is okay with centos with python 2.6.6. I've checked that again. Although I cann't figure out why, it is the situation now. — dudu
– dudu, Commented Aug 31, 2015 at 6:54

DWilches · Accepted Answer · 2015-08-31 03:59:50Z

There are two problems there:

When you print your array at the end, how do you know your processes have already finished? You need to invoke join() on each process to ensure they have finished.
Each process has a copy of "myarray". If you want to communicate several processes you need to either use a Queue or a Pipe. Check the documentation that talks about exchanging data between processes

Here is a working example using as base the one you posted (it is not intended to be fast, just to show how the communication is done):

from multiprocessing import Process, freeze_support, Queue def func(my_id, q, begin, end): global myarray print('Process %d has range: %d - %d' % (my_id, begin, end)) for i in range(begin,end): q.put((i, i * 2)) if __name__ == "__main__": freeze_support() q = Queue() processes = [] myarray=[0] * 12 print("At the beginning the array is ", myarray) for i in range(4): begin = i*3 end = (i+1)*3 p = Process(target=func, args=(i, q, begin, end)) p.start() processes.append(p) for p in processes: p.join() while not q.empty(): (index, value) = q.get() myarray[index] = value print("At the end the array is ", myarray)

Try to change the line p.join() by pass to see what happens :)

Yes, join is a must. If I replace the p.join() with pass, some values are changed while others are not.

peihuadeng · Accepted Answer · 2015-08-31 05:40:57Z

That is because the print(myarray) statement execute before the process end. You should wait for the process end and then execute this statement. Please note the thread.join() statement at the following.

import numpy as np import threading lock = threading.RLock() thread_list = [] def func(begin,end): print('*'*5) print('begin=%d' %(begin)) for i in range(begin,end): for j in range(10): with lock: myarray[i][j]=1 myarray=np.zeros((12,10)) print(myarray) for i in range(4): begin=i*3 end=(i+1)*3 p=threading.Thread(target=func,args=(begin,end,)) p.start() thread_list.append(p) print('*'*5) for thread in thread_list: thread.join() print(myarray)

Collectives™ on Stack Overflow

python multiprocessing: processes don't work

2 Answers 2

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Related