Cannot use result from Multiprocess Pool directly

Question

I have the following example code:

def my_function_caller(): samples = [] for t in range(2): samples.append(my_function(t)) return samples def my_function(t): results = [] if __name__ == '__main__': pool = Pool() results = pool.map(task, range(5)) pool.close() pool.join() A = results[0] return A def task(k): time.sleep(1) result = k return result

When I call my_function(t), I get the following error:

 A = results[0] IndexError: list index out of range

I expected pool.close() and pool.join() to make the program wait for all processes to finish so that I could then use the jointly computed result "results" afterwards. How can I force the program to wait or more generally, how can I directly use "results" in the function "my_function"?

EDIT: To recreate the error: This is the complete code that I am running (simply copied and pasted). The python file called main.py is located in a standard Python project and I am using Windows.

from multiprocessing import Pool import time def my_function_caller(): samples = [] for t in range(2): samples.append(my_function(t)) return samples def my_function(t): results = [] if __name__ == '__main__': pool = Pool() results = pool.map(task, range(5)) pool.close() pool.join() A = results[0] return A def task(k): time.sleep(1) result = k return result a = my_function_caller()

Maybe, as additional information, I get the error message

 A = results[0] IndexError: list index out of range

several times, not just once.

Please update your question with the code that calls my_function(). Also, try your code again, but omit the first line of your function, ie omit results = []. — quamrana
– quamrana, Commented Aug 23, 2021 at 9:49
This is not a minimal, reproducible example. I should be able to copy and paste what you post and reproduce the problem without adding anything. — Booboo
– Booboo, Commented Aug 23, 2021 at 10:02
@quamrana When I removed results = [], I got the error message: UnboundLocalError: local variable 'results' referenced before assignment. I added the function that is calling my_function(t). — Felix P.
– Felix P., Commented Aug 23, 2021 at 10:08
Very weird indeed, thank God I don't use Windows. See this answer: stackoverflow.com/a/53924048/10499398 — swaggg
– swaggg, Commented Aug 23, 2021 at 11:17
The solution therefore is to put the if statement at the top of your script, otherwise Windows will still execute the rest of the program and then cause an unhandled error. — swaggg
– swaggg, Commented Aug 23, 2021 at 11:20

Roman Pavelka · Accepted Answer · 2021-08-23 11:26:39Z

It worked for me on Linux. However, I consider the structure little bit messy, consider e.g. this to more easily debug your problem:

from multiprocessing import Pool import time def my_function_caller(): samples = [] for t in range(2): samples.append(my_function(t)) return samples def my_function(t): with Pool(5) as p: results = p.map(task, range(5)) A = results[0] return A def task(k): time.sleep(1) result = k return result if __name__ == "__main__": a = my_function_caller() print(a)

swaggg · Accepted Answer · 2021-08-23 11:36:57Z

It's not really my answer but I'm going to post it as an answer anyway. Windows displays some really messed up behaviour as mentioned here:

python multiprocessing on Windows

The process is supposed to only call your function, but it ends up executing the whole program all over again.

You have to prepend the entry point with if __name__ == "__main__":

if __name__ == "__main__": a = my_function_caller()

Separately, you should still use if __name__ == "__main__" or __name__ == "__mp_main__": in your threaded function, but either at the top or at least making sure the program won't try to access a non-existent value if being imported.

Collectives™ on Stack Overflow

Cannot use result from Multiprocess Pool directly

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related