1

I have a big code that take a while to make calculation, I have decided to learn about multithreading and multiprocessing because only 20% of my processor was being used to make the calculation. After not having any improvement with multithreading, I have decided to try multiprocessing and whenever I try to use it, it just show a lot of errors even on a very simple code.

this is the code that I tested after starting having problems with my big calculation heavy code :

from concurrent.futures import ProcessPoolExecutor def func(): print("done") def func_(): print("done") def main(): executor = ProcessPoolExecutor(max_workers=3) p1 = executor.submit(func) p2 = executor.submit(func_) main()

and in the error message that I amhaving it says

An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ...

this is not the whole message because it is very big but I think that I may be helpful in order to help me. Pretty much everything else on the error message is just like "error at line ... in ..."

If it may be helpful the big code is at : https://github.com/nobody48sheldor/fuseeinator2.0 it might not be the latest version.

Improve this question
4
  • You are running on Windows? How do you call main()? That's where the problem is. On Windows, multiprocessing starts a new python and then imports your module. Code at module level runs a second time. If you call main(), it gets called again in the subprocess and your whole program re-executes. Python detected that and gave the warning instead. Stuff inside if __name__ == "__main__": is not executed on mere importing. So stick your top level script code inside there and it works. Commented Apr 19, 2021 at 1:39
  • yes, I m running on windows. But I don't understand why would my code run 2 times because the function is just being called at the end, and even if I don't make a main function for the multiprocessing stuffs it still show the same message Commented Apr 19, 2021 at 1:48
  • But where do you call main()? Can you add that to the example code to make it runnable? See Safe importing of main module in The spawn and forkserver start methods - Windows spawns. Commented Apr 19, 2021 at 2:14
  • Ah my bad I forgot to past it but I calles main() at the end. And btw I am not using the multiprocessing library but the concurrent.futures I knows it is very similar but the way to add it is not the same, and in my big code I have to return lists so I prefer to use concurrent.futures Commented Apr 19, 2021 at 2:28

1 Answer 1

Reset to default
2

I updated your code to show main being called. This is an issue with spawning operating systems like Windows. To test on my linux machine I had to add a bit of code. But this crashes on my machine:

# Test code to make linux spawn like Windows and generate error. This code # # is not needed on windows. if __name__ == "__main__": import multiprocessing as mp mp.freeze_support() mp.set_start_method('spawn') # test script from concurrent.futures import ProcessPoolExecutor def func(): print("done") def func_(): print("done") def main(): executor = ProcessPoolExecutor(max_workers=3) p1 = executor.submit(func) p2 = executor.submit(func_) main()

In a spawning system, python can't just fork into a new execution context. Instead, it runs a new instance of the python interpreter, imports the module and pickles/unpickles enough state to make a child execution environment. This can be a very heavy operation.

But your script is not import safe. Since main() is called at module level, the import in the child would run main again. That would create a grandchild subprocess which runs main again (and etc until you hang your machine). Python detects this infinite loop and displays the message instead.

Top level scripts are always called "__main__". Put all of the code that should only be run once at the script level inside an if. If the module is imported, nothing harmful is run.

if __name__ == "__main__": main()

and the script will work.

There are code analyzers out there that import modules to extract doc strings, or other useful stuff. Your code shouldn't fire the missiles just because some tool did an import.

Another way to solve the problem is to move everything multiprocessing related out of the script and into a module. Suppose I had a module with your code in it

whatever.py

from concurrent.futures import ProcessPoolExecutor def func(): print("done") def func_(): print("done") def main(): executor = ProcessPoolExecutor(max_workers=3) p1 = executor.submit(func) p2 = executor.submit(func_)

myscript.py

#!/usr/bin/env pythnon3 import whatever whatever.main()

Now, since the pool is laready in an imported module that doesn't do this crazy restart-itself thing, no if __name__ == "__main__": is necessary. Its a good idea to put it in myscript.py anyway, but not required.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

No, no, no! You still need if __name__ == '__main__': whatever.main().
thanks it works! but I have to plot the whole thing in the main() function, but in my big code I make a figure with a lot of plots, how can I make the plotting outside main() without making a circular import ?
@nobody48sheldor - you can put them in another function that is called from main. The trick is to make sure that there is no path to the code that runs merely from import.
I like the second solution generally where code that does stuff is separated into importable modules while top level scripts that are called to start a python program are relatively small and concerned with command line parsing, etc. That way, the bulk of the code is more easity unit tested.
thanks you all ! I finally get it working !

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.