Multiprocessing in Python using Process, how to limit the number of processes

Question

I'd like to run multiple processes concurrently, but using Process I cannot limit the number of processes at a time, so that my computer becomes unusable for anything else. In my problem I have to run the main_function for all of the data in my_dataset. Here is a short sample of my code, is it possible to limit the number of processes at a time?

from multiprocessing import Process def my_function(my_dataset): processes = [] for data in my_dataset: transformed_data = transform(data) p = Process(target=main_function, args=(data, transformed_data)) p.start() processes.append(p) for p in processes: p.join()

Yes. Look at using a pool of worker processes: docs.python.org/3/library/… — Jerry101
– Jerry101, Commented Dec 13, 2022 at 9:26

Meny Issakov · Accepted Answer · 2022-12-13 09:36:18Z

1

You can utilize the multiprocessing's Pool https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool

from multiprocessing import Pool names = ["Joe", "James", "Jimmy"] * 10 def print_name(name): print(f"Got Name: {name}") def runner(): p = Pool(4) p.map(print_name, names) if __name__== "__main__": runner()

edited Dec 13, 2022 at 9:36

answered Dec 13, 2022 at 9:28

Meny Issakov

1,4512 gold badges15 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ahmed AEK Over a year ago

add if __name__== "__main__": before runner() orelse his PC will crash

Meny Issakov Over a year ago

Depends where you're running it, I ran it on Jupyter notebook. but will add it for clarity

Sam Mason Over a year ago

Just for completeness, the check is needed for Windows and OSX

Collectives™ on Stack Overflow

Multiprocessing in Python using Process, how to limit the number of processes

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related