Python Multiprocessing to invoke a class function

Question

I am trying to multiprocess a large number of objects that I need to invoke a class function off. I cant pull the function outside of the class and so just need to figure out a way to speed up the calling of this function.

Many thanks in advance

import time # - Bespoke Class that is standalone - # class sub_class: def __init__(self, item_id, assumptions = "BlahBlahBlah"): self.item_id = item_id self.assumptions = assumptions self.test_count = 0 def some_func(self, adder=1): self.test_count += adder time.sleep(0.1) # - initialise a list of classes - # item_ids = ['A', 'B', 'C', 'D'] items = [sub_class(_id) for _id in item_ids] print( sum(_item.test_count for _item in items) ) # Prints out 0 # - invoke a class level function - # [_item.some_func(adder=2) for _item in items] print( sum(_item.test_count for _item in items) ) # Prints out 8

Does this answer your question? python Pool with worker Processes — stovfl
– stovfl, Commented Feb 27, 2020 at 12:12

FiddleStix · Accepted Answer · 2020-02-27 17:14:09Z

This works with only a minor change to some_func().

import time from concurrent.futures import ProcessPoolExecutor, wait, ALL_COMPLETED # - Bespoke Class that is standalone - # class MyClass: def __init__(self, item_id, assumptions="BlahBlahBlah"): self.item_id = item_id self.assumptions = assumptions self.test_count = 0 def some_func(self, adder=1): self.test_count += adder print(f"{self.item_id} executing some_func()") time.sleep(0.1) # note: we must return ourselves return self # Don't omit this on Windows if __name__ == '__main__': # - initialise a list of classes - # item_ids = ['A', 'B', 'C', 'D'] items = [MyClass(item_id) for item_id in item_ids] print(sum(item.test_count for item in items)) # Prints out 0 # - invoke a class level function - # with ProcessPoolExecutor(max_workers=4) as executor: future_list = [] for item in items: future_list.append(executor.submit(item.some_func, adder=2)) wait(future_list, return_when=ALL_COMPLETED) # note: we must use the returned self to get the test count print(sum(future.result().test_count for future in future_list)) # Prints out 8 # note: the original items haven't changed print(sum(item.test_count for item in items)) # Prints out 0

The "items" get copied to the new Process so you have to use the future return() values instead. Therefore, you can replace the original items using items = [future.result() for future in future_list] at some point after the wait().

This works perfectly, exactly what I was looking for. ( Ive add a little progress bar and it work like a charm)
Perhaps, I shouldn't have been as hasty: how would you store the new outputs? e.g. Doing the below prints 0 tmp_list = [future.result().test_count for future in future_list] print(sum(tmp_list))
Apologies, this works; I had adapted a few things and left out the "return self" bit in MyClass.some_func (despite your comment not to forget it). thanks a lot

Collectives™ on Stack Overflow

Python Multiprocessing to invoke a class function

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related