Python multiprocessing continuous processing with await

Question

I am using an event based system using the new Python 3.5 coroutines and await. I register events and these events are called by the system.

@event aysnc def handleevent(args): # handle the event

I need to initialize some classes to handle the work(time consuming). Then call instance methods, also time consuming (they actually use selenium to browse certain sites).

Ideally I would want something like the following code

# supposedly since this is multiprocessing this is a different driver per process driver = None def init(): # do the heavy initialization here global driver driver = webdriver.Chrome() def longworkmethod(): ## need to return some data return driver.dolongwork() class Drivers: """ A class to handle async and multiprocessing""" def __init__(self, numberOfDrivers): self.pool = multiprocessing.Pool(processes=numberOfDrivers, initializer=init) async def dowork(self, args): return self.pool.apply_async(longworkmethod, args=args) ### my main python class drivers = Drivers(5) @event aysnc def handleevent(args): await drivers.dowork(args) @event aysnc def quit(args): ## do cleanup on drivers sys.exit(0)

This code doesn't work, but I have tried many different ways and none seem to be able to do what I want.

It doesn't have to be this exact form, but how do I go about mixing the await and coroutines with a program that needs multiprocessing?

Community · Accepted Answer · 2017-05-23 12:25:27Z

While there nothing technically speaking that would limit you from mixing asyncio and multiprocessing, I would suggest avoiding doing so. It's going to add a lot of complexity as you'll end up needing an event loop per thread and passing information back and forth will be tricky. Just use one or the other.

asyncio supplies functions for running tasks in another thread - such as AbstractEventLoop.run_in_executor. Take a look at these answers

https://stackoverflow.com/a/33025287/66349 (calling selenium within a coroutine)
https://stackoverflow.com/a/28492261/66349

Alternatively you could just use multiprocessing as selenium has a blocking (non asyncio) interface, however it sounds like some of your code is using already using asyncio so maybe stick with the above.

I can't avoid the mixing because I need selenium(multiprocessing), and the event system I'm using is strictly asyncio. But I'll check those links out. See if I can make them work.
selenium is blocking but that doesn't mean you need multiprocessing. Just use run_in_executor for the blocking bits and keep everything else in the main thread/event loop.
So those links seem to help. Should I be use run_in_executor for individual expensive calls like "get", or can I do large expensive functions with multiple expensive calls? Would there be a difference?
@leftsync It depends on the structure of your calls. Can they all be run at once or do they need to be run in sequence? How many threads can you feasibly support? Is selenium thread safe?

Collectives™ on Stack Overflow

Python multiprocessing continuous processing with await

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related