Execute ThreadPoolExecutor asynchronously but use only 1 process and 1 thread

Question

Can I limit ThreadPoolExecutor to use only 1 processor core, 1 process and 1 thread at the same time but run a function asynchronously?

import time, concurrent.futures def ts(start): return round(time.perf_counter() - start, 3) def fn(sec): print(f'{ts(start)}\tstart sleeping {sec}') time.sleep(sec) def main(): lst = [1, 1, 1, 1, 1, 1] with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor: executor.map(fn, lst) start = time.perf_counter() if __name__ == "__main__": main() print(f'{ts(start)}\tfinish') # 0.001 start sleeping 1 # 1.008 start sleeping 1 # 2.01 start sleeping 1 # 3.021 start sleeping 1 # 4.024 start sleeping 1 # 5.038 start sleeping 1 # 6.051 finish

I'd like to run all the functions successively but instead of waiting the result after each run, wait all the results at the end like this:

# 0.001 start sleeping 1 # 0.002 start sleeping 1 # 0.003 start sleeping 1 # 0.004 start sleeping 1 # 0.005 start sleeping 1 # 0.006 start sleeping 1 # 1.051 finish

alex_noname · Accepted Answer · 2020-07-23 22:28:08Z

1

Thread pool works as follows, it starts a task in a thread and until it returns control the thread is busy, even if it sleeps. Therefore, if you want to run them in parallel, you have to either use multiple threads or use asyncio coroutines.

Here is asyncio sample:

import asyncio import time def foo(): print('Start ...') time.sleep(5) print('... End') async def main(): loop = asyncio.get_running_loop() count = range(2) futures = [loop.run_in_executor(None, foo) for i in count] results = await asyncio.gather(*futures) asyncio.run(main())

Output

Start ... Start ... ... End ... End

edited Jul 23, 2020 at 22:28

answered Jul 23, 2020 at 21:49

alex_noname

33.2k6 gold badges95 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

dereks Over a year ago

How can I do this with asyncio coroutines? Can you show code example?

alex_noname Over a year ago

Modified answer

dereks Over a year ago

I know that asyncio.sleep(5) will work. My question is how to run blocking code like time.sleep(sec). time.sleep(sec) MUST be inside the function.

alex_noname Over a year ago

Ok, you can run blocking code using run_in_executor.

dereks Over a year ago

I have an error trying to run your code on Windows Python 3.8.5:

\Python\Python38\lib\asyncio\base_events.py:1860: RuntimeWarning: coroutine 'foo' was never awaited handle = None # Needed to break cycles when an exception occurs. RuntimeWarning: Enable tracemalloc to get the object allocation traceback

|

Collectives™ on Stack Overflow

Execute ThreadPoolExecutor asynchronously but use only 1 process and 1 thread

1 Answer 1

8 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Related