How to set up a timeout for multithreading?

Question

I have a function that parses documents in a MongoDB collection. I would like to set up a timeout for each thread because I won't know if function2 will take too much time to finish. I tried setting up a @timeout_decorator.timeout(60, use_signals=False), but apparently it does not work.

Better alternatives are considered as well. Any suggestion is greatly appreciated!

def function(collection): with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: for document in collection.find({}, no_cursor_timeout=True): executor.submit(function2, document) def function2(collection, document): try: ... something ...

gold_cy · Accepted Answer · 2020-05-05 11:34:04Z

Since you are using executor.submit all your submitted tasks are returning a Future object which has its own way to declare a timeout.

def function(collection): futures = [] with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: for document in collection.find({}, no_cursor_timeout=True): future = executor.submit(function2, document) futures.append(future) futures = function(collection) for future in concurrent.futures.as_completed(futures): result = future.result(timeout=60)

You can find more documentation here.

Thank you, that worked out! I was wondering if with this workaround some threads (especially the first ones) will have a little bit more of time than 60 seconds because they are executed whilst for is still running before returning futures to the invocation.
you can just iterate over the futures themselves in that case and try to get the result since futures are unevaluated at that point.

Collectives™ on Stack Overflow

How to set up a timeout for multithreading?

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related