The CollapsingThreadPoolExecutor is inspired by and compatible with the ThreadPoolExecutor from the "futures" module, it operates differently in that worker threads are handled with a stack which results in the same worker or workers doing all the work (and idle workers being destroyed).
$ pip install collapsing-thread-pool-executor Prerequisites
- python3 w/ pip
- python2 w/ pip
- virtualenvwrapper
- entr
Set up the environments
$ mkvirtualenv -p `which python2.7` collapsing-thread-pool-executor-py2 $ pip install . $ pip install -r requirements.txt $ mkvirtualenv -p `which python3` collapsing-thread-pool-executor-py3 $ pip install . $ pip install -r requirements.txt Watch the tests
# watch python2 tests in one window $ workon collapsing-thread-pool-executor-py2 $ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor # watch python3 tests in one window $ workon collapsing-thread-pool-executor-py3 $ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor The example below will execute some_task() 100 times; as some_task() should take a second to execute and as we've allocated 10 workers, the whole thing should take about 10 seconds.
import time from collapsing_thread_pool_executor import CollapsingThreadPoolExecutor def some_task(): time.sleep(1) # all arguments are optional pool = CollapsingThreadPoolExecutor( workers=10, thread_name_prefix='SomePool', permitted_thread_age_in_seconds=60, ) for i in range(0, 100): pool.submit(some_task)