6

In a typical JAVA application, one configures a global ExecutorService for managing a global thread pool. Lets say I configure a fixed thread pool of 100 threads:

ExecutorService threadPool = Executors.newFixedThreadPool(100);

Now lets say that I have a list of 1000 files to upload to a server, and for each upload I create a callable that will handle the upload of this one file.

List<Callable> uploadTasks = new ArrayList<Callable>(); // Fill the list with 1000 upload tasks

How can I limit the max number of concurrent uploads to, lets say, 5?

if I do 

threadPool.invokeAll(uploadTasks);

I dont have control on how many threads my 1000 tasks will take. Potentially 100 uploads will run in parallel, but I only want max 5. I would like to create some sort of sub-executor, wich uses a subset of the threads of the parent executor. I dont want to create a new separated executorService just for upload, because i want to manage my thread pool globally.

Does any of you know how do to that or if an existing implementation exists? Ideally something like

ExecutorService uploadThreadPool = Executors.createSubExecutor(threadPool,5);

Many thanks,

Antoine

Improve this question
4
  • Can you take a look at stackoverflow.com/q/19819837/2231632 and see if that applies to your case? And you can't use a Semaphore or a latch of some sort for your upload logic alone? Commented May 22, 2014 at 13:54
  • The question you mention is more about rate limitation. I could use a semaphore in my client code, was wondering if there is an existing facility for doing that. Commented May 22, 2014 at 14:07
  • 1
    I would argue that a typical Java application has numerous thread-pools, whose configuration matches their use case. So I would expect your application to have a separate thread pool of 5 threads for that purpose. Commented May 22, 2014 at 14:10
  • 2
    I understand your point, but wouldnt it be better if this separate 5 thread pool was a sub-threadpool of a globally capped thread pool that you can control? makes sense only to me? Commented May 22, 2014 at 16:03

1 Answer 1

Reset to default
1

In a typical JAVA application, one configures a global ExecutorService for managing a global thread pool.

I'm not doing this, but maybe i'm atypical. :-) Back to your question:

As having a guard (possibly using a Semaphore inside your Callables will only clutter your global Executor which tasks waiting for each other, you'll have to either

  • use some external logic which ensures only 5 jobs are running at any time, which could in itself be a Callable submitted to your one Executor. This could be done by wrapping your download jobs with some logic which will pull the next job from a queue (containing the URLs or whatever) once one job is completed. Then you submit five of those "drain download queue" jobs to your Executor and call it done. But atypical as I am, I'd just
  • use a separate Executor for the downloads. This also gives you the ability to name your threads appropriately (in the Executors ThreadFactory), which might help with debugging and makes nice thread dumps.
Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

I guess I would go with your first proposition, was wondering if there was some existing facility for doing this, something like threadPool.createSubThreadpool(5)
May I ask why you want only one Executor in your program?
Because if each component in your application uses his own thread pool, you dont have any visibility on the total number of threads your program will spawn. You could end up exhausting resources without knowing it.
@user3665244 I think you could make one or two exceptions to that rule without growing confused about total numbers of threads. Particularly if you centralise the configuration data that defines the number of threads in those pools.
@Duncan Yes that would mitigate the risks, but its still different from a global thread pool. Suppose I have one particular big thread pool that I use only once per week, I would "steal" those threads from the rest of the app most of the time.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.