10

I've been puzzling over a good implementation for this for a while. I have a program that does a long-running I/O operation (downloading a file) and a long-running CPU operation (parsing its contents). To improve efficiency, I wanted to have one thread pool perform the I/O section and have it pass tasks to another thread pool that performs the CPU section. That way, it's unlikely that all threads will be stuck either hogging the I/O or hogging the CPU. Depending on the file, one operation will likely take longer than the other.

What's the best way to perform a hand-off between thread pools? Assuming I'm using Java with two ExecutorServices, like so:

int threads = Runtime.getRuntime().availableProcessors(); ExecutorService ioService = Executors.newFixedThreadPool(threads); ExecutorService cpuService = Executors.newFixedThreadPool(threads); public BigFile ioTask(){ return Connection.downloadBigFile(); } public void cpuTask(BigFile bigFile){ processBigFile(bigFile); }

What's the best way to have a task running under ioService to add a task to cpuService while passing data into it?

What I've considered:

  • Submit a Runnable to cpuService that executes a Callable in ioService. Blocks cpuService while running the Callable.
  • Submit a Runnable to ioService that executes a Runnable in cpuService. Blocks ioService while running the second Runnable.
  • Create a Runnable implementation that has an IOTaskResult constructor, allowing ioService runnables to submit these implementations to the cpuService. Turns a reusable processing object into a consumable process object, which adds more overhead.
  • Add a method to generate a Runnable to the CPU task processor. This feels like a workaround and kind of "hack-y", considering I have a method I can call and the Runnable just wraps the method and fills in the parameters for me.

Is there a good way of handling this? I'd love to hear some thoughts.

Improve this question

2 Answers 2

Reset to default
8

To improve efficiency, I wanted to have one thread pool perform the I/O section and have it pass tasks to another thread pool that performs the CPU section. That way, it's unlikely that all threads will be stuck either hogging the I/O or hogging the CPU.

You don't actually need two thread pools to pull this off, and doing it that way can be counterproductive from execution and complexity standpoints.

First, threads are cheap, and you shouldn't hesitate to use one per download-and-process job if you're not going to be running hundreds of them. The I/O-bound portion of the process is going to spend a lot of its time asleep, and you really want to be able to grab a few cycles of CPU when the next frame arrives and then go back to sleep. I'm going to hand-wave the slowdown that happens if you have lots of files being downloaded simultaneously and will come back to it later.

Based on your current design, it looks like that in this case, your definition of efficiency means that files downloaded first get processed first.* This means you want to put a core on each processing job and keep it there until it's finished. You can regulate this using a semaphore that counts the number of cores in the system:

int num_cores = Runtime.getRuntime().availableProcessors(); core = new Semaphore(num_cores);

This will allow the first num_cores threads that want to use a core to acquire one, and any others will have to wait until one is released. With a way to regulate the number of threads using cores, the entire process can be cut back to a single routine:

void downloadFileAndProcess(String url) { BigFile bigFile = Connection.downloadBigFile(url); core.acquire(); // Hold here for a core // TODO: Wrap in try/catch/finally for error resistance. processBigFile(bigFile); core.release(); }

The number of simultaneous downloads can be regulated with a second semaphore that's acquired and released around the first statement.

*If that isn't the case, there's no efficiency to be gained by limiting the number of threads as long as you keep the number running down to the number of cores. If you have n CPU cycles of work to do and c cores, it's going to take n/c cycles' worth of time total no matter how you slice it up.

Improve this answer
12
  • The processing order doesn't matter. If it did, I'd put up with the efficiency drop for the sake of maintaining order. I like the idea that if a file is taking an unusually long time to process, other files can still download. I see where you're coming from, as this makes things much simpler implementation-wise, but I'm not sure I understand the semaphore bit; wouldn't a thread pool backed by an executor service be much more efficient at limiting access, since tasks are queued rather than started and set to wait? Commented Apr 15, 2017 at 3:01
  • 1
    @ndm13 Executors are just wrappers that make using thread pools a more asynchronous affair. They don't do anything to solve your problem, which is narrowing a lot of download threads into a few processing threads. The semaphore takes care of that by counting the number of available cores and putting the after-download process on hold until one becomes available. Commented Apr 15, 2017 at 3:30
  • 1
    @ndm13 None that I can think of. Idle threads consume no resources and a well-written executor should do its job as long as whatever container it uses to hold the pool doesn't overflow. (Not very likely.) Commented Apr 15, 2017 at 18:50
  • 1
    "threads are cheap" [Compared to IO] ? Commented Nov 1, 2017 at 14:39
  • 2
    @TheCatWhisperer Cheaper than writing and maintaining software to pass the parts of the job around among thread pools. Commented Nov 1, 2017 at 14:51
6

ioService must not be of availableCores() size. Its size depends on network characteristics of a computer and may even be unbounded.

Don't block or wait, this wastes time. Submit a Runnable to ioService, download file, post another Runnable to cpuService and parse its contents there.

You can also try using nio2's asynchronous IO which does not block threads, so you may end up using only cpuService. You may also try using ForkJoinPool which uses lock-free queues.

Improve this answer
2
  • 2
    This doesn't seem to add much value to a year old question with an accepted answer, otherwise I would suggest expanding your answer. Commented May 25, 2018 at 21:04
  • 1
    This is the correct answer though. Read this for more info. Commented May 4, 2020 at 4:16

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.