Skip to main content

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

6
  • Put a timer around the creation and starting of those worker processes. This task may just not be long enough to make parallelizing across processes worth the process creation overhead. Note that your 5 processes cumulatively spent 6.6 seconds working, but overlapped their work such that the elapsed time was only 5.5 seconds. Commented Jan 25, 2016 at 0:03
  • What is astore? It's hard to tell if its an operation that can be well-parallelized. Commented Jan 25, 2016 at 0:16
  • On looking at the documentation a bit, I think the problem is that Queue (or any communication between the processes spawned using multiprocessing, I think), needs the data to be pickled. The added overhead from that makes the parallel version slow. @PaulMcGuire: I think the pickling is included in times of the individual tasks, which is why their aggregate is slower than the serial function. Commented Jan 25, 2016 at 0:26
  • @Will: astore is a store of data, the atomic version of what I am reading. In the "serial" version, I read the data in one process (in a for loop) the parallel version is above. Commented Jan 25, 2016 at 0:28
  • 1
    Did some profiling, it seems the parallel version is much faster without passing everything back to the master process. So you are correct in that it's probably the problem of the Queue, which is based on interprocess communication. Commented Jan 25, 2016 at 2:23