6
$\begingroup$

Is ParallelMap and Parallelize@Map expected to work with Sow and Reap construct.

This is a rather brief toy code to find all Primes just to force the use of Sow/Reap inside Map

Reap[Map[ If[PrimeQ@#, Sow@#] &, Range[10] ]][[2]]

and the output is, as expected

{{2, 3, 5, 7}}

But when I do 

Reap[Parallelize@Map[ If[PrimeQ@#, Sow@#] &, Range[1000] ]][[2]]

I got the output

{}

And the same was the output when I ran

Reap[ ParallelMap[ If[PrimeQ@#, Sow@#] &, Range[10] ] ][[2]]

2 questions :

1) How is this explainable ?

2) If this is the case, how do I efficiently I spawn parallel jobs which accumulate something ?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
8
$\begingroup$

I guess this will work! Check documentation for SetSharedFunction but be careful of the parallel overhead of following implementation. 

SetSharedFunction[PSow]; PSow[x_] := Sow[x]; Reap[Parallelize@Map[If[PrimeQ@#, PSow@#] &, Range[10]]][[2]]

{{2, 5, 3, 7}}

Improve this answer
$\endgroup$
8
$\begingroup$

I would recommend parallelizing these types of calculations using the ParallelCombine paradigm.

First, let's make a function out of your example:

fun = Function[list, Reap[Map[If[PrimeQ@#, Sow@#] &, list]][[2, 1]]]

The non-parallel version will be fun[Range[200]]. Parallelize it using 

ParallelCombine[ fun, Range[200], Join ]

I spelt out Join here because in other applications you may need a different function there. Here it can be omitted.

This is clearly more work than the (also valid) SetSharedFunction approach, but it is going to avoid some overhead. SetSharedFunction simply forces the function to be evaluated on the main kernel. If the equivalent of PrimeQ for your real calculation gives True for a sizeable fraction of the values, the SetSharedFunction-based parallelization won't buy much performance, or might even be slower than a serial computation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for sueggesting ParallelCombine. An interesting find! $\endgroup$ Commented Oct 3, 2013 at 21:53
  • $\begingroup$ But how is this behavior of Reap@Map@Sow giving {} explainable? $\endgroup$ Commented Oct 3, 2013 at 21:54
  • 1
    $\begingroup$ @myaccount_ram You were running Sow in a subkernel and Reap in the main kernel, so it can't work. These are separate processes that do not share any memory. The main kernel merely submits evaluations to the subkernels and collects results. $\endgroup$ Commented Oct 3, 2013 at 21:59

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.