36
$\begingroup$

This is the most common pattern to compute a table of results:

Table[function[p], {p, parameters}]

(regardless of how it's implemented, it could be a Map)

The problem with this is that if the calculation is interrupted before it's finished, the partial results will be lost.

We can do this in a safely interruptible way like so:

Do[AppendTo[results, {p, function[p]}], {p, parameters}]

If this calculation is interrupted before it's finished, the intermediate results are still preserved. We can easily restart the calculation later, for those parameter values only for which function[] hasn't been run yet.

Question: What is the best way to achieve this when running calculations in parallel?

Assume that function[] is expensive to calculate and that the calculation time may be different for different parameter values. The parallel jobs must be submitted in a way to make best use of the CPU. The result collection must not be shared between the parallel kernels as it may be a very large variable (i.e. I don't want as many copies of it in memory as there are kernels)

Motivation: I need this because I want to be able to make my calculations time constrained. I want to run the function for as many values as possible during the night. In the morning I want to stop it and see what I got, and decide whether to continue or not.

Notes:

I'm sure people will mention that AppendTo is inefficient and is best avoided in a loop. I think this is not an issue here (considering that the calculations run on the subkernels and function[] is expensive). It was just the simplest way to illustrate the problem. There could be other ways to collect results, e.g. using a linked list, and flattening it out later. Sow/Reap is not applicable here because they don't make it possible to interrupt the calculation.

About the long running time: The most expensive part of the calculations I'm running are in C++ and called through LibraryLink, but they still take a very long time to finish.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
29
$\begingroup$

Regarding using Sow instead of AppendTo, you may find this trick useful:

Last[Last[Reap[CheckAbort[Do[Pause[0.1]; Sow[x], {x, 30}], ignored]]]]

(Try running this and aborting it partway through. It runs for 3 seconds due to the Pause[0.1] commands.)

Do is used instead of Table, and the results are returned with Sow. The CheckAbort catches when you abort your computation partway through and does the useful tidying up (in this case, returning something, anything, to the enclosing Reap).

You can combine this with a version of Sow that always run on the master kernel:

SetSharedFunction[ParallelSow]; ParallelSow[expr_] := Sow[expr]

(Tangentially related blog post I did: http://blog.wolfram.com/2011/04/20/mathematica-qa-sow-reap-and-parallel-programming/)

Then you could use this parallelized version:

In[3]:= Last[ Last[Reap[ CheckAbort[ParallelDo[Pause[0.1]; ParallelSow[x], {x, 30}], ignored]]]] Out[3]= {6, 1, 7, 2, 8, 3, 9, 4, 10, 5, 16, 11, 17, 12, 18, 13, 19, \ 14, 20, 15, 21, 26, 22, 27, 23, 28, 24, 29, 25, 30}

However, as you can see, the results come in in an unpredictable order so something slightly cleverer is in order. Here is one way (probably not the best but the first thing I thought of):

In[5]:= Catch[ Last[Last[ Reap[CheckAbort[ Throw[ParallelTable[Pause[0.1]; ParallelSow[x], {x, 30}]], ignored]]]]] Out[5]= {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, \ 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30}

The Throw is used to jump outside the Reap if the ParallelTable finishes. (Getting messy!)

To be safe this should be wrapped up in a function and tags (a.k.a. the optional second argument) should be used on the Throw, Catch, Sow, Reap.

Improve this answer
$\endgroup$
8
  • $\begingroup$ It's not a problem if the results come in in an unpredictable order. All I need is to run function for each value of p, I can always correct the order later if it necessary (but in my practical problem it probably won't be) $\endgroup$ Commented Jan 20, 2012 at 8:49
  • 2
    $\begingroup$ Andrew, it is not completely clear to me why ParallelSow will always run Sow on the master kernel if it is set as a shared function. It is not obvious from looking at the documentation of SetSharedFunction. Do shared functions always get evaluated on the master kernel, even when called from a parallel kernel? $\endgroup$ Commented Jan 20, 2012 at 17:28
  • 3
    $\begingroup$ Yes they do always get evaluated on the master kernel. You're right, the main page for SetSharedFunction doesn't seem to directly state this. This tutorial comes closer to stating it: reference.wolfram.com/mathematica/ParallelTools/tutorial/… $\endgroup$ Commented Jan 20, 2012 at 17:55
  • 2
    $\begingroup$ This single piece of information just solved (or simplified) many more parallel-related problems I had! $\endgroup$ Commented Jan 20, 2012 at 20:13
  • $\begingroup$ Does SetSharedVariable act the same way? I objected to the other answer here because I thought that after doing SetSharedVariable[var], each kernel would have a separate copy of var(meaning $KernelCount + 1 copies in memory). Is this really the case? If not, does var get transferred to a parallel kernel at every access, then transferred back (meaning temporary duplication of var)? It's important to underatnd this to be able to optimize memory usage and performance. $\endgroup$ Commented Jan 20, 2012 at 23:58
9
$\begingroup$

Here's another possible way using dynamic, first the serial version:

timeLeft[start_, frac_] := With[ {past = AbsoluteTime[] - start}, If[frac == 0 || past < 1, "-", Floor[past/frac-past]] ]; SafeMap[func_, list_] := DynamicModule[ {len, size, abortedQ, lastresult, starttime, n}, len = Length[list]; size = 0; abortedQ = False; lastresult; starttime = AbsoluteTime[]; n = 0; Monitor[ Table[ If[TrueQ[abortedQ], $Aborted, lastresult = func[list[[n]]]; size += ByteCount[lastresult]; lastresult ], {n, Range[len]} ], Refresh[Panel@Column[{ ProgressIndicator[n/len, ImageSize -> 350], Row[{Button["Abort", abortedQ = True], Grid[{{"Element", "Memory (kB)", "Time left (s)"}, {StringForm["``/``", n, len], ToString @ NumberForm[size/10.^3, {3, 1}], ToString @ timeLeft[starttime, n/len]} }, Spacings -> {1, 1}, ItemSize -> {10, 1}, Dividers -> Center ]}, Spacer[5]] }], UpdateInterval -> 0.5, TrackedSymbols -> {} ] ] ]

Try running the following example:

results = SafeMap[(Pause[1];Plot[x^2-#^2==0,{x,0,10}])&, Table[i, {i,10}]]

The panel displays the current element being evaluated as well as the total memory used thus far, and the estimated time remaining. If you click "abort" you get the partially generated list. For parallelization, only a minor change is needed:

Clear[SafeMap]; SafeMap[func_, list_, ker_:$KernelCount] := DynamicModule[ {len, bag, size, lastresults, starttime, n, results, t}, len = Length[list]; size = 0; starttime = AbsoluteTime[]; results = {}; SetSharedVariable[results, size]; Monitor[ t = Table[ParallelSubmit[{i}, With[{r = func[list[[i]]]}, size += ByteCount[r]; AppendTo[results, {i, r}]]], {i, Range[len]}]; CheckAbort[WaitAll[t], AbortKernels[]]; SortBy[results, First] , Dynamic@Refresh[Panel @ Column[{ ProgressIndicator[Length[results]/len, ImageSize -> 350], Row[{Button["Abort", AbortKernels[]], Grid[{{"Element", "Memory (kB)", "Time left (s)"}, {StringForm["``/``", Length[results], len], ToString @ NumberForm[size/10.^3, {3, 1}], ToString @ timeLeft[starttime, Length[results]/len]} }, Spacings -> {1, 1}, ItemSize -> {10, 1}, Dividers -> Center ]}, Spacer[5]] }], UpdateInterval -> 0.1, TrackedSymbols -> {} ] ] ]

This allows you to abort by CMD+. or by pressing the button. For example

SafeMap[(Pause[1]; #^2)&, Table[i, {i, 30}], 4]
Improve this answer
$\endgroup$
3
  • $\begingroup$ Nice progress indicator! :-) Can you make it work for parallel calculations? (The main problem when I asked the question was to preserve all results computed so far when using several subkernels, e.g. ParallelTable, ParallelMap, ParallelDo, etc.) $\endgroup$ Commented May 24, 2012 at 19:39
  • $\begingroup$ I amended the function to use ParallelSubmit but I think the other functions wouldn't be difficult to use instead. $\endgroup$ Commented May 24, 2012 at 22:10
  • $\begingroup$ @Szabolcs can you think of other questions like this where a nice GUI interface would be appropriate? $\endgroup$ Commented May 24, 2012 at 22:15
7
$\begingroup$

In this answer, I have implemeted Abort-able Table based on Do loops and Reap-Sow, using a technique similar to what is described in the answer of @Andrew (look at the bottom of the post for the second, more compact implementation). It seems that all you have to do is take that code and replace Do with ParallelDo.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I'd forgotten about that answer! Have you ever thought about compiling your own Mathematica recipe book? $\endgroup$ Commented Jan 20, 2012 at 23:57
  • 2
    $\begingroup$ @Simon Yes, I thought of it - but I still wanted to avoid the recipe format. My idea is to select the best "recipes" and include them in my new book, but in a way which would not take them out of context. Many of these code snippets represent complex code and / or concepts, so in a recipe format I will either have to not explain them properly, or repeat the same rather complex and / or long explanations over and over. This is one of the major challenges of writing an advanced book with the goal of making it accessible, I think. $\endgroup$ Commented Jan 21, 2012 at 9:09
  • $\begingroup$ @Simon By the way, I checked the code and the situtation is not as simple as I thought. My suggestion will also have to use Andrew's ParallelSow, and even then, is quite sow for some reason. I will look into that as time permits. $\endgroup$ Commented Jan 21, 2012 at 9:11
6
$\begingroup$

Have you tried ParallelDo? Here's an example implementation:

First, we need a function to simulate a lengthy calculation. f randomly generates a number $0<p<1$; if the number is $>0.5$ it calls Abort[], otherwise it returns the number afterwards.

f := If[# > .5, Abort[], #] &@RandomReal[]

Generate some dummy data (not sure whether SetSharedVariable is necessary, at least it doesn't hurt),

data = ConstantArray[0, 10] SetSharedVariable[results]; results = {};

Launch the calculation and return results,

ParallelDo[AppendTo[results, f], {i, 10}] results

The program aborts the calculation (almost) every time, and then prints the values calculated.

enter image description here

Note that this method may become very ineffective for longer result lists, as they're stored as fixed-size arrays internally. In that case.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I'd prefer not sharing the result collection between kernels as it may be a very large data structure. $\endgroup$ Commented Jan 20, 2012 at 0:06
  • 1
    $\begingroup$ Then you'll have to create a result variable for each individual kernel and combine them afterwards. What you want to have is a single object filled by multiple kernels after all, so it has to be shared somehow by them. $\endgroup$ Commented Jan 20, 2012 at 0:08
  • $\begingroup$ The single object that holds the result doesn't really need to be shared between kernels (i.e. I don't want all of them to have a full copy of it, as it happens in the SetSharedVariable method you describe). I would simply like to return results from each kernel, and let the main kernel add them to the result collection. It is not quite trivial how to do this. I'll edit my question and clarify about not sharing the result collection (should have done it before) $\endgroup$ Commented Jan 20, 2012 at 0:14
  • $\begingroup$ (Actually if I do this in practice and leave the calculation running indefinitely, I'll need to protect against running out of memory.) $\endgroup$ Commented Jan 20, 2012 at 0:18
  • $\begingroup$ David, though I never used it before, I think this needs CriticalSection to make it safe. $\endgroup$ Commented Jan 20, 2012 at 0:29

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.