Skip to main content
Mathematica

Return to Answer

Bounty Awarded with 300 reputation awarded by Emilio Pisanty
added 857 characters in body
Source Link
Szabolcs
Szabolcs
  • 238.9k
  • 32
  • 653
  • 1.3k

In short, the reason is that SumCommand's (temporary) definition won't automatically get distributed to the parallel kernels because now they liveSumCommand lives in the package`Private` context, not Global` (after correcting a small mistake, see below). This means that SumCommand won't get evaluated to Sum on the subkernels. It gets returned as-is to the main kernel, where now SumCommand does have a definition, gets evaluated to Sum, which in turn gets evaluated to the desired result. But all the evaluation happens on the main kernel.

Aside: Note that Begin["Private`"] should be Begin["`Private`"] so that private symbols will go into package`Private` and not into Private` .

When you put the function into a package, itsthe context changes whichof all the auxiliary symbols (such as SumCommand and TableCommand) change. This prevents the distribution mechanism from kicking in. Package symbols, residingwhich reside in contexts other than Global` , do not get distributed by default. But This is to prevent the distribution of package symbol definitions to subkernels, which would break packages which must also do initialization (such as loading LibraryFunctions) in addition to issuing definitions. Instead packages should be properly loaded on each subkernel, which can be automated with ParallelNeeds.

Unfortunately I do not fully understand the distribution rules for contexts though ... you can read more at DistributedContexts and links within that page.

This ideatheory that the problem is that SumCommand doesn't get distributed can be verified by adding DistributeDefinitions[SumCommand] right after TableCommand = ParallelTable; SumCommand = Sum;. This will make it run fast again (but it is not a good workaround, see below).

TheOne problem with your use ofthe way Block is used here is that Block won't have any effect across kernels. It only workworks on the main kernel. Thus if we simply insert DistributeDefinitions[SumCommand] inside of the Block body, the definition will get distributed to subkernelthe subkernels, but it won't get cleared on the subkernels when the Block finishes. Instead it will persist even after function finishes. You can verify this with ParallelEvaluate[package`Private`SumCommand].

Instead I suggest never sending the symbol SumCommand to the subkernel in the first place. Just send Sum instead. One way to achieve this is with a With-definition (instead of a Block-definition), which does a direct replacement of SumCommand within the body of With.

(To avoid the red colouring you might consider using a different name for the With variable.)

This version is robust and runs fast.

In short, the reason is that SumCommand won't automatically get distributed to the parallel kernels because now they live in the package`Private` context, not Global` (after correcting a small mistake, see below). This means that SumCommand won't get evaluated to Sum on the subkernels. It gets returned as-is to the main kernel, where now SumCommand does have a definition, gets evaluated to Sum, which in turn gets evaluated to the desired result. But all the evaluation happens on the main kernel.

Note that Begin["Private`"] should be Begin["`Private`"] so that private symbols will go into package`Private` and not into Private` .

When you put the function into a package, its context changes which prevents the distribution mechanism from kicking in. Package symbols, residing in contexts other than Global` , do not get distributed by default. But I do not fully understand the distribution rules for contexts ... you can read more at DistributedContexts.

This idea can be verified by adding DistributeDefinitions[SumCommand] right after TableCommand = ParallelTable; SumCommand = Sum;.

The problem with your use of Block is that Block won't have any effect across kernels. It only work on the main kernel. Thus if we simply insert DistributeDefinitions[SumCommand], the definition will get distributed to subkernel, but it won't get cleared on subkernels when the Block finishes.

Instead I suggest never sending the symbol SumCommand to the subkernel in the first place. Just send Sum instead. One way to achieve this is with a With-definition (instead of a Block-definition)

(To avoid the red colouring you might consider using a different name for the With variable.)

In short, the reason is that SumCommand's (temporary) definition won't automatically get distributed to the parallel kernels because now SumCommand lives in the package`Private` context, not Global` . This means that SumCommand won't get evaluated to Sum on the subkernels. It gets returned as-is to the main kernel, where now SumCommand does have a definition, gets evaluated to Sum, which in turn gets evaluated to the desired result. But all the evaluation happens on the main kernel.

Aside: Note that Begin["Private`"] should be Begin["`Private`"] so that private symbols will go into package`Private` and not into Private` .

When you put the function into a package, the context of all the auxiliary symbols (such as SumCommand and TableCommand) change. This prevents the distribution mechanism from kicking in. Package symbols, which reside in contexts other than Global` , do not get distributed by default. This is to prevent the distribution of package symbol definitions to subkernels, which would break packages which must also do initialization (such as loading LibraryFunctions) in addition to issuing definitions. Instead packages should be properly loaded on each subkernel, which can be automated with ParallelNeeds.

Unfortunately I do not fully understand the distribution rules for contexts though ... you can read more at DistributedContexts and links within that page.

This theory that the problem is that SumCommand doesn't get distributed can be verified by adding DistributeDefinitions[SumCommand] right after TableCommand = ParallelTable; SumCommand = Sum;. This will make it run fast again (but it is not a good workaround, see below).

One problem with the way Block is used here is that Block won't have any effect across kernels. It only works on the main kernel. Thus if we simply insert DistributeDefinitions[SumCommand] inside of the Block body, the definition will get distributed to the subkernels, but it won't get cleared on the subkernels when the Block finishes. Instead it will persist even after function finishes. You can verify this with ParallelEvaluate[package`Private`SumCommand].

Instead I suggest never sending the symbol SumCommand to the subkernel in the first place. Just send Sum instead. One way to achieve this is with a With-definition (instead of a Block-definition), which does a direct replacement of SumCommand within the body of With.

(To avoid the red colouring you might consider using a different name for the With variable.)

This version is robust and runs fast.

Source Link
Szabolcs
Szabolcs
  • 238.9k
  • 32
  • 653
  • 1.3k

In short, the reason is that SumCommand won't automatically get distributed to the parallel kernels because now they live in the package`Private` context, not Global` (after correcting a small mistake, see below). This means that SumCommand won't get evaluated to Sum on the subkernels. It gets returned as-is to the main kernel, where now SumCommand does have a definition, gets evaluated to Sum, which in turn gets evaluated to the desired result. But all the evaluation happens on the main kernel.

Note that Begin["Private`"] should be Begin["`Private`"] so that private symbols will go into package`Private` and not into Private` .

Why is the function slow when in a package?

Most parallel functions, such as ParallelTable, automatically distribute the definitions of symbols. (The notable exception is ParallelEvaluate.)

When you put the function into a package, its context changes which prevents the distribution mechanism from kicking in. Package symbols, residing in contexts other than Global` , do not get distributed by default. But I do not fully understand the distribution rules for contexts ... you can read more at DistributedContexts.

This idea can be verified by adding DistributeDefinitions[SumCommand] right after TableCommand = ParallelTable; SumCommand = Sum;.

How to fix it?

I have never written packages which use parallelization, so I have no experience with this, and no experience with what the major pitfalls are. But I would not use DistributeDefinitions the way I suggested above (which was only for testing the theory that the problem is with distribution).

The problem with your use of Block is that Block won't have any effect across kernels. It only work on the main kernel. Thus if we simply insert DistributeDefinitions[SumCommand], the definition will get distributed to subkernel, but it won't get cleared on subkernels when the Block finishes.

Instead I suggest never sending the symbol SumCommand to the subkernel in the first place. Just send Sum instead. One way to achieve this is with a With-definition (instead of a Block-definition)

Here's the final code:

BeginPackage["package`"]; function::usage = "function[x] is a function to calculate stuff"; RunInParallel::usage = "RunInParallel is an option for function which determines whether it runs in parallel or not."; Begin["`Private`"]; Options[function] = {RunInParallel -> False}; function[x_, OptionsPattern[]] := Block[{TableCommand, SumCommand}, Which[ OptionValue[RunInParallel] === False, TableCommand = Table; SumCommand = Sum;, OptionValue[RunInParallel] === True, TableCommand = ParallelTable; SumCommand = Sum;, True, TableCommand = OptionValue[RunInParallel][[1]]; SumCommand = OptionValue[RunInParallel][[2]]; ]; With[{SumCommand=SumCommand}, TableCommand[ SumCommand[ BesselJ[0, 10^-9 k]/(n + x^k), {k, 0, 50000} ] , {n, 0, 12}] ] ] End[]; EndPackage[];

(To avoid the red colouring you might consider using a different name for the With variable.)