3
$\begingroup$

I have created an entity store for financial data, as follows:enter image description here

It generally works well, except that when I try to use parallelization, it fails. For example:

first distribute the definition of the store:

DistributeDefinitions[StockDataStore]

Then try to use ParallelTable:

 perfData = ParallelTable[ Performance[stock, formationDate, currentDate, {60, 12, 9, 6, 3}], {stock, universe[[1 ;; 20]]}]; Missing["UnknownType", "Stocks"],

etc etc

The code works fine if I use the regular Table function.

It seems as if parallelization doesn't work with EntityStores. or am I missing something?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You would need to register the entity store on all parallel kernels. $\endgroup$ Commented Oct 29, 2022 at 17:52
  • $\begingroup$ Yes that works. Thank you. $\endgroup$ Commented Oct 30, 2022 at 9:03

2 Answers 2

Reset to default
2
$\begingroup$

Converting comment into an answer for posterity:

It is not enough to distribute the definition of the new EntityStore to all parallel kernels, you must register the store so that EntityValue can know about it.

DistributeDefinitions[StockDataStore] ParallelEvaluate[EntityRegister[StockDataStore]]
Improve this answer
$\endgroup$
1
  • $\begingroup$ I am going to accept this answer, because it identified the problem and set me on the right path. But it's achingly slow, because DistributDefinitions takes an age to distribution the entity store to all the parallel kernels. In my case I gave up waiting more than 20 mins! There is a much faster procedure that I have given in my own answer. $\endgroup$ Commented Nov 11, 2022 at 12:08
0
$\begingroup$

Jason B. identified the key issue, which is that the entity store needs to be defined on each parallel kernel before it can be registered. But using DistributeDefinitions is achingly slow.

A much faster procedure is as follows:

  1. Download the Entity Store in each kernel (in this case from an mx file, for speed):

    ParallelEvaluate[ Get[StringJoin[dirEntityStore, "EquitiesEntityStore.mx"]]]

  2. Then register the store on each kernel, as before:

    ParallelEvaluate[EntityRegister[EquitiesEntityStore]]

    {{"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}}

NOTE: the important thing is to download and register the Entity Store on the parallel kernels BEFORE you do it on the main kernel.

I suspect this may be because if you register the store on the main kernel first, MMA will resort to using DistributeDefintions.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.