2
$\begingroup$

I try dynamically to add new keys and update the values of existing keys in given dataset (Association). I implement this by the following code:

test = Association[{a -> 1, b -> 2, c -> 3}] (If[MissingQ[test[#]], AssociateTo[test, # -> 1], AssociateTo[test, # -> (test[#] + 1)]]) & /@ {a, b, c, d, e} {<|a -> 2, b -> 2, c -> 3|>, <|a -> 2, b -> 3, c -> 3|>, <|a -> 2, b -> 3, c -> 4|>, <|a -> 2, b -> 3, c -> 4, d -> 1|>, <|a -> 2, b -> 3, c -> 4, d -> 1, e -> 1|>}

But instead of updating the dataset the result is replications of the dataset with updates

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Map (/@) returns the results of the iterations of AssociateTo in a list and that is confusing you. You can suppress the output by using Scan instead of Map. Actually, the output of the mapped function is not of interest to you. What matters is the value of test afterwards, as AssociateTo uses call by reference.

test = Association[{a -> 1, b -> 2, c -> 3}] Scan[ (If[MissingQ[test[#]], AssociateTo[test, # -> 1], AssociateTo[test, # -> (test[#] + 1)] ]) &, {a, b, c, d, e} ]; test

<|a -> 1, b -> 2, c -> 3|>

<|a -> 2, b -> 3, c -> 4, d -> 1, e -> 1|>

Using Scan instead of Map increases performance since for returning the intermediate results of AssociateTo requires copying them. But that is what you actually try to avoid by using AssociateTo instead of using Associate recursively. Here is an illustration of the performance difference:

n = 100000; a = b = AssociationThread[Range[n], RandomInteger[10, n]]; rand = RandomInteger[n, n]; a == b Scan[ (If[MissingQ[a[#]], AssociateTo[a, # -> 1], AssociateTo[a, # -> (a[#] + 1)] ]) &, rand ]; // AbsoluteTiming //First Map[ (If[MissingQ[b[#]], AssociateTo[b, # -> 1], AssociateTo[b, # -> (b[#] + 1)] ]) &, rand ]; // AbsoluteTiming //First a == b

0.30575

0.517976

True

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you. It is a great solution; Scan function is parallelization function (as well as Map) then I can speed up the computation (I have a huge dataset ~500 MByte). $\endgroup$ Commented Apr 29, 2018 at 10:07
  • $\begingroup$ Hm. Beware: If you parallelize that, all subprocesses have to write into the same data structure (the association). That means that the subprocesses have to communicate a lot with each other in order to avoid race conditions. This can slow down parallelization so much that is even slower than unparallelized code. Try it but don't be surprised if it doesn't work. If it is really only about counting occurences of expressions in a list, you might also be interested in Tally. $\endgroup$ Commented Apr 29, 2018 at 10:17
  • $\begingroup$ I thought so :) Anyways, if you are going to parallelize that, try to build one association per kernel and Merge the results in the end. $\endgroup$ Commented Apr 29, 2018 at 10:35
  • $\begingroup$ thanks a lot. I agree with you that a parallelization may run slower than a sequence computation. I try to do something more interesting than to count the frequency of expressions in a list:). I want to create a bag of bigrams in given text (more specific a set of documents) and calculate the TF-IDF vector of each document. Maybe you can suggest me a good way to implement this problem? $\endgroup$ Commented Apr 29, 2018 at 10:40
  • $\begingroup$ Yes I can. Could you please start a new question for that, also shortly explaining what a bigram is and what you mean by TF-IDF vector? That would make it also easier for me to post code that does not into a comment. $\endgroup$ Commented Apr 29, 2018 at 10:48

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.