8
$\begingroup$

I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files AppendingTo a List. The problem is that my computer keeps running out of memory even though the memory available should be more than enough to store the whole set. When researching the issue I discovered a strange behaviour (see the code below). The list produced by AppendTo consumes much more memory than would correspond to the size of an integer. I presume that, in the latter List, Mathematica uses some different kind of objects to store the data.

Any suggestions on how to reduce the memory usage?

In[78]:= ByteCount[Range[1, 10000]] Out[78]= 40168 In[79]:= data = List[]; For[i = 0, i &lt 10000, ++i, AppendTo[data, i]; ]; In[81]:= ByteCount[data] Out[81]= 320040
Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ In addition to Rojo's answer, for better performance avoid AppendTo and do something similar to this: Join @@ (Import /@ FileNames["data*.txt"]). Also avoid For if you can. $\endgroup$ Commented May 29, 2012 at 14:45

1 Answer 1

Reset to default
16
$\begingroup$

You're looking to pack your array.

It's done with Developer`ToPackedArray

dataPacked = Developer`ToPackedArray[data]; ByteCount[dataPacked] 40168

There's more on this following the link.

The links were taken from the packed array section of this answer.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You beat me, have an upvote. $\endgroup$ Commented May 29, 2012 at 13:15

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.