I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files AppendingTo[] a List[]. The problem is that my computer keeps running out of memory even though the memory available should be more than enough to store the whole set. When researching the issue I discovered a strange behaviour (see the code below). The list produced by AppendTo[] consumes much more memory than would correspond to the size of an integer. I presume that, in the latter List[], Mathematica uses some different kind of objects to store the data.

Any suggestions on how to reduce the memory usage?

  In[78]:= ByteCount[Range[1, 10000]]   Out[78]= 40168 In[79]:= data = List[]; For[i = 0, i < 10000, ++i, AppendTo[data, i]; ]; In[81]:= ByteCount[data]   Out[81]= 320040 

Thanks.

I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files AppendingTo a List. The problem is that my computer keeps running out of memory even though the memory available should be more than enough to store the whole set. When researching the issue I discovered a strange behaviour (see the code below). The list produced by AppendTo consumes much more memory than would correspond to the size of an integer. I presume that, in the latter List, Mathematica uses some different kind of objects to store the data.

Any suggestions on how to reduce the memory usage?

In[78]:= ByteCount[Range[1, 10000]] Out[78]= 40168 In[79]:= data = List[]; For[i = 0, i &lt 10000, ++i, AppendTo[data, i]; ]; In[81]:= ByteCount[data] Out[81]= 320040