12
$\begingroup$

I have a simple Mathematica program which writes some plots to image files for later conversion into a movie. Unfortunately, the program leaks so much memory that it quickly exhausts all 12G of RAM on my machine. The only way out is to quit the kernel(s).

I can't see why this program shouldn't use a bounded amount of memory. I've read through Debugging memory leaks which unfortunately hasn't helped - the only "heavy" symbol is 'data', whose size is fixed. I don't see what's growing!

Note that the same problem occurs regardless of whether the loop is Map or Do, Parallel- or not. The problem also occurs in both Mathematica 8 and 9, both under Linux.

Help?

(* number of frames *) n = 1000; (* just some bogus data *) makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}]; data = Array[makeData, n]; (* Export the frames *) ParallelMap[ Export[ "movie-" <> IntegerString[#, 10, 4] <> ".png", ListVectorPlot[data[[#]]]] &, Range[1, n]];

Edit:
Some more information: after running this code (the non-parallel version), I checked the memory usage of the processes involved. The percentages are out of 12GB, so both the frontend and the kernel are consuming quite a bit of memory, while PNG.exe is almost nonexistent.

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24575 gredner 20 0 4400 740 588 S 0 0.0 0:00.00 Mathematica 24646 gredner 20 0 3029m 2.3g 22m S 0 20.5 1:43.02 Mathematica 24655 gredner 20 0 3384m 1.6g 14m S 0 13.7 3:31.47 MathKernel 24984 gredner 20 0 387m 7364 2464 S 0 0.1 0:20.92 PNG.exe
Improve this question
$\endgroup$
14
  • $\begingroup$ Have you tried using ParallelDo rather than ParallelMap? I know it shouldn't make a difference, but some of these things are more involved than they seem. Also try just plain Do. Maybe your system is launching like, infinity kernels all of which are running 100 computations, so you may need to specify things like Method -> "EvaluationsPerKernel" -> 1 $\endgroup$ Commented May 1, 2013 at 19:08
  • $\begingroup$ I've now tried ParallelDo with the same outcome. $\endgroup$ Commented May 1, 2013 at 19:13
  • 2
    $\begingroup$ What Mathematica version are you using? I know from my own experience doing this exact sort of thing (exporting images) that there is an inexplicable memory issue in my version 8 which didn't occur at all in my V9 trial from the same exact copy-pasted code. $\endgroup$ Commented May 1, 2013 at 20:55
  • 3
    $\begingroup$ Have you checked if the MathKernel itself leaks memory or the spawned PNG.exe processes? $\endgroup$ Commented May 1, 2013 at 21:57
  • 1
    $\begingroup$ It could also be the front end that leaks, check that process as well. $\endgroup$ Commented May 1, 2013 at 23:13

2 Answers 2

Reset to default
3
$\begingroup$

Extended comment

I ran this code several times, with different values for n, and using Map or Do as well as ParallelMap. Each time a plot was created and exported, Mathematica's memory in use increased by about 1.7MB. We can call this a 'leak', but it seems to be by design.

So, with n as 1000, and allowing for some memory for the front end and the kernel to start with, 1000 plots uses up some but not all the available memory. (Hence my original answer - your code runs fine as it stands.)

When n reached 4000, my computer started to run out of memory — paging continuously, or 'thrashing'. Although it says 12GB total memory, the operating system needs some to work with as well.

I was unable to get to 10000 plots, for which my calculations suggest I'd need over 15GB of free memory devoted to Mathematica.

ParallelMap speeds up the process of generating the plots considerably, but it seems that each kernel uses free memory at the same rate as a single kernel, so the end result is that the computer starts to thrash sooner rather than later.

Anyway, my opinion is that there should be a way to make Mathematica forget about each plot once you've written it to disk.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Can you try increasing the number of frames (say increasing it by a factor of 10), re-running the experiment, and looking at memory usage again? $\endgroup$ Commented May 3, 2013 at 15:02
  • 1
    $\begingroup$ @GRedner might be able to some time, but my kernels are having a break for the weekend. (As am I...) $\endgroup$ Commented May 3, 2013 at 16:25
3
$\begingroup$

Another glorified comment

The leak is not in Export. It is in ListVectorPlot.

Compare

tt = Table[{y, -x}, {x, -3, 3, 0.2}, {y, -3, 3, 0.2}]; before = MemoryInUse[]; ListVectorPlot[tt]; after = MemoryInUse[]; after - before

with

before = MemoryInUse[]; Identity[tt]; after = MemoryInUse[]; after - before
Improve this answer
$\endgroup$
1
  • $\begingroup$ Just a (late) note for people hitting on this question: The problem seems to have been fixed in Mathematica 10.0.2.0. $\endgroup$ Commented Jan 19, 2015 at 17:10

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.