Timeline for Debugging memory leaks
Current License: CC BY-SA 4.0
31 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 20, 2021 at 12:04 | comment | added | Leonid Shifrin | @Kvothe Excellent. Glad it was helpful. | |
| Jun 19, 2021 at 17:16 | comment | added | Kvothe | Thanks, indeed your update fixed the evaluation leak. | |
| Jun 18, 2021 at 11:58 | history | edited | Gustavo Delfino | CC BY-SA 4.0 | Typo |
| Jun 18, 2021 at 10:31 | history | edited | Szabolcs | CC BY-SA 4.0 | update markdown to the new StackExchange standard |
| Jun 18, 2021 at 9:48 | comment | added | Leonid Shifrin | @Kvothe There was indeed an evaluation leak coming from Options not holding its argument - which would show up for symbols with OwnValues, such as e.g. a := Print["***"] - then calling getDefinitions on a would lead to printing. I have made and edit that should fix it, check it out. Regarding parallel, it looks like you already found an answer. | |
| Jun 18, 2021 at 9:46 | history | edited | Leonid Shifrin | CC BY-SA 4.0 | Fixed evaluation leak due to Options not holding the arguments |
| Jun 18, 2021 at 9:26 | comment | added | Kvothe | Ah profiling memory usage in subkernels is simple you just need to use ParallelEvaluate so in this case for example ParallelEvaluate[heavySymbols["Global`", 10^6]] | |
| Jun 18, 2021 at 8:52 | comment | added | Kvothe | Also how does this work with different Wolfram Kernels? I can see, in Gnome-Monitor, a high memory usage (3.2GB) in 6 different kernels (I called 6 kernels in a ParallelMap so that part makes sense). Apparently these are still alive after the parallel computation has already finished. In addition I see the main Kernel which has an memory usage that is close to the value returned by MemoryInUse[] (~1GB). So do these debugging tools only show the usage in the main kernel. It seems my problem isn't there. | |
| Jun 18, 2021 at 8:45 | comment | added | Kvothe | When I run heavySymbols["Global`", 10^6] after some code some functions from my code start evaluating. Could some HoldAll be missing? Or any idea what might be causing this? | |
| May 30, 2012 at 14:55 | comment | added | Vince | @Ajasja The sticking point is in your first parenthetical remark. My inner products, though large, were very simple operations and did not leave residual temporaries (at the user level). Yet the size of the final product was much smaller than indicated in the memory profile. Though please take these observations carefully; it's been a long time since I've run this diagnostic. | |
| May 30, 2012 at 14:02 | comment | added | Ajasja | @Vince I think such a shape could be common. Some memory is allocated (that will not get released) and then some further processing allocates even more memory, but then frees it. BTW, you can't embed images in comments, you can only link to them (to the best of my knowledge). | |
| May 30, 2012 at 14:00 | vote | accept | Ajasja | ||
| May 30, 2012 at 12:14 | comment | added | Vince | @Ajasja No doubt it's at least a reliable relative measure. I needed a system-level metric. That is, how well math.exe would play with others in the system process list. Working Set was as close as I could get. (MemoryInUse[] always under-reported.) I'll mention again that, though the scale is different, our memory profiles have very similar shapes. I find that intriguing. But perhaps common, as I also obliquely commented above somewhere. | |
| May 30, 2012 at 8:30 | comment | added | Ajasja | @Vince I got the memory profile from pm = {} Dynamic@Refresh[pm = Append[pm, MemoryInUse[]]; ListPlot[pm/1024/1024, AxesLabel -> {"Time [s]", "Memory [MB]"}, PlotRange -> {0, All}], UpdateInterval -> 1, TrackedSymbols :> {}] so it should be reliable. | |
| May 30, 2012 at 3:28 | comment | added | Vince | @OleksandrR. The bit about "recycling" was something I got from WRI's tech support. Voodoo to me. | |
| May 30, 2012 at 3:25 | comment | added | Oleksandr R. | @Vince: in that case, working set seems like a reasonable metric, considering that a pessimistic estimate is probably better for this application. It's not very useful for assessing possible memory leaks though. Personally I'd tend not to put much faith in claims like "Mathematica tries to..." unless you know what they really mean in practice. Mathematica doesn't have a great deal of control over the OS's memory management policies, which vary by system load, OS version, and many other unpredictable factors. | |
| May 30, 2012 at 3:13 | comment | added | Vince | @OleksandrR. I was trying to find the total RAM footprint of jobs which I was running remotely on other users' Windows XP desktops (to maintain their performance). perfmon's "Process > Working Set" seemed to approximate that footprint best. Also, supposedly, MMA tries very hard to recycle memory it has already taken from the system. I didn't see that in my memory profiles. By your clarification, it seems that recycling would keep the Working Set down, since it would eliminate many system memory allocations. Again, that was not apparent in the profiles. But maybe I'm not following. | |
| May 30, 2012 at 1:59 | comment | added | Oleksandr R. | @Vince: working set is probably not a reliable indicator of actual memory usage. A program that repeatedly allocates and frees memory can result in a highly inflated working set metric, because in a system under a light memory load, the memory manager will just put freed pages on the standby list without actually freeing them in case they're allocated again soon afterward. Just because the working set or virtual size is large, it doesn't necessarily mean that the process is using all that memory. | |
| May 29, 2012 at 14:30 | comment | added | Vince | @Ajasja Buried in the link I included above, you can see that my problematic memory profile resembles yours. Not sure if all memory profiles of MMA functions look this way (if not flat). !memory profile. (How to display an image in a Comment?) | |
| May 29, 2012 at 14:19 | comment | added | Leonid Shifrin | @Vince Thanks Vince! Lots of great folks accumulated here, from Stack Overflow, Mathgroup, WRI, and elsewhere. I think what makes this place stand out and compare favorably to, say, Mathgroup (with all due respect for the latter), is the level of collaboration (by which I also mean a healthy competition, which is often very beneficial for everybody), and the sense of community as a whole, rather than a set of isolated experts. Hope to see a lot of your answers appearing here! | |
| May 29, 2012 at 13:49 | comment | added | Vince | @Leonid Thanks Leonid. It's a sporadic presence, and usually rushed. While I'm off-topic, let me say how impressed I am with what has been created here on SE for MMA. A lot of impressive dedication and substantial contribution. | |
| May 29, 2012 at 13:04 | comment | added | Leonid Shifrin | @Vince Hi Vince, nice to see that you are here! (I mean, I did not realize before that you are you :)). | |
| May 29, 2012 at 12:25 | comment | added | Vince | @Ajasja There does seem to be an unstoppable (inaccessible) memory sink somewhere deep in Mathematica. I've run into it with my own application. The only reliable way around it seems to be re-architecture (cf acl above). In my case, the sink was somewhere under a huge inner product. Leonid eliminated that for me, i.e. forums.wolfram.com/mathgroup/archive/2010/Jan/msg00829.html. | |
| May 29, 2012 at 12:16 | comment | added | Ajasja | @acl Brr... that would be a worst case scenario! I spent two weeks on this code and now I can't get a single result... | |
| May 29, 2012 at 12:14 | comment | added | Ajasja | @LeonidShifrin I don't create symbols in any other context. What I'm doing a lot is assigning to the same Module variable multiple times. For example data=rawdata; data=GetDiffrences[data]; data=Mean[data]... Return[data] But I belive this shuld be safe. I'm putting memory print statements all over my code as we speak... | |
| May 29, 2012 at 12:13 | comment | added | acl | @Ajasja I also had a situation with the kernel eating up gigabytes of memory, linearly increasing in time even though I never kept more than 1GB of data in symbols. I also had checked and no symbols had anywhere near the memory consumed. in the end I rewrote my program in a completely different way and now it doesn't eat memory. have no clue what it was. | |
| May 29, 2012 at 12:08 | history | edited | Leonid Shifrin | CC BY-SA 3.0 | Added a section on UI-related memory leaks |
| May 29, 2012 at 12:00 | comment | added | Leonid Shifrin | @Ajasja Well, then I am at a loss. Do you create symbols in other contexts? Or, do you return any expression involving local variables, or embed it into UI, for example? UI-s are another cause of memory leaks, see e.g. my answer in this thread | |
| May 29, 2012 at 11:55 | comment | added | Ajasja | +1 Thank you for the example. Now I understand the comment you made here. Unfortunately, there appear to be no heavy symbols. heavySymbols[""], returns {"alldata", "GeoProjectionData"} which is that data I loaded at the beginning and an always present built-in. | |
| May 29, 2012 at 11:50 | history | edited | Leonid Shifrin | CC BY-SA 3.0 | Corrected a wrong statement |
| May 29, 2012 at 11:39 | history | answered | Leonid Shifrin | CC BY-SA 3.0 |