I believe JoinAcross may have a memory leak but would appreciate some help looking at it. It appears to be small when used just once but when iterated a lot it eventually crashes my wolframscripts. I've tried Share[] and ClearSystemCache[] but they don't fix the issue. I've written the below toy example to show the leak:
SeedRandom[1234]; data1 = Table[<|"Date" -> DatePlus[Today, t],"Value" -> RandomReal[]|>, {t,1, 10000}]; data2 = Table[<|"Date" -> DatePlus[Today, t],"Value" -> RandomReal[]|>, {t,1, 10000}]; Do[ memoryBefore = MemoryInUse[]; data = JoinAcross[data1, data2, "Date", "Inner", KeyCollisionFunction -> ({"Data1" <> #, "Data2" <> #} &)]; ClearAll[data]; memoryAfter = MemoryInUse[]; memoryChange = memoryAfter - memoryBefore; memoryInMegabytes = N[memoryChange/2^20]; WriteLine[$Output, "Memory Change (MB): " <> ToString@memoryInMegabytes]; , {10} ]; This gives the output:
(* Memory Change (MB): 7.17958 Memory Change (MB): 4.73375 Memory Change (MB): 4.73399 Memory Change (MB): 4.7343 Memory Change (MB): 4.7346 Memory Change (MB): 4.7346 Memory Change (MB): 4.7346 Memory Change (MB): 4.7346 Memory Change (MB): 4.7346 Memory Change (MB): 4.73454 *) Thank you for any help!
$HistoryLength = 0will give some remedy. However, a small increase is still visible. $\endgroup$$HistoryLength = 0and it gives me more time but still does eventually crash. $\endgroup$