###Preamble
It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are accumulated but not cleared.
I will show here one rather obscure case of leaking of local Module variables, which happens when the variable is referenced by some object / symbol, external w.r.t. its scope. In such cases, such variables are not garbage-collected even after the symbols referencing them get Remove-d, in case if they get assigned DownValues, SubValues or UpValues (OwnValues are ok).
###One subtle case with a memory leak
MemoryInUse[] 17350016
$HistoryLength = 0; Module[{g}, Module[{f}, g[x_] := f[x]; Do[f[i] = Range[i], {i, 5000}]; ]; g[1]] {1}
MemoryInUse[] 72351376
One way to ensure that this does not happen is to insert Clear[f] at the end of the outer Module, storing the result in a separate variable and returning it afterwards. There are more advanced ways to prevent such things as well. I may elaborate on those at some later time.
###Memory leaks assocaited with UI-building
One common cause of memory leaks which is often ovelooked is when some local symbols make it into UI elements. The problem is that UI elements are Mathematica expressions, which do reference those symbols, and therefore, they are not garbage-collected.
Here is an example I borrowed from this thread
memModule[] := Module[{data, memBefore, mu}, mu := Grid[{{"Memory in use: ", MemoryInUse[]/(2^30.), "GB"}}]; memBefore = mu; data = RandomReal[1, {300000, 20}]; DynamicModule[{d1}, d1 := data[[1]]; Panel[Grid[{{memBefore}, {mu}}]] , UnsavedVariables -> {dl} ] ]; Now, every time when it gets executed, more memory is being leaked:
memModule[] memModule[] memModule[] Please see my answer in the linked thread for one way out, in this particular case. Generally, this is something to watch out for.
###Monitoring symbols
So, one good place to start is to call
Names["Global`*"] {"f", "f$", "f$119", "g", "i", "x", "x$"}
or whatever main context you are using (or other contexts, if you create symbols there), and watch for some symbols with high memory usage. In this particular case, the culprit it f$119.
Here are some utility functions which may help with monitoring symbols:
Clear[$globalProperties]; $globalProperties = {OwnValues, DownValues, SubValues, UpValues, NValues, FormatValues, Options, DefaultValues, Attributes, Messages}; ClearAll[getDefinitions]; SetAttributes[getDefinitions, HoldAllComplete]; getDefinitions[s_Symbol] := Flatten@Through[ Map[Function[prop, Function[sym, prop[sym], HoldAll]], $globalProperties ][Unevaluated[s]] ]; ClearAll[symbolMemoryUsage]; symbolMemoryUsage[sname_String] := ToExpression[sname, InputForm, Function[s, ByteCount[getDefinitions[s]], HoldAllComplete] ]; ClearAll[heavySymbols]; heavySymbols[context_, sizeLim_: 10^6] := Pick[#, UnitStep[# - sizeLim] &@Map[symbolMemoryUsage, #], 1] &@ Names[context <> "*"]; For example, calling
heavySymbols["Global`"] returns
{f$119}