###Preamble
It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are accumulated but not cleared.
I will show here one rather obscure case of leaking of local `Module` variables, which happens when the variable is referenced by some object / symbol, external w.r.t. its scope. In such cases, such variables are not garbage-collected even after the symbols referencing them get `Remove`-d.
###One subtle case with a memory leak
MemoryInUse[]
> 17350016
$HistoryLength = 0;
Module[{g},
Module[{f},
g[x_] := f[x];
Do[f[i] = Range[i], {i, 5000}];
];
g[1]]
> {1}
MemoryInUse[]
> 72351376
One way to ensure that this does not happen is to insert `Clear[f]` at the end of the inner `Module`. There are more advanced ways to prevent such things as well. I may elaborate on those at some later time.
###Monitoring symbols
So, one good place to start is to call
Names["Global`*"]
> {"f", "f$", "f$119", "g", "i", "x", "x$"}
or whatever main context you are using (or other contexts, if you create symbols there), and watch for some symbols with high memory usage. In this particular case, the culprit it `f$119`.
Here are some utility functions which may help with monitoring symbols:
Clear[$globalProperties];
$globalProperties =
{OwnValues, DownValues, SubValues, UpValues, NValues,
FormatValues, Options, DefaultValues, Attributes, Messages};
ClearAll[getDefinitions];
SetAttributes[getDefinitions, HoldAllComplete];
getDefinitions[s_Symbol] :=
Flatten@Through[
Map[Function[prop, Function[sym, prop[sym], HoldAll]],
$globalProperties
][Unevaluated[s]]
];
ClearAll[symbolMemoryUsage];
symbolMemoryUsage[sname_String] :=
ToExpression[sname, InputForm,
Function[s, ByteCount[getDefinitions[s]], HoldAllComplete]
];
ClearAll[heavySymbols];
heavySymbols[context_, sizeLim_: 10^6] :=
Pick[#, UnitStep[# - sizeLim] &@Map[symbolMemoryUsage, #], 1] &@
Names[context <> "*"];
For example, calling
heavySymbols["Global`"]
returns
> {f$119}