Memory leaks can be a real pain and quite frustrating to trace on a large scale system. I'd try to copy the entire server into a test environment, starting services one at a time to isolate the problem.

After you check each of the services (the user mode processes) separately and individually and still can't seem to find the source of the leak, you should check the kernel. Dealing with the kernel takes time and experienced hands, I'd recommend advising a kernel specialist.

Another possibility is the presence of a malware. Dealing with malware as entirely different opera.

Sometimes there aren't shortcuts :\

Memory leaks can be a real pain and quite frustrating to trace on a large scale system. I'd try to copy the entire server into a test environment, starting services one at a time to isolate the problem.

After you check each of the services (the user mode processes) separately and individually and still can't seem to find the source of the leak, you should check the kernel. Dealing with the kernel takes time and experienced hands, I'd recommend consulting a kernel specialist.

Another possibility is the presence of a malware. Dealing with malware is an entirely different opera.

Sometimes there aren't shortcuts :\