Is there a way to have limits for (let's say) apache2, that limits each forked child worker process to use max X memory, and not the entire process tree (apache2 and all of its forked children) to max X memory?
I have tried solutions with cgroups/systemd but everything limits the main process and all of its children combined, and not each child individually.
I am looking for a solution not only for apache2 but for all processes that spawn children.
Just improve the source code of your application (maybe Apache2 in your case, which is open source) to call setrlimit(2) after the successful call to fork(2), but before the call to execve(2).
Because you really want to have the setrlimit system call done after the fork (what you have then set remains till some further call to setrlimit, perhaps by the ulimit builtin of your shell).
Take some time to understand better how Linux syscalls(2) work (in particular fork & execve & setrlimit) and how and when you should use them. I recommend reading a good Linux system programming book, such as ALP.
Therefore, what you want to achieve is not realistic in general, unless you accept to change slightly every program you are using. The Unix philosophy which inspired the design of fork, execve , setrlimit forbids that.
Of course, you could in principle do insane LD_PRELOAD tricks to overload the behavior of fork (calling setrlimit after it returns 0) or of execve (calling setrlimit syscall before it) for dynamically linked executables (or equivalently, patch your own variant of libc.so), but that is not reasonable. See also file(1) or ldd(1) to detect such dynamically linked executables.
You could however use the ulimit shell builtin to put limits on Apache2 and all its children. And you might dive into Apache2 documentation, it has tons of settings regarding its child processes.
