Limit available memory to a program

cgroups is the feature or modern linux kernels which allow you to limit resources like memory for group of processes (or for single process with threads). More about cgroups: https://en.wikipedia.org/wiki/Cgroups https://man7.org/linux/man-pages/man7/cgroups.7.html

The cgroups feature should be already enabled in your ubuntu 18.04 kernels. There are some descriptions how to use cgroups to limit memory:

• for systemd managed service: https://www.paranoids.at/cgroup-ubuntu-18-04-howto/,
• for docker containers docker run --memory=1G ….
• for custom process tree with cgcreate/cgset/cgexec commands: https://dev.to/vga/how-to-see-and-limit-memory-consumption-of-an-application-5bfl something like

# Create a group for memory named “limited_group_1” cgcreate -g "memory:limited_group_1" -t USERNAME:GROUPNAME # Specify memory limit to 1G for this group cgset -r memory.limit_in_bytes=1G "limited_group_1" # Launch the application in this group cgexec -g "memory:limited_group_1" ./YOUR_APPLICATION # If needed, we can remove the group cgdelete "memory:limited_group_1" 

https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process/279175#279175 was also mentioned in https://dev.to/vga/how-to-see-and-limit-memory-consumption-of-an-application-5bfl

PS: Default memory allocators in older glibc versions (malloc, new) has awful behavior for freed regions: they are not returned back without periodic malloc_trim() library calls. You should try to link your application with libjemalloc or libtcmalloc which will replace malloc implementation of glibc with some code better in memory returning.