15

I'm running a number of short-lived docker containers each of which does some memory-intensive batch processing. I'm looking for a way to find the peak memory usage each container hit while it was running. Knowing this will allow me to optimize the infrastructure I run these containers on for future runs.

One naive way to achieve this is redirecting the streaming output of docker stats to some file: docker stats container_id > stats.log. However, this requires running a process for each container and then sorting through very verbose logs to find the peak usage. I'm wondering if there's not an easier way.

Improve this question
0

3 Answers 3

Reset to default
17

If you are interested in the process with PID=1 inside the container, you can find the PID this process has on the host and then use:

grep VmPeak /proc/$PID/status

Example with a mongo container:

This container has a single process:

$ docker container exec -it mongo top -bn 1 top - 10:04:51 up 32 min, 0 users, load average: 0.36, 0.52, 0.55 Tasks: 2 total, 1 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.1 us, 2.0 sy, 0.3 ni, 90.6 id, 0.7 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem : 6103572 total, 2642744 free, 1352032 used, 2108796 buff/cache KiB Swap: 1942896 total, 1942896 free, 0 used. 4277928 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 64 root 20 0 38624 3116 2724 R 6.7 0.1 0:00.89 top 1 mongodb 20 0 1094540 80100 35916 S 0.0 1.3 0:22.51 mongod

To get the PID of this process from the host's perspective:

$ docker inspect -f '{{.State.Pid}}' mongo 2532

and finally:

$ grep VmPeak /proc/2532/status VmPeak: 1094540 kB

Links:

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

grep: /proc/36133/status: No such file or directory on macOS
Looks like on macOS you can obtain this data by reading the /proc thing inside the container.
5

docker stats has the --format option to customize the output. For example we can output only memory usage:

 docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID 516KiB / 8GiB

CONTAINER_ID in the command must be replaced with the ID of our container (which we can get with docker ps).

516KiB is the current memory usage of the container and 8GiB is the maximum memory that the container is allowed to use.

On GNU/Linux, we can filter out the memory usage from the output with cut and redirect the output into a file:

docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID | cut -d '/' -f 1 >>docker-stats

Finally, we can put this command into a while loop and add a one second delay between iterations:

while true; do docker stats --no-stream --format '{{.MemUsage}}' CONTAINER_ID | cut -d '/' -f 1 >>docker-stats; sleep 1; done

This way, in the docker-stats file, we will get a list of memory usage values of the container with one second resolution. Of course, if a peak lasts less than one second we may miss it.

Docker stats docs.

Improve this answer

Comments

0

(The following answer is inspired by the information given in an "Is it possible to implement max_usage_in_bytes for cgroup v2?" answer)

If docker (the service) was started from systemd on a 5.19 or later Linux kernel, you can look at the memory.peak cgroup memory interface file for the container using a command like the following:

cat /sys/fs/cgroup/system.slice/docker-<containerid>.scope/memory.peak

and it will tell you the peak memory usage of the whole container (which covers the whole process tree) since its birth. Note that the container must still be running (i.e. it can't be in a stop state) for this file to be present.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.