5

When a service has configured a MemoryHigh value and this value is exceeded, the documentation says that this happens:

Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases.

Does that mean that ALL the processes of a service are "heavily slowed down"? Or does the system have some kind of discretion about which processes it throttles?

It would be useful if it mostly throttles the processes that actually uses most memory first, and leaves low memory processes alone.

Improve this question
1
  • ... a CGroup is basically a container and the limits are applied to the processes within collectively and not selectively AFAIK (at least from the perspective of these limits) ... an example of how a CGroup might look in this regard is shown in my answer to a different topic here. Commented Sep 11 at 9:32

2 Answers 2

Reset to default
5

Such limits are set and examined against CGroups ... These are basically containers of processes.

It's worth noting that different settings are handled differently ... For example with MemoryMax and when a limit is challenged, the kernel's OOM killer might be unleashed upon the processes within the violating CGroup.

However, to answer your question, with MemoryHigh when a limit is challenged/exceeded, the extra memory is reclaimed from the CGroup by e.g. most likely swapping pages of memory allocated to processes within the violating CGroup to disk and freeing memory. Hence the slowness i.e. "the processes are heavily slowed down and memory is taken away aggressively in such cases.".

In this case and for this setting, which controls the memory.high control group attribute. If a CGroup’s usage goes over the high boundary, the processes of the CGroup are throttled and put under heavy reclaim pressure.

This appears to happen collectively on all contained processes and not selectively so all processes are affected relatively to the same extent.

Improve this answer
3

While MemoryHigh is an attribute of a cgroup, the throttling is done per process (more precisely, per task). The kernel throttles allocating tasks that cross the MemoryHigh boundary proportionally to the number of pages allocated above the MemoryHigh limit (current->memcg_nr_pages_over_high variable, where current is the current task).

Based on the comments in the kernel mm/memcontrol.c.

The allocating tasks in this cgroup will need to do reclaim or be throttled to prevent further growth of the memory or swap footprints. Target some best-effort fairness between the tasks, and distribute reclaim work and delay penalties based on how much each task is actually allocating.

The function mem_cgroup_handle_over_high in mm/memcontrol.c has more details:

The allocating task should reclaim at least the batch size, but for subsequent retries we only want to do what's necessary to prevent oom or breaching resource isolation

memory.high is breached and reclaim is unable to keep up. Throttle allocators proactively to slow down excessive growth.

The function calculate_high_delay

We use overage compared to memory.high to calculate the number of jiffies to sleep (penalty_jiffies). Ideally this value should be fairly lenient on small overages, and increasingly harsh when the memcg in question makes it clear that it has no intention of stopping its crazy behaviour, so we exponentially increase the delay based on overage amount.

Factor in the task's own contribution to the overage, such that four N-sized allocations are throttled approximately the same as one 4N-sized allocation.

Improve this answer
2
  • You talk about (active) "throttling; isn't it that the processes are indirectly throttled by the extra I/O as pages are taken away (paged out) from a process to allow another (or the same) process to add pages (page in)? Just as if you are low on RAM? Commented Sep 12 at 7:14
  • @U.Windl When the documentation talks about processes being "heavily slowed down", it talks about active throttling. If the processes are slowed down due to heavy swap I/O and resource starvation, it happens to the entire system, and it is a misconfiguration. Cgroup limits, such as MemoryHigh, are the tools to prevent such unnecessary waste of system resources and slow down only the leaking processes. Commented Sep 17 at 15:47

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.