We are currently testing tiered memory demotion on a machine equipped with a CXL device.
To facilitate this, we created a specific script (https://github.com/hyun-sa/comem) and are using the memory.high setting within a cgroup to force memory demotion.
These are the commands we used to enable demotion:
echo 1 > /sys/kernel/mm/numa/demotion_enabled echo 2 > /proc/sys/kernel/numa_balancing The issue we're facing is that while demotion does occur, it proceeds extremely slowly—even slower than swapping to disk. Furthermore, during a 7-Zip benchmark, we observe a severe drop in CPU utilization, as if some process is causing a lock.
This is our running example (7zr b -md25 while memory is limited as 8G by memory.high)
7-Zip (r) 23.01 (x64) : Igor Pavlov : Public domain : 2023-06-20 64-bit locale=C.UTF-8 Threads:128 OPEN_MAX:1024 d25 Compiler: 13.2.0 GCC 13.2.0: SSE2 Linux : 6.15.6 : #1 SMP PREEMPT_DYNAMIC Tue Jul 15 06:39:48 UTC 2025 : x86_64 PageSize:4KB THP:madvise hwcap:2 hwcap2:2 AMD EPYC 9554 64-Core Processor (A10F11) 1T CPU Freq (MHz): 3710 3731 3732 3733 3733 3732 3732 64T CPU Freq (MHz): 6329% 3674 6006% 3495 RAM size: 386638 MB, # CPU hardware threads: 128 RAM usage: 28478 MB, # Benchmark threads: 128 Compressing | Decompressing Dict Speed Usage R/U Rating | Speed Usage R/U Rating KiB/s % MIPS MIPS | KiB/s % MIPS MIPS 22: 477942 10925 4256 464943 | 5843081 12451 4001 498193 23: 337115 8816 3896 343480 | 5826376 12606 3999 504053 24: 1785 108 1772 1919 | 5654618 12631 3928 496161 25: 960 63 1739 1097 | 1767869 4606 3415 157287 ---------------------------------- | ------------------------------ Avr: 204451 4978 2916 202860 | 4772986 10573 3836 413924 Tot: 7776 3376 308392 execution_time(ms): 2807639 Is there a potential misunderstanding of how cgroups function or a misconfiguration in my setup that could be causing this behavior?
Our machine specifications are as follows:
Mainboard : Supermicro H13SSL-NT
CPU : Epyc 9554 (nps 1)
Dram : 128G
CXL device : SMART Modular Technologies Device c241
OS : Ubuntu 24.04 LTS
Kernel : Linux 6.15.6
numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 node 0 size: 128640 MB node 0 free: 117909 MB node 1 cpus: node 1 size: 257998 MB node 1 free: 257840 MB node distances: node 0 1 0: 10 50 1: 255 10 Thank you for your help.
