This is not so much a performance problem as it is a desire to understand what is happening and how it works. I have a system with lots of resources, including 128 GB of RAM. What I have discovered (which happens every time given enough up time) is that the system will start using the entire swap space available to it when there is still 86 GB of memory left.
Here is a screenshot of htop demonstrating the system state:
Swappiness is set to 5:
$ sysctl vm.swappiness vm.swappiness = 5 vmstat shows mostly 0s, but one event:
$ vmstat 60 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 8305756 5218320 31872 83460448 0 0 0 1411 11432 17991 1 1 98 0 0 2 0 8305616 5042872 31872 83571392 66 0 1193 2068 12097 18650 2 1 98 0 0 System details:
| Key | Value |
|---|---|
| Linux Distro | Fedora 37 |
| Kernel version | 6.4.11-100.fc37.x86_64 |
Top processes using swap (sorted highest at the bottom)
PID User Command Swap USS PSS RSS ... 2835417 ben /usr/lib64/firefox/firefox 60104 703736 710125 818252 2835448 ben /usr/lib64/firefox/firefox 63260 1179992 1185457 1288392 2833473 ben /opt/google/chrome/chrome - 64748 32896 34897 146000 3090226 100998 /usr/lib/chromium/chromium 67612 101820 104321 136248 2835394 ben /usr/lib64/firefox/firefox 72476 1177840 1187600 1295372 12678 100998 /usr/local/bin/python /usr/ 75788 330820 330927 332536 3682935 ben /opt/google/chrome/chrome - 87196 207280 210962 350268 1516354 ben /home/ben/.rubies/ruby-3.0. 91728 28868 29182 32820 3621825 ben /opt/google/chrome/chrome - 108724 175184 178064 310136 2833257 ben /opt/google/chrome/chrome 110132 513256 545048 710296 599920 ben /usr/lib64/erlang/erts-13.2 113620 27728 28951 34700 1281109 ben /opt/google/chrome/chrome - 139388 24860 27513 153668 1013489 qemu /usr/bin/qemu-system-x86_64 774020 16138492 16138790 16150960 1013519 root /usr/libexec/virtiofsd --fd 816020 24 154 1880 2863657 ben /usr/lib/slack/slack 1504064 124068 143980 222196 EDIT - January 2024: Still don't have the answer, but adding some additional information that I've picked up from continued usage. The most common processes that seem to accumulate swap are usually the same, and are KVM/qemu related, and Google Chrome/Electron related

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less