1

We have kernel dirty bytes settings configured as below.

vm.dirty_background_bytes = 262144000 vm.dirty_background_ratio = 0 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 0 vm.dirty_ratio = 10 vm.dirty_writeback_centisecs = 0 vm.dirtytime_expire_seconds = 43200

According to these settings, the kernel should kickoff the background threads for the writeback process. I can see all other process doing writeback except rsyslog. rsyslog seems to be retaining all dirty pages in memory and it is not flushing it. This issue is not consistent and happens periodically.

Why is rsyslog not flushing dirty pages to disk? How to troubleshoot?

# cat /sys/fs/cgroup/system.slice/rsyslog.service/memory.stat anon 4542464 file 5007929344 kernel_stack 65536 pagetables 262144 percpu 0 sock 0 shmem 0 file_mapped 0 file_dirty 4028043264 file_writeback 0 swapcached 0 anon_thp 0 file_thp 0 shmem_thp 0 inactive_anon 4538368 active_anon 4096 inactive_file 5007929344 active_file 0 unevictable 0 slab_reclaimable 148468184 slab_unreclaimable 103840 slab 148572024 workingset_refault_anon 0 workingset_refault_file 0 workingset_activate_anon 0 workingset_activate_file 0 workingset_restore_anon 0 workingset_restore_file 0 workingset_nodereclaim 0 pgfault 377712 pgmajfault 0 pgrefill 0 pgscan 0 pgsteal 0 pgactivate 4 pgdeactivate 0 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0

You can notice the file_writeback is always 0 when the server gets into this mode and this is happening only on rsyslog.

rsyslog version - 8.2112.0-2ubuntu2.2 OS - Ubuntu jammy 22.04 LTS Kernel - 5.15.0-52-lowlatency

Improve this question

1 Answer 1

Reset to default
3

rsyslogd, like syslogd before it, doesn't sync log files unless you tell it to (otherwise it would likely tank filesystem performance with all of the writes it does).

From man rsyslog.conf:

By default, files are not synced after each write. To enable syncing of log files globally, use either the $ActionFileEnableSync directive or the sync parameter to omfile.

Enabling this option degrades performance and it is advised not to enable syncing unless you know what you are doing.

To selectively disable syncing for certain files, you may prefix the file path with a minus sign (-).

It's best to just turn on syncing for a few critical log files that you need to be synced in case of crash or power failure, e.g. kern.log and auth.log

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.