2

On my system with 4.9.86 kernel I have noticed weird behaviour with my disk (HDD with 5400 rpm), the %util goes 100% for quite some time constantly (for 5 minutes or so), I do see the avrg-rq size is 8K when this happens. avgqu-sz and await is also very high, causing many processes going into D state (including jdb2 thread) . I have also noticed KBDirty going high this time (658 MB in this case which is usually in few KBs otherwise), Am I hitting disk saturation? 

SAR Memory Usage:====================================== Linux 4.9.86     01/07/19        _x86_64_        (32 CPU) 11:29:20    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty 11:29:21     80270488  52009236     39.32    354368  17373312  15789156      7.92  10257860  15388656    658488 Average:     80270488  52009236     39.32    354368  17373312  15789156      7.92  10257860  15388656    658488 SAR IO Usage:====================================== Linux 4.9.86     01/07/19        _x86_64_        (32 CPU) 11:29:22          tps      rtps      wtps   bread/s   bwrtn/s 11:29:23       351.00      0.00    351.00      0.00   2808.00 Average:       351.00      0.00    351.00      0.00   2808.00 SAR Device IO activity:====================================== Linux 4.9.86     01/07/19        _x86_64_        (32 CPU) 11:29:23          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util 11:29:24        loop5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 11:29:24          sda    285.00      0.00   2280.00      8.00    143.51    510.94      3.51    100.00 11:29:24        vault      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util Average:        loop5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 Average:          sda    285.00      0.00   2280.00      8.00    143.51    510.94      3.51    100.00 Average:        vault      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 SAR Queue and Load avg:====================================== Linux 4.9.86     01/07/19        _x86_64_        (32 CPU) 11:29:25      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked 11:29:26            0      1043      3.39      2.30      2.15         2 Average:            0      1043      3.39      2.30      2.15         2

the file system mounted as ext3 with ext4 driver, data=ordered,barrier=0 setting with journaling enabled.

Raid configuration:

  Model:SAS2008 Firmware Version: 9.00.00.00 RAID Level:RAID1  

Improve this question

1 Answer 1

Reset to default
0

This looks like what you'd expect to see if process were to send a lot of non-sequential (random) small writes. You'd have a relatively small average request size (8, which probably means 8×512 byte sectors = 4K, so the minimum with normal writes). Having more dirty buffers also is consistent, it means the writes have been passed off to the kernel & the kernel is working on writing them to disk. 285 tps is pretty darn nice performance for a magnetic disk.

You need to investigate what programs are writing to disk and see if they're exhibiting abnormal behavior. Or if the programs can be configured to spread write better (e.g., if it's a database, the dirty page writeback speed is often configurable).

ext3 isn't really recommended for, well, anything. ext4 is a conservative choice for a replacement (but with barrier=0, you're clearly not concerned about that); XFS is another good choice (and still very reliable). But I doubt that'll really help here. SSDs, though, certainly will give far higher IOPS.

Improve this answer
1
  • thanks. the issue is more frequent after kernel upgrade from 3.x to 4.9.86. I have noticed one weird thing after this upgrade. the mount point was in writeback mode in 3.x kernel (using ext3 driver) which got changed to ordered mode (using ext4 driver but fs mounted as ext3). Can this be a factor? Commented Jan 9, 2019 at 16:52

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.