36

I am interested in a utility or process for monitoring disk IO per file on CentOS.

On Win2008, the resmon utility allows this type of drilldown, but none of the Linux utilities I have found do this (iostat, iotop, dstat, nmon).

My interest in monitoring IO bottlenecks on database servers. With MSSQL, I have found it an informative diagnostic to know which files / filespaces are getting hit the hardest.

Improve this question
3
  • 2
    Maybe Linux Performance Analysis and Tools: Brendan Gregg's Talk at SCaLE 11x can help you; see slide 72/115. Commented Oct 3, 2013 at 2:14
  • 1
    If this is possible, note most files are mapped into pagecache so your numbers could be all over the place depending on what bytes are in pagecache and what are on disk. Commented Oct 10, 2013 at 18:07
  • @Matt But with a working answer! Commented Oct 17, 2013 at 22:29

13 Answers 13

Reset to default
21

SystemTap is probably your best option.

Here is how the output from the iotime.stp example looks like:

825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0 825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9 [...] 117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0 117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7 [...] 3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0 3973744 2886 (sendmail) iotime /proc/loadavg time: 11

The disadvantage (aside from the learning curve) is that you will need to install kernel-debug, which may not be possible on a production system. However, you can resort to cross-instrumentation where you compile a module on a development system, and run that .ko on the production system.

Or if you are impatient, look at Chapter 4. Useful SystemTap Scripts from the beginners guide.

Improve this answer
19
+500

SystemTap* script: 

#!/usr/bin/env stap # # Monitor the I/O of a program. # # Usage: # ./monitor-io.stp name-of-the-program global program_name = @1 probe begin { printf("%5s %1s %6s %7s %s\n", "PID", "D", "BYTES", "us", "FILE") } probe vfs.read.return, vfs.write.return { # skip other programs if (execname() != program_name) next if (devname=="N/A") next time_delta = gettimeofday_us() - @entry(gettimeofday_us()) direction = name == "vfs.read" ? "R" : "W" # If you want only the filename, use // filename = kernel_string($file->f_path->dentry->d_name->name) # If you want only the path from the mountpoint, use // filename = devname . "," . reverse_path_walk($file->f_path->dentry) # If you want the full path, use filename = task_dentry_path(task_current(), $file->f_path->dentry, $file->f_path->mnt) printf("%5d %1s %6d %7d %s\n", pid(), direction, $return, time_delta, filename) }

The output looks like this:

[root@sl6 ~]# ./monitor-io.stp cat PID D BYTES us FILE 3117 R 224 2 /lib/ld-2.12.so 3117 R 512 3 /lib/libc-2.12.so 3117 R 17989 700 /usr/share/doc/grub-0.97/COPYING 3117 R 0 3 /usr/share/doc/grub-0.97/COPYING

Or if you choose to display only the path from the mountpoint:

[root@f19 ~]# ./monitor-io.stp cat PID D BYTES us FILE 26207 R 392 4 vda3,usr/lib64/ld-2.17.so 26207 R 832 3 vda3,usr/lib64/libc-2.17.so 26207 R 1758 4 vda3,etc/passwd 26207 R 0 1 vda3,etc/passwd 26208 R 392 3 vda3,usr/lib64/ld-2.17.so 26208 R 832 3 vda3,usr/lib64/libc-2.17.so 26208 R 35147 16 vdb7,ciupicri/doc/grub2-2.00/COPYING 26208 R 0 2 vdb7,ciupicri/doc/grub2-2.00/COPYING [root@f19 ~]# mount | grep -E '(vda3|vdb7)' /dev/vda3 on / type ext4 (rw,relatime,seclabel,data=ordered) /dev/vdb7 on /mnt/mnt1/mnt11/data type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Limitations/bugs:

  • mmap based I/O doesn't show up because devname is "N/A"
  • files on tmpfs don't show up because devname is "N/A"
  • it doesn't matter if the reads are from the cache or the writes are to the buffer

The results for Matthew Ife's programs:

  • for mmaptest private:

     PID D BYTES us FILE 3140 R 392 5 vda3,usr/lib64/ld-2.17.so 3140 R 832 5 vda3,usr/lib64/libc-2.17.so 3140 W 23 9 N/A,3 3140 W 23 4 N/A,3 3140 W 17 3 N/A,3 3140 W 17 118 N/A,3 3140 W 17 125 N/A,3

  • for mmaptest shared:

     PID D BYTES us FILE 3168 R 392 3 vda3,usr/lib64/ld-2.17.so 3168 R 832 3 vda3,usr/lib64/libc-2.17.so 3168 W 23 7 N/A,3 3168 W 23 2 N/A,3 3168 W 17 2 N/A,3 3168 W 17 98 N/A,3

  • for diotest (direct I/O):

     PID D BYTES us FILE 3178 R 392 2 vda3,usr/lib64/ld-2.17.so 3178 R 832 3 vda3,usr/lib64/libc-2.17.so 3178 W 16 6 N/A,3 3178 W 1048576 509 vda3,var/tmp/test_dio.dat 3178 R 1048576 244 vda3,var/tmp/test_dio.dat 3178 W 16 25 N/A,3

*Quick setup instructions for RHEL 6 or equivalent: yum install systemtap and debuginfo-install kernel

Improve this answer
6
  • Thats some pretty impressive systemtap right there. An excellent demonstration of its versatility. Commented Oct 16, 2013 at 19:34
  • Does this measure direct I/O and asyncronous I/O? (using io_submit, not posix) Commented Oct 16, 2013 at 19:49
  • @Mlfe: thanks! As a side note, while writing the script I managed to discover a tiny bug in pv and another one in SystemTap (task_dentry_path) :-) I have no idea about that I/O, but I can test it if you give me a command or a sample program. For example I used Python to test mmap. dd iflag=direct oflag=direct shows up. Commented Oct 16, 2013 at 19:58
  • 2
    Try this for mmap: gist.github.com/anonymous/7014284 I'm betting private mappings aren't measured but shared ones are.. Commented Oct 16, 2013 at 20:29
  • 2
    Heres a direct IO test: gist.github.com/anonymous/7014604 Commented Oct 16, 2013 at 20:50
9

You'd actually want to use blktrace for this. See Visualizing Linux IO with Seekwatcher and blktrace.

I'll see if I can post one of my examples soon.

Edit:

You don't mention distribution of Linux, but maybe this is a good case for a dtrace script on Linux or even System Tap, if you're using a RHEL-like system.

Improve this answer
2
  • 2
    Thanks, good thing and very close to the point, but it provides detailed block-layer info, i need something that works on VFS abstraction layer. Commented Oct 3, 2013 at 1:41
  • I started to attempt some systemtap scripts to get this running. I failed because the server crashed. I can get this info on Dtrace on Solaris. I'll attempt with Linux today. Commented Oct 12, 2013 at 15:02
5

use iotop to get the PIDs of processes that contribute high IO

run strace against the PID you generated, you will see which files are being accessed by a particular process

strace -f -p $PID -e trace=open,read,write
Improve this answer
2
  • strace will provide information on syscalls and accessed files, it will be very hard to parse and get stats on the usage... Commented Oct 3, 2013 at 14:35
  • 1
    Thought I'd try this. It generates ALOT of data. And can crash the process when you ctrl+c. It seems to be rather dangerous. Commented Oct 10, 2013 at 3:26
4

The only tool I know of that can monitor I/O activity by file is inotifywatch. It's part of the inotify-tools package. Unfortunately, it only gives you operation counts.

Improve this answer
4

I eventually used Sysdig for this

Improve this answer
2
  • How-to: Install sysdig, run csysdig, press F2, and select Files view. You will see top of accessed files by OPS column (can be changed pressing F9). Commented Jul 29, 2017 at 18:48
  • csysdig -v files goes directly to the "Files" view Commented Jan 21, 2019 at 9:14
2

I'd argue you might have asked the wrong question. if you're looking for i/o bottlenecks, it may be just as important to see what's happening on your disk. db's are notorious for doing random i/o which can significantly reduce throughput, especially if you only have a few spindles.

what may be more interesting is to see if you're having long wait times on the disks themselves. you can do this with collectl via the command "collectl -sD", which will show individual disk performance stats. Are --home to turn it into a top-like utility. If there are lots of disks involved, run it via colmux: colmux -command "-sD" and it will let you sort by a column of your choice, even across multiple systems.

Improve this answer
2
  • I don't disagree with you from a disk perspective. Where I can get some insight is when a database filespaces are used to partition data, indexes, logs, etc., but mounted on shared disks when resources are limited - development servers for example. Ideally, each of these filespaces would be on separate volumes, thus looking at the IO from the disk perspective would be adequate - which is likely why all the monitoring utilities are disk, not file based. Commented Nov 5, 2011 at 16:18
  • It's the right question; the goal is trying to figure out "which table is all this I/O happening to?", and in most databases a table is one or more files. Any disk is going to end up with many files on it, and determining which of those are the hotspots is a useful database tuning input. Commented Aug 1, 2012 at 0:53
2

You can monitor i/o per block device (via /proc/diskstats) and per process (io accounting via /proc/$PID/io or taskstats), but I don't know of a way to do it per-file.

Improve this answer
0

May be inotify will resolve solve this.

The inotify API provides a mechanism for monitoring filesystem events.Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

Monitor File System Activity with inotify

inotify Reference

Improve this answer
3
  • This may provide the calls done on the file, but offers little to help discover what pid did it, how big the write was, where and how long it took. Commented Oct 12, 2013 at 10:45
  • The question does not ask about the process (that can possibly be discovered by other means, like lsof) Commented Jan 21, 2019 at 9:20
  • As it's mostly to be expected, the IBM link is no longer valid. Commented Jul 4, 2022 at 10:03
0

While there's lots of good info in the answers here, I'm wondering if it's actually applicable?

If you are talking about files in the 10s of gigabytes, constantly being written to, then unless they are log files or similar which are constantly appended to (in which case just monitor file sizes), it's most likely that the files are mmap'd. If that's the case, then the best answer may be that you should stop looking at most solutions. The first thing you should then ask of any other proposed solution is "does it work with mmap", because mostly they wont.However, you may be able to turn the problem into monitoring a block device rather than monitoring a file.

When a program asks for a page from a mmap'd file, it's just referencing a location in virtual memory. That page might or might not already be in memory. IF it's not, then that generates a page fault, which triggers the page being loaded in from disk, but that happens within the virtual memory system, which is not easily tied to a specific application process or to a specific file. Similarly, when your app updates a mmap'd page, depending on flags, that might not trigger an immediate write to disk, and in some cases may not go to disk at all (though presumably those last aren't the cases you are interested in).

The best I can think of for mmap'd files, if it's viable for you, is to put each of the files of interest onto a separate device, and use the device statistics to collect your usage info. You could use lvm partitions for this. A bind mount won't work though as it doesn't create a new block device.

Once you have your files on separate block devices you can get stats from /sys/block/*, or /proc/diskstats

It might be too disruptive to introduce this to a production server, but maybe you can make use of it.

IF the files are not mmapped, then yes, you can choose one of the other solutions here.

Improve this answer
2
  • Read carefully please, I don't need block-level stats :) Commented Oct 18, 2013 at 9:12
  • Right, but the kind of stats you are asking for are not possible for mmapped files, so if you're running up against that, then this gives you a possible way to get stats about files by using one file per device, and reading the device stats. Commented Oct 23, 2013 at 18:13
-1

I recently was tinkering with collectl , it looks a great tool, and pretty straigforward to install. The most interesting is that you can find out which is the responsible process for IO bottlenecks. I would recommend you to read Using Collectl , it might be useful.

Improve this answer
1
  • 1
    collectl does not monitor per file, it works per process Commented Aug 1, 2012 at 0:49
-1

I would recommend you to check http://dag.wieers.com/home-made/dstat/. This great tool allows to check a lot of stats.

Improve this answer
1
  • 1
    dstat does not monitor per file, it summarizes per process Commented Aug 1, 2012 at 0:50
-2

I think that iotop is one of the best tool on Linux for identify bottlenecks on IO.

Improve this answer
1
  • 3
    -1 iotop does not monitor per file, it works per process Commented Mar 7, 2012 at 13:42

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.