2

I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive).

At the moment, I'm archiving using:

tar -cvf /dev/nst0 --totals --warning=no-file-changed $OLDEST_DIR

This is getting me about 90MBps from a disk that should be capable of about three times that (and the tape should be capable of 2-3 times that). Looking closely, it seems like I'm CPU-bound since tar is taking 100% of one CPU.

This is particularly annoying since I'm trying to verify if the archive is the right size by first doing this 

tar -cP --warning=no-file-changed $OLDEST_DIR | wc -c

... and then comparing the sizes of the archives produced.

So, is there any quicker way?

Improve this question
2
  • 2
    Modern PCs have terrible architecture. With older architectures, the software could tell the tape-drive to write the tar header, then tell the disk-drive a list of blocks to write directly to tape (not via the cpu). and x86 CPUs have terrible data throughput. Commented Sep 1, 2016 at 19:09
  • Try using star instead of GNU tar. star has been tuned for performance and should improve this. Commented Jun 20, 2018 at 9:39

1 Answer 1

Reset to default
2

x86-64 CPU's data throughput is on the order of 64GB/s, so I don't think that's your problem. Is this even x86-64 Linux, or something else? Most likely the problem is some CPU work is done per transaction, and it's using too small blocks. Try:

strace -fo /tmp/tar.rw.txt -eread,write tar -cvf /dev/nst0 --totals --warning=no-file-changed $OLDEST_DIR

To see what tar wants to do with blocking in I/O in the resulting /tmp/tar.rw.txt file. Most likely you'll find it reading and writing 10KB blocks. You can fix that with the -b flag, which supposedly defaults to 20. I bet your hardware can handle I/O in the megabytes, and your OS will split it back down if it can't, so try -b $[1024*2*32] for 32MB transactions.

Next, you should check what the OS wants to do with the transactions. Try tar with the new -b value, make sure you have sysstat installed, and while it's running, check iostat -xm 4 and watch the counters. The main thing to pay attention to is the 'avgrq-sz' column. If no splits are going on, that should be about 64000. If it's splitting, your OS thinks it can't read or write that many bytes in one transaction. That's a topic in and of itself, but you can quickly up the limits by marking down the drive (I think nst0 should be in there), and

cd /sys/block/nst0/queue cat max_hw_sectors_kb > max_sectors_kb`

and the same with every layer of the disk you're reading from (including lvm and dm layers). It's critical that you increase max_sectors_kb from the lowest (sda) level first and the highest (eg. dm23) level last. Check recursively for /sys/block/<dm>/holders/*/holders/*/.... .

Now with these new settings, you have to be careful of two things. One is to md5sum the original files, tar and untar from tape, and check the md5sums to make sure the files are still getting recorded properly. -b shouldn't cause a problem like that, but I haven't tested your tape hardware, etc. The second is to make sure you aren't running low on RAM because of the bigger transaction sizes. You may need to make sure the sysctl vm.min_free_kbytes is plenty big, because if it runs out during a disk transaction, really bad things happen.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.