13

I have 1000000 4-20 kb files in a dir (You can generate similar files like this: seq 10000 | gzip > a; seq 1000000 | parallel --bar 'head -c{=$_=int(rand()*16)+4=}k a > {}')

. I need to copy that dir. But it seems I have to do a seek for each files so this takes quite a while.

Is there a way in which I can speed this up?

I am currently thinking that if I could get the disk blocks that these files occupy, I could sort those, merge the blocks that were close (given that sequential read is often faster than seeking) and read these blocks, so that they were in RAM cache (I have 32 GB RAM) before doing the copy.

But for that to work I need a way to identify which blocks the files are on.

I am using EXT4 on a magnetic device (i.e. not SSD).

Edit:

This ought to work but it does not:

ls | parallel -IOO --pipe "sudo parallel -j100 hdparm --fibmap {}'|tail -n +5'" | sort -nk 2 | perl -ane 'if($u+10000 < $F[1]) { print "$l ",($u-$l),"\n"; $l=$F[1] } $u=$F[2]' | sudo parallel --colsep ' ' dd if=/dev/sda1 skip={1} bs=512 count={2} '| cat >/dev/null'

When testing it on a big file it does not cache the file.

Edit2:

Here are some benchmarks. Cache was flushed (echo 3 >/proc/sys/vm/drop_caches) between each run. Measurements done with iostats -dkx 5.

rsync -Hav foo/ bar/: 1800 KB/s cp -a foo/ bar/: 3600 KB/s cat sort-by-inode | parallel -j1 -X cp foo/{} bar/: 5000 KB/s cat sort-by-inode | shuf | parallel -j1 -X cp foo/{} bar/: 3000 KB/s cat sort-by-inode | shuf | parallel -j10 -X cp foo/{} bar/: 7000 KB/s cat sort-by-inode | parallel -j10 -X cp foo/{} bar/: 8000 KB/s cat sort-by-inode | parallel -j100 -X cp foo/{} bar/: 9000 KB/s cat sort-by-inode | parallel -j500 -X cp foo/{} bar/: 10000 KB/s

So what can we learn from that?

It seems sorting by inode is a good idea. But it seems parallelizing multiple cp boosts performance even further. It is worth stressing that the source foo/ is a magnetic disk, so this attacks the myth that parallelizing I/O to a single spindle will not speed up I/O: Parallelizing clearly and consistently speeds up the copying here.

Improve this question
14
  • Does your directory contain only files? Is your target location on the same filesystem? Commented Apr 13, 2014 at 10:18
  • For exact command you are using for copying? Something like cp -r /mnt/dir1 /mnt/dirdest or something like cp /mnt/dir1/* /mnt/dirdest? Commented Apr 13, 2014 at 10:26
  • Only files but not the same filesystem: cp -r /mnt/dir1 /mnt2/dirdest Commented Apr 13, 2014 at 11:44
  • btw, I don't think it helps if you execute hard disk accesses in parallel when you want to minimize disk seeks. Commented Apr 13, 2014 at 13:20
  • 1
    I wrote a program that orders files by their extent number (more likely to be the occurrence on disk) here: github.com/nh2/diskorder Commented Feb 19, 2017 at 11:09

3 Answers 3

Reset to default
11

Assuming that

  • entries returned by readdir are not sorted by inode number
  • reading files in inode order reduces the number of seek operations
  • the content of most files are in the initial 8k allocation (an ext4 optimization) which also should yield less seek operations

you can try to speed up copying via copying files in inode order.

That means using something like this:

$ cd /mnt/src $ ls -U -i | sort -k1,1 -n | cut -d' ' -f2- > ~/clist $ xargs cp -t /mnt2/dst < ~/clist
Improve this answer
4
  • @mikeserv, what do you mean? ls -U is not enough because it does not sort by inode numbers ... and why should I want -1? Commented Apr 13, 2014 at 16:19
  • @mikeserv, 'in directory order' is not the same as inode order! If this would be the case you would not have to use a different word for this. What you find strange is not relevant. I've even tested it on a ext4 filesystem. And there the directory order is indeed different from the inode order. -1 just lists 'one file per line' - it does not help with newlines in filenames. For that you can use find -print0/xargs -O. Commented Apr 13, 2014 at 16:36
  • @mikeserv, what are you talking about? Counter example: mkdir tmp; cd tmp; touch foo"<RETURN>"bar; ls prints 'foo?bar'. A ls -1 also prints 'foo?bar'. A ls -1 | wc -l prints '2'. A find -ls prints the filename as './foo\nbar'. A cp -i ls -1` x` fails with 'cp: target ‘x’ is not a directory'. Commented Apr 13, 2014 at 17:04
  • Damn - you're teaching me left and right! -q does what I thought -1 would! Again, my apologies - not to mention thanks. Commented Apr 13, 2014 at 17:13
6

GNU tar - in the pax tradition - handles hardlinks on its own.

cd "$srcdir" ; tar --hard-dereference -cf - ./* | tar -C"${tgtdir}" -vxf -

That way you only have the two tar processes and you don't need to keep invoking cp over and over again.

Improve this answer
0
3

On a similar vein to @maxschlepzig's answer, you can parse the output of filefrag to sort files in the order that their first fragments appear on disk:

find . -maxdepth 1 -type f | xargs -d'\n' filefrag -v | sed -n ' /^ 0: 0../ { s/^.\{28\}\([0-9][0-9]*\).*/\1/ h } / found$/ { s/:[^:]*$// H g s/\n/ /p }' | sort -nk 1,1 | cut -d' ' -f 2- | cpio -p dest_dir

MMV with the above sed script, so be sure to test thoroughly.

Otherwise, whatever you do, filefrag (part of e2fsprogs) will be much faster to use than hdparm as it can take multiple file arguments. Just the overhead of running hdparm 1,000,000 times is going to add a lot of overhead.

Also it probably wouldn't be so difficult to write a perl script (or C program), to a FIEMAP ioctl for each file, create a sorted array of the blocks that should be copied and the files the belong to and then to copy everything in order by reading the size of each block from the corresponding file (be careful not to run out of files descriptors though).

Improve this answer
2
  • This is nice, see also home.ifi.uio.no/paalh/publications/files/ipccc09.pdf for a paper that describes the approach and shows a ~4x speedup for tar for their files. Commented Dec 10, 2016 at 21:40
  • 2
    I've emailed the authors of the paper, asking whether they could release qtar as open source; it is now at github.com/chlunde/qtar Commented Dec 12, 2016 at 19:06

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.