4

I have copied a folder using rsync including symlinks, hard links, permissions, deleting files on destination et cetera. They should be pretty identical.

One folder is on a USB drive and the other on a local disk.

If I run: du -bls on both folders, the size comes up as slightly different.

My du supports --apparent-size and it is applied by -s and -l should count the content of the hard links.

How can this difference be explained and how do I get the actual total?

Both file systems are ext4, the only difference is that the USB drive is encrypted.

EDIT:

I digged down to find the folders that were actually different, I found one and the content is not special (no block device, no pipes, no hard or symlinks, no zero bytes files), the peculiarity may be having several small files within it. The difference is 872830 vs 881022 of this particular folder.

I also ran du -blsc in both folders and the result is the same in this case.

Some extra details on the commands I used:

$ du -Pbsl $LOCALDIR $USBDIR | cut -f1 872830 881022 $ du -Pbslc $LOCALDIR/* [...] 868734 total $ du -Pbslc $USBDIR/* [...] 868734 total $ ls -la $USBDIR | wc 158 1415 9123 $ ls -la $LOCALDIR | wc 158 1415 9123 $ diff -sqr --no-dereference $LOCALDIR $USBDIR | grep -v identical [No output and all identical if I remove the grep]
Improve this question
6
  • 5
    Define "pretty identical" and "slightly different". Commented May 11, 2019 at 14:47
  • How much of a difference? Commented May 11, 2019 at 14:55
  • @Kusalananda pretty identical = identical content, user, permissions, timestamp for each file/folder, slightly different = a small amount of bytes in difference Commented May 11, 2019 at 14:56
  • @Atul 836034841990 vs 836037115270 (the content is then roughly 800GB) Commented May 11, 2019 at 14:57
  • That's about 2 MB. Are you able to run a md5sum over the files and then verify that against the other set? I wonder if you have a lot of directories that could account for the difference (some filesystems don't truncate the directories when you delete entries)? Commented May 11, 2019 at 15:00

2 Answers 2

Reset to default
10

Since you have copied the files using rsync and then compared the two sets of files using diff, and since diff reports no difference, the two sets of files are identical.

The size difference can then probably be explained by the sizes of the actual directory nodes within the two directory structures. On some filesystems, the directory is not truncated if a file or subdirectory is deleted, leaving a directory node that is slightly larger than what's actually needed.

If you have, at some point, kept many files that were later deleted, this might have left large directory nodes.

Example:

$ mkdir dir $ ls -ld dir drwxr-xr-x 2 kk wheel 512 May 11 17:09 dir 
$ touch dir/file-{1..1000} $ ls -ld dir drwxr-xr-x 2 kk wheel 20480 May 11 17:09 dir 
$ rm dir/* $ ls -ld dir drwxr-xr-x 2 kk wheel 20480 May 11 17:09 dir $ du -h . 20.0K ./dir 42.0K . $ ls -R dir ./dir:

Notice how, even though I deleted the 1000 files I created, the dir directory still uses 20 KB.

Improve this answer
7
  • That is quite interesting, I also used du -bs and I was able to reproduce what you described. Would be interesting to know what ext4 does. Commented May 11, 2019 at 15:17
  • @Stefanod'Antonio I believe that ext4 behaves the same. My tests were on an OpenBSD system using its native FFS filesystem. Commented May 11, 2019 at 15:20
  • @Kusalanada what I meant is how this works behind the scenes: what's the threshold and why it does that. Commented May 11, 2019 at 15:21
  • 1
    @Stefanod'Antonio Possibly to reduce filesystem fragmentation. There is no threshold. The directory node is simply never truncated. Commented May 11, 2019 at 15:22
  • 1
    @danieldeveloper001 I'm not a Linux user and don't know if there's some specific tool for doing this on ext4 filesystems, but the portable way would be to move the contents of a directory to a new directory and then rmdir the original directory. Or, for a whole hierarchy, use rsync to copy it, then delete the original (as the user in the question actually did). Commented May 11, 2019 at 15:31
1

Have you checked the filesystem block size? Even though both devices use the same filesystem, it is a possibility that the block sizes are different and this could explain the "slightly different" sizes.

When, for instance, storing a bunch of 1KiB files in a device with filesystem set to use 8KiB block size, there will be a waste of 7KiB per used block. The actual size that your files are taking from your disk, is the size of the used blocks, not the size of the files itself in this case (unless there is some kind of tool to store multiple files per block). Try checking your different devices block size with the command below.

# blockdev --getbsz <DEVICE>
Improve this answer
4
  • Should this not be covered by using --block-size=1 as a du option? Commented May 11, 2019 at 15:17
  • Just checked they both return 4096. Commented May 11, 2019 at 15:18
  • Sorry, started writing before your edit ;) Commented May 11, 2019 at 15:18
  • No worries, it was a good guess. Commented May 11, 2019 at 15:19

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.