1

For backing up data in my office I use a Raspberry Pi model B (I had a spare one) running rsnapshot. Basically, every night it copies data from a bunch of smb mounted folders to a couple of external hard drives (fuseblk).

I gradually added data to back up and recently the whole process became really slow: it takes something like 15 hrs to perform the entire operation.

This is the log of a copy (only on one disk):

[07/Nov/2018:21:16:05] /usr/bin/rsnapshot -c /etc/rsnapshot.conf Daily: started [07/Nov/2018:21:16:05] echo 28378 > /var/run/rsnapshot.pid [07/Nov/2018:21:16:08] /bin/rm -rf /mnt/Disk1/Backup/Daily.4/ [07/Nov/2018:23:31:33] mv /mnt/Disk1/Backup/Daily.3/ /mnt/Disk1/Backup/Daily.4/ [07/Nov/2018:23:31:33] mv /mnt/Disk1/Backup/Daily.2/ /mnt/Disk1/Backup/Daily.3/ [07/Nov/2018:23:31:33] mv /mnt/Disk1/Backup/Daily.1/ /mnt/Disk1/Backup/Daily.2/ [07/Nov/2018:23:31:33] /bin/cp -al /mnt/Disk1/Backup/Daily.0 /mnt/Disk1/Backup/Daily.1 [08/Nov/2018:02:17:45] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld01 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:43:28] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld02 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:46:29] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld03 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:54:05] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld04 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:54:48] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld05 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:54:49] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld06 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:02:54:49] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld07 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:00:10] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld08 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:25:57] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld09 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:25:57] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld10 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:28:42] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld11 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:53:39] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld12 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:03:58:05] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld13 /mnt/Disk1/Backup/Daily.0/./ [08/Nov/2018:04:00:24] touch /mnt/Disk1/Backup/Daily.0/ [08/Nov/2018:04:00:24] rm -f /var/run/rsnapshot.pid [08/Nov/2018:04:00:24] /usr/bin/rsnapshot -c /etc/rsnapshot.conf Daily: completed successfully

Now, I know that the RPi is not fast, nor are the external drives. Still, the problems seem to be here

[07/Nov/2018:21:16:08] /bin/rm -rf /mnt/Disk1/Backup/Dayly.4/

and especially here

[07/Nov/2018:23:31:33] /bin/cp -al /mnt/Disk1/Backup/Dayly.0 /mnt/Disk1/Backup/Daily.1

Keep in mind that I have probably tens of thousands of files (I'm counting them as I write but I don't know how long it will take). (EDIT: there are 250k files in ~30 GB of space)

Any idea on what could be the problem and if/how I could solve it?

While I'm here, I have no clue on the --relative [...] --no-relative option on the rsync command. I honestly don't remember how I came to it, it's been some time since I configured it. Given that I need to save the tree, should I just use relative? Or is it ok this way, since it works?

-=* UPDATE *=-

I did as I was suggested and formatted as ext4 the usb drives. This is the log after the operation:

[16/Nov/2018:21:16:04] /usr/bin/rsnapshot -c /etc/rsnapshot.conf Daily: started [16/Nov/2018:21:16:04] echo 19966 > /var/run/rsnapshot.pid [16/Nov/2018:21:16:04] /bin/rm -rf /mnt/Disk1/Backup/Daily.4/ [16/Nov/2018:21:18:52] mv /mnt/Disk1/Backup/Daily.3/ /mnt/Disk1/Backup/Daily.4/ [16/Nov/2018:21:18:52] mv /mnt/Disk1/Backup/Daily.2/ /mnt/Disk1/Backup/Daily.3/ [16/Nov/2018:21:18:52] mv /mnt/Disk1/Backup/Daily.1/ /mnt/Disk1/Backup/Daily.2/ [16/Nov/2018:21:18:52] /bin/cp -al /mnt/Disk1/Backup/Daily.0 /mnt/Disk1/Backup/Daily.1 [16/Nov/2018:21:22:25] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld01 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:19] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld02 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:27] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld03 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:41] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld04 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:44] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld05 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:44] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld06 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:24:45] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld07 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:25:04] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld08 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:26:04] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld09 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:26:04] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld10 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:26:20] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld11 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:26:58] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld12 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:28:54] /usr/bin/rsync -a --stats --relative --delete --no-relative /mnt/Backup/Fld13 /mnt/Disk1/Backup/Daily.0/./ [16/Nov/2018:21:30:03] touch /mnt/Disk1/Backup/Daily.0/ [16/Nov/2018:21:30:03] rm -f /var/run/rsnapshot.pid [16/Nov/2018:21:30:03] /usr/bin/rsnapshot -c /etc/rsnapshot.conf Daily: completed successfully

As you can see the overall time was drastically reduced: 15 mins vs. ~7hrs.

Thank you all, I'm honestly impressed.

The only doubt I have left is about what was discussed in the comments: I believe that rsync does an incremental copy, even if it sees the smb source folders as local. Some of these folders contain 10k+ files (probably even more, I can't check in this very moment) and there's just no way that all of these are copied in just, say, 2 minutes.

Improve this question
4
  • Welcome to Unix & Linux! ;-) Why do you use both cp and rsync? why not use rsync only to make your backups? You can use rsync to make daily, weekly, monthly backups. Then there is no need to rm --recursive neither, saving you lots of time... Commented Nov 8, 2018 at 15:16
  • 1
    @Fabby the cp is hard linking, which rsync doesn't do. This is a manual method of how rsnapshot works. Commented Nov 8, 2018 at 16:08
  • @jordanm Thanks for the info. :-) Bailing as this needs someone smarter than me... Commented Nov 8, 2018 at 16:11
  • Thank you both for the welcome and the answers. This is in fact the log of rsnapshot, it's not me manually doing it. I just need to know if there's some options or some workaround that can accelerate the process, maybe I'm doing something I don't actually need. Commented Nov 8, 2018 at 16:31

1 Answer 1

Reset to default
0

There are a couple of problems here slowing the backup solution down.

  1. You're using rsync to copy between two "local" filesystems.

    Just because one of them happens to be SMB is irrelevant to rsync. If the filesystem is mounted as part of the local system then rsync has to treat it as local. This means that any changed file has to be copied from the SMB network share in its entirely, not just the changed parts.

    If your fileserver can run rsync directly, modify the backup process so that it can start a remote rsync process and gain the benefit of incremental copies.

  2. You're writing to your backup disks via fuseblk.

    I assume this is because the disks have NTFS filesystems on them. If you can reformat them to use a native Linux filesystem such as ext4 you will see a significant increase in file IO speed. Including the rm -rf that is taking so long.

    If you're writing to VFAT then you've also got the problem of reduced quality timestamps and you'll need to warn rsync accordingly so that it doesn't try to keep copying otherwise-identical files to your backup media.

    I understand from comments that you are indeed using NTFS and you want to continue using that so the disks can be read under Windows. An alternative is to install an ext4 disk driver into Windows. I use ext2fs, which I find pretty solid.

Improve this answer
10
  • 1 - I'm copying the files from two Windows machines, not from a fileserver. I don't know if I can run rsync on Windows, I'll look into it. Btw, is there another way to do incremental copies through smb (I guess not, since you said that)? Commented Nov 9, 2018 at 19:39
  • 2 - you're right about the NTFS, but I'd rather keep it this way: the RPI is the only Linux machine and if it fails I'd like to be able to read the disks from a Windows pc. I'll think about it, though. Can't I warn rsync about the NTFS filesystem just like you mentioned about the VFAT? Commented Nov 9, 2018 at 19:43
  • There's nothing to warn with NTFS. It's just slow through the FUSE interface. Commented Nov 10, 2018 at 7:48
  • @pecheronza alternative to NTFS posted in the answer. Commented Nov 10, 2018 at 8:17
  • I've been thinking about your answer. 1. Are you sure about what you said about smb and incremental copies? Because I have like 10 timestamps of the backup and while a single one weights 30 GB, all of them use 43 GB. So I think I'm actually using incremental copies. 2. On second thought, it's quite unlikely that both the RPi fails and I need the backups, so I think that I can reformat to ext4, given also that I can install a Windows driver. Commented Nov 10, 2018 at 9:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.