8

I've been playing around with obnam these last few days, and although it looks very promising and seems to offer basically everything I ever wanted in a backup tool, I am pretty disappointed in its performance. In fact, it's so slow, I'm suspecting that obnam is not even at fault here, but something in my environment is causing it.

So I'm mainly wondering, if anyone else is using obnam or knows its internals well enough to maybe identify the problem.

From what I could tell so far, obnam seems to fork an individual gpg process for each file that is backed up. Judging from htop, strace, and iostat, the speed of an initial backup is mostly bounded by the constant forking, while the CPU and the drives (no networking is involved) are mostly idling below 20% utilization.

My backup amounts to about 500.000 files with 170 GiB of data in total. So for each backup run, gpg is forked 500.000 times. I'm actually not even surprised that this takes almost an entire day for the initial run and over three hours for another run with most files unchanged. But is this really the performance obnam users should expect? For comparison: an incremental run of rsnapshot (same data, same machine, same drives) takes about four minutes. Granted, there is no encryption involved, but this shouldn't be that significant.

So, to ask plainly: Is everyone else's machine not able to run gpg (encrypting a small chunk of data) more than 50 times per second either, ultimately making obnam an almost unusably slow tool? Or is it just me?

(FWIW, my machine is a Core i5-2500 with 8G of RAM and SSD drives, running Gentoo. Backup is done onto an HDD, but I couldn't observe any difference to backing up to the SSD, since it's not I/O-bound.)

Improve this question

3 Answers 3

Reset to default
4

Here's a good read on how to speed up obnam (may run up to 10 times faster): http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html

Summary: add "--lru-size=1024 --upload-queue-size=512" to your commandline or config file. Note that it increases obnam's memory usage a bit.

Improve this answer
3
  • On a memory-rich system, these can be set even higher. I've gotten more than 10x speed gains with these settings! Commented Oct 16, 2014 at 22:22
  • default is --lru-size=256 --upload-queue-size=128 What would be a good value on my Ubuntu with 8GB RAM that should backup onto a quite slow online server with only 2GB RAM? Commented May 28, 2015 at 21:24
  • If the backup target is really slow, then your bottleneck might not be the LRU size or the upload queue size. Please open a new question. Commented May 29, 2015 at 13:46
3

I think I would attack this issue in a couple of ways. For starters I would try and diagnose it myself using the following methodologies.

1. obnam logs

For starters you can log messages from obnam like so:

$ obnam --log obnam.log

You can increase the logging level via the --log-level switch as well to get more details.

--log=FILE write log entries to FILE (default is to not write log files at all); use "syslog" to log to system log, or "none" to disable logging --log-level=LEVEL log at LEVEL, one of debug, info, warning, error, critical, fatal (default: info)

2. Profiling

You can also get a profile of what obnam is doing as follows from this excerpt in the project's FAQs:

If the OBNAM_PROFILE environment variable contains a file name, the profiling data gets stored there, and can be viewed later with obnam-viewprof:

 $ OBNAM_PROFILE=obnam.prof obnam ... obnam-viewprof obnam.prof | less

Performance issues that are not related to a particular setup can also be observed using the obnam-benchmark tool.

3. Open a ticket

If the performance is still not determined doing some self driven investigation then I would open a ticket up on the project's website. From what I was able to gather the developer(s) are somewhat responsive and they would likely be the best at rooting out issues with their project.

obnam appears to only use SFTP so it should be pretty obvious what's causing the problem. I would also consider baselining the SFTP performance by itself so you can see what the theoretical maximum should be with your system + network connection prior to trying to get this info out of obnam tests themselves.

Additional data points

#1 - blog post comparing obnam vs. rsnapshot

Found this blog post where the author did a comparison of several options in this category. The article is titled: Comparing rsnapshot and obnam for scheduled large backups.

The article highlighted some very poor performance, IMO, with obnam that would seem to jive with what you're describing.

obnam performance

After backing up /home completely (which took several days!), a new run, several days later took (timing by the Linux time command):

Backed up 3443706 files, uploaded 94.0 GiB in 127h48m49s at 214.2 KiB/s average speed830 files; 1.24 GiB (0 B/s) real 7668m56.628s user 4767m16.132s sys 162m48.739s

From the obname log file:

 2012-11-17 12:41:34 INFO VFS: baseurl=/home read=0 written=0 2012-11-21 23:09:36 INFO VFS: baseurl=/backups/backup_home read=2727031576964 written=150015706142 2012-11-21 23:09:36 INFO Backup performance statistics: 2012-11-21 23:09:36 INFO * files found: 3443706 2012-11-21 23:09:36 INFO * uploaded data: 100915247663 bytes (93.9846482715 GiB) 2012-11-21 23:09:36 INFO * duration: 460128.627629s 2012-11-21 23:09:36 INFO * average speed: 214.179341663 KiB/s 2012-11-21 23:09:36 INFO Backup finished. 2012-11-21 23:09:36 INFO Obnam ends 2012-11-21 23:09:36 INFO obnam version 1.2 ends normally

So: ~5 days for backing up ~100 GB of changed data… Load was not high on the machines, neither in terms of CPU, nor in terms of RAM. Disk usage in /backups/backup_home was 5.7T, disk usage of /home was 6.6T, so there is some dedup, it seems.

rsnapshot performance

A full backup of /home to (according to the log file):

 [27/Nov/2012:12:55:31] /usr/bin/rsnapshot daily: started [27/Nov/2012:12:55:31] echo 17632 > /var/run/rsnapshot.pid [27/Nov/2012:12:55:31] mkdir -m 0700 -p /backups/backup_home_rsnapshot/ [27/Nov/2012:12:55:31] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/ [27/Nov/2012:12:55:31] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /home /backups/backup_home_rsnapshot/daily.0/localhost/ [28/Nov/2012:23:16:16] touch /backups/backup_home_rsnapshot/daily.0/ [28/Nov/2012:23:16:16] rm -f /var/run/rsnapshot.pid [28/Nov/2012:23:16:16] /usr/bin/rsnapshot daily: completed successfully

So: ~1.5 days for a full backup of 6.3TB. An incremental backup a day later took:

 [29/Nov/2012:13:10:21] /usr/bin/rsnapshot daily: started [29/Nov/2012:13:10:21] echo 20359 > /var/run/rsnapshot.pid [29/Nov/2012:13:10:21] mv /backups/backup_home_rsnapshot/daily.0/ /backups/backup_home_rsnapshot/daily.1/ [29/Nov/2012:13:10:21] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/ [29/Nov/2012:13:10:21] /usr/bin/rsync -a --delete --numeric-ids -- relative --delete-excluded --link-dest=/backups/backup_home_rsnapshot/daily.1/localhost/ /home/backups/backup_home_rsnapshot/daily.0/localhost/ [29/Nov/2012:13:25:09] touch /backups/backup_home_rsnapshot/daily.0/ [29/Nov/2012:13:25:09] rm -f /var/run/rsnapshot.pid [29/Nov/2012:13:25:09] /usr/bin/rsnapshot daily: completed successfully

So: 15 minutes... and the changed data amounted to 21GB.

*attic vs. obnam

Not as thorough but makes mention that one of the cons of obnam is that it's very slow vs. attic.

Obnam pros:

  • well documented
  • active mailing list
  • packages available

Obnam cons:

  • very slow
  • large backups

Attic pros:

  • much smaller backups (even without deduplication)
  • much better deduplication
  • much faster

Attic cons:

  • repository format not documented
  • not a large user community

Some testing data is shown which would seem to indicate that obnam is just really slow.

From local SSD to remote HD, over a so-so wifi connection:

 rsync: 0:24 0:01 Attic ssh: 0:28 0:05 Attic sshfs: 0:51 0:08 Obnam sftp: 8:45 0:21 Obnam sshfs: 25:22 0:22

References

Improve this answer
3

Obnam's default configuration (as of 2015-02-08) does not work well for backing up directories with a large number of small files. I had exactly the same problem as mentioned above.

The solution for me was to add --lru-size=8192 --upload-queue-size=8192 to the command line. This solved the problem and turned a frustrated into a very happy Obnam user. (I have these settings in my standard config files now.)

Unfortunately, Obnam's tutorial does not mention upfront how important these settings are. The FAQ gives more details. Setting the performance parameters is really mandatory on systems with many small files.

Improve this answer
1
  • 1
    Can you say what difference the new settings make to resource usage? Commented Nov 28, 2015 at 17:30

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.