Skip to main content
separate out "what should you do instead" part.
Source Link
Marcus Müller
  • 52.7k
  • 4
  • 80
  • 123

Let's define roles here:

  • primary: the machine you're backing up from
  • store: the machine holding the backups
  • secondary: the machine you're getting the backups to from store.

I'd go about this as follows:

  1. install restic on both primary and secondary. Assuming you're on a non-esoteric Linux distro, Mac OS, Free- or OpenBSD, that's one {your package manager} install restic away, see instructions.
  2. Set up SSH public key authentication for the user root for the store server as described above, i.e., run sudo -i -H ssh-keygen and sudo -H -i ssh-copy-id [email protected] on both primary and secondary.
  3. generate a secure password in a file only readable by root. On the primary, machine run, `sudo sh -c 'touch /etc/restic-backup-pass && chmod 600 /etc/restic-backup-pass && head -c 256 /dev/random > /etc/restic-backup-pass'.
  4. Copy that file to the same location on the secondary. Make sure it's still only readable by root.
  5. Initialize the repository. On primary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" init 
  1. Run the first backup: on the primary, do
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home 
  1. Get the backup: on the secondary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home 

When 1. – 7. work, well, time for automation. On the primary, you create a service that does the backup, on the secondary, you create a service that restores the backup. You run the service on the primary a) on every shutdown, and b) every day at noon, and you run the service on the secondary a) on every boot before nginx starts and b) every day at 13h. (and of course, you can start it easily manually).

To set up the backup service: onI've written down what I would do primaryinstead,

  1. run (I'm assuming nvim is your favourite text editor. If not, replace it with something else, or omit the EDITOR=nvim altogether to use the default editor):
sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.service 

and put in something like

[Unit] Description="Backing up /home to store" Wants=network.target After=network.target [Service] ExecStart=/usr/bin/restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home Type=oneshot [Install] WantedBy=shutdown.target 

and save and exit the editor. 9. Test that service: sudo systemd-analyze verify /etc/systemd/system/backup-*. If that's OK, sudo systemctl start backup-to-store.serviceyour situation, followed by journalctl -xef. You should see the output of restic in that system log! 10. Enable the service to be automatically run at shutdown: sudo systemctl enable backup-to-store.service 11. Add a timer that runs the service Monday through Saturday at noon (note the .timer):

sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.timer 

contents would be:

[Unit] Description="Run workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 12:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/backup-*, and then enable the timer:

sudo systemctl enable backup-to-store.timer 

Great! Now we have automated, and incremental backups with passwords and authentication keys unreadable to normal users.

Time to work on restoring. On secondary,

  1. run
sudo env EDITOR=nvim systemctl --force --full edit get-backup-from-store.service 

with contents something like

[Unit] Description="Restoring /home from store" Wants=network.target After=network.target Before=nginx.service [Service] ExecStart=restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home Type=oneshot [Install] WantedBy=multi-user.target 

verify and enablemy answer here: sudo systemd-analyze verify /etc/systemd/system/get-backup-*, sudo systemctl enable get-backup-from-store.service 
13. Set up the restore timer:

with content

[Unit] Description="Get workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 13:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/get-backup-*, and then enable the timer, sudo systemctl enable get-backup-from-store.timer.

Great, now we have automated, password-safe, incremental, backup restoreHow do I automatically do daily backups and on every shutdown, and restore them elsewhere daily and on boot? on the secondary.

You can run a backup manually by starting the service you created manually on the primary (sudo systemctl start backup-to-store.service) and you can download the latest manually by running the service on the secondary manually (sudo systemctl start get-backup-from-store.service).

Let's define roles here:

  • primary: the machine you're backing up from
  • store: the machine holding the backups
  • secondary: the machine you're getting the backups to from store.

I'd go about this as follows:

  1. install restic on both primary and secondary. Assuming you're on a non-esoteric Linux distro, Mac OS, Free- or OpenBSD, that's one {your package manager} install restic away, see instructions.
  2. Set up SSH public key authentication for the user root for the store server as described above, i.e., run sudo -i -H ssh-keygen and sudo -H -i ssh-copy-id [email protected] on both primary and secondary.
  3. generate a secure password in a file only readable by root. On the primary, machine run, `sudo sh -c 'touch /etc/restic-backup-pass && chmod 600 /etc/restic-backup-pass && head -c 256 /dev/random > /etc/restic-backup-pass'.
  4. Copy that file to the same location on the secondary. Make sure it's still only readable by root.
  5. Initialize the repository. On primary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" init 
  1. Run the first backup: on the primary, do
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home 
  1. Get the backup: on the secondary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home 

When 1. – 7. work, well, time for automation. On the primary, you create a service that does the backup, on the secondary, you create a service that restores the backup. You run the service on the primary a) on every shutdown, and b) every day at noon, and you run the service on the secondary a) on every boot before nginx starts and b) every day at 13h. (and of course, you can start it easily manually).

To set up the backup service: on primary,

  1. run (I'm assuming nvim is your favourite text editor. If not, replace it with something else, or omit the EDITOR=nvim altogether to use the default editor):
sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.service 

and put in something like

[Unit] Description="Backing up /home to store" Wants=network.target After=network.target [Service] ExecStart=/usr/bin/restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home Type=oneshot [Install] WantedBy=shutdown.target 

and save and exit the editor. 9. Test that service: sudo systemd-analyze verify /etc/systemd/system/backup-*. If that's OK, sudo systemctl start backup-to-store.service, followed by journalctl -xef. You should see the output of restic in that system log! 10. Enable the service to be automatically run at shutdown: sudo systemctl enable backup-to-store.service 11. Add a timer that runs the service Monday through Saturday at noon (note the .timer):

sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.timer 

contents would be:

[Unit] Description="Run workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 12:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/backup-*, and then enable the timer:

sudo systemctl enable backup-to-store.timer 

Great! Now we have automated, and incremental backups with passwords and authentication keys unreadable to normal users.

Time to work on restoring. On secondary,

  1. run
sudo env EDITOR=nvim systemctl --force --full edit get-backup-from-store.service 

with contents something like

[Unit] Description="Restoring /home from store" Wants=network.target After=network.target Before=nginx.service [Service] ExecStart=restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home Type=oneshot [Install] WantedBy=multi-user.target 

verify and enable: sudo systemd-analyze verify /etc/systemd/system/get-backup-*, sudo systemctl enable get-backup-from-store.service 13. Set up the restore timer:

with content

[Unit] Description="Get workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 13:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/get-backup-*, and then enable the timer, sudo systemctl enable get-backup-from-store.timer.

Great, now we have automated, password-safe, incremental, backup restore on the secondary.

You can run a backup manually by starting the service you created manually on the primary (sudo systemctl start backup-to-store.service) and you can download the latest manually by running the service on the secondary manually (sudo systemctl start get-backup-from-store.service).

I've written down what I would do instead in your situation, see my answer here:  
How do I automatically do daily backups and on every shutdown, and restore them elsewhere daily and on boot?

Source Link
Marcus Müller
  • 52.7k
  • 4
  • 80
  • 123

Admittedly, a bit of speculation, because I can't know how all your SSH and askpass are set up, but, since that's the answer to easily 90% of crontab-related questions here:

When you run something from crontab, the environment is different than when you run it from your session. That's an architectural limitation of crontab; it just doesn't start an interactive login shell, but simply runs the specified commands as your user, which is not the same.

Paired with the fact that you're doing two user changes (through your double-sudo), my guess is that scp/ssh look into the wrong configuration files, or can't read them. Doesn't really matter: you'd really want to do this without the sshpass and without the sudo:

  1. This is clearly a system job – the fact that you need to become root through the use of sudo shows that beyond doubt. So, instead of your user's crontab, this should be root's crontab. This completely removes the need for sudo.
  2. Since you seem to control the backup server, you very clearly should not be using password authentication (since your the password as part of the command line can be read by any process, including rogue PHP scripts etc through simple things like ps -AF as long as the backup job runs). I'd consider that password compromised now and would heavily suggest you change it immediately. Especially since you just told us one character of your password and that makes it easy for any interested party to be sure that they found the right one. Also, make sure your passwords are securely generated and not just something like "usernamePassword!" or other easy to automatically try out combination. (Guidedly) Brute forcing passwords is still a thing.
    Passwordless public-key authentication for user root is really easy to set up (sudo -i -H ssh-keygen, simply press enter to accept defaults and don't specify a password; then sudo -H -i ssh-copy-id [email protected], enter password once, done). You want to do this as root, since you want these keys to be available (only!) to the root user, not your normal local user (which has no business being able to log in without knowing the password). No excuses there.
  3. Since that reduces the amount of surprise in the environment you get, I'd personally recommend (that's a preference, so hm, heed it or not) that you'd avoid using crontabs alltogether, and just go for the easier to set up systemd-timers. (That has a lot of advantages, as the environment is then more like an interactive session, but you also get better logging, the ability to say something like "OK, get the backup every workday at 00:13:00 +- randomized 10min and every boot just before nginx runs"). If you're interested in getting that to work and this instruction isn't helpful (and you can't find an existing answer here), do open a new question post!
  4. Your scp seems to be, all in all, a bad solution: it copies the whole backup, whether all files (or any) changed or not, resets the modification time (which, by the way, your nginx delivers to the clients, which means you're invalidating their caches every day, thus putting unnecessary load on your server), and isn't careful with file attributes. At the very least here, you'd use rsync instead of scp. More realistically, though:

Aside from changing your auth method, which you really must do here, and from changing the user trying to download the backup, you should probably think of this in a less "bespoke scripts that work (or don't) for you today" way, to be honest.

Let's define roles here:

  • primary: the machine you're backing up from
  • store: the machine holding the backups
  • secondary: the machine you're getting the backups to from store.

I'd go about this as follows:

  1. install restic on both primary and secondary. Assuming you're on a non-esoteric Linux distro, Mac OS, Free- or OpenBSD, that's one {your package manager} install restic away, see instructions.
  2. Set up SSH public key authentication for the user root for the store server as described above, i.e., run sudo -i -H ssh-keygen and sudo -H -i ssh-copy-id [email protected] on both primary and secondary.
  3. generate a secure password in a file only readable by root. On the primary, machine run, `sudo sh -c 'touch /etc/restic-backup-pass && chmod 600 /etc/restic-backup-pass && head -c 256 /dev/random > /etc/restic-backup-pass'.
  4. Copy that file to the same location on the secondary. Make sure it's still only readable by root.
  5. Initialize the repository. On primary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" init 
  1. Run the first backup: on the primary, do
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home 
  1. Get the backup: on the secondary, run
sudo restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home 

When 1. – 7. work, well, time for automation. On the primary, you create a service that does the backup, on the secondary, you create a service that restores the backup. You run the service on the primary a) on every shutdown, and b) every day at noon, and you run the service on the secondary a) on every boot before nginx starts and b) every day at 13h. (and of course, you can start it easily manually).

To set up the backup service: on primary,

  1. run (I'm assuming nvim is your favourite text editor. If not, replace it with something else, or omit the EDITOR=nvim altogether to use the default editor):
sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.service 

and put in something like

[Unit] Description="Backing up /home to store" Wants=network.target After=network.target [Service] ExecStart=/usr/bin/restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" backup /home Type=oneshot [Install] WantedBy=shutdown.target 

and save and exit the editor. 9. Test that service: sudo systemd-analyze verify /etc/systemd/system/backup-*. If that's OK, sudo systemctl start backup-to-store.service, followed by journalctl -xef. You should see the output of restic in that system log! 10. Enable the service to be automatically run at shutdown: sudo systemctl enable backup-to-store.service 11. Add a timer that runs the service Monday through Saturday at noon (note the .timer):

sudo env EDITOR=nvim systemctl --force --full edit backup-to-store.timer 

contents would be:

[Unit] Description="Run workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 12:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/backup-*, and then enable the timer:

sudo systemctl enable backup-to-store.timer 

Great! Now we have automated, and incremental backups with passwords and authentication keys unreadable to normal users.

Time to work on restoring. On secondary,

  1. run
sudo env EDITOR=nvim systemctl --force --full edit get-backup-from-store.service 

with contents something like

[Unit] Description="Restoring /home from store" Wants=network.target After=network.target Before=nginx.service [Service] ExecStart=restic --password-file /etc/restic-backup-pass --repo "sftp://[email protected]:restic" restore latest --target /home Type=oneshot [Install] WantedBy=multi-user.target 

verify and enable: sudo systemd-analyze verify /etc/systemd/system/get-backup-*, sudo systemctl enable get-backup-from-store.service 13. Set up the restore timer:

with content

[Unit] Description="Get workday backup" [Timer] OnCalendar=Mon..Sat *-*-* 13:00:* Unit=backup-to-store.service [Install] WantedBy=multi-user.target 

and verify, sudo systemd-analyze verify /etc/systemd/system/get-backup-*, and then enable the timer, sudo systemctl enable get-backup-from-store.timer.

Great, now we have automated, password-safe, incremental, backup restore on the secondary.

You can run a backup manually by starting the service you created manually on the primary (sudo systemctl start backup-to-store.service) and you can download the latest manually by running the service on the secondary manually (sudo systemctl start get-backup-from-store.service).