2

I am looking to compare lots of local files with remote files in a http repository. The idea is that a bash script will check every file in the directory and compare with the remote version. If the remote version is different or only present remotely download and replace it.

I am looking at using md5 and maybe creating an md5 dir mirroring the repo with .md5 files. However the only issue is that to get the md5sum for each file can be slow and resource intensive locally as I am dealing with a few GB worth of files.

So my questions is what is the fastest method to compare the files?

Thank you

Improve this question
7
  • compare only the file size + date. Commented Aug 29, 2016 at 14:54
  • 2
    This is exactly what the rsync program is designed to do; detect differences and update the target if it's changed. Commented Aug 29, 2016 at 15:02
  • so can you use rsync with http urls? Commented Aug 29, 2016 at 15:25
  • What's the end goal? Why compare the files? If you want to compare a hash of the two files you will need to either download the remote file in order to hash it or have a way of running the hash command on the remote host. If you are able to run arbitrary commands on the remote host, I suggest using a protocol other than http (such as rsync, which has already been suggested). Commented Aug 29, 2016 at 15:38
  • the remote host will be a repo of files for a game server that are publicly accessible. The local machine can be anyone's server that wants to host a game server. So im not sure having anyone login to the remote server with ssh would be a good idea? Unless I can correctly lock it down I guess. Should an update for the game server be released the remote repo can be updated with the new files. The local game server can compare these and download required files. Kind of like how SteamCMD downloads updates to game servers. My idea was for the to use .md5 files that I generate on remote host. Commented Aug 29, 2016 at 16:08

1 Answer 1

Reset to default
1

You could use the wget command with the -N flag. The -N flag causes wget to download only the files that are newer on the server or missing from the local directory.

Improve this answer
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.