Skip successfully downloaded files using Wget

Question

So I have this Bash subroutine to download files using wget and my problem now is how to skip successfully downloaded files. The script downloads a lot of files and once the download fails, it re-downloads all files overwriting those successful downloads from the start (which may be incomplete because of the re-download).

So how do I skip those files downloaded successfully?

DownloadFile() { paramURL=$1 paramFilename=$2 if [ $flag_archive_fetch = "false" ]; then wget "--timeout=180" "--tries=5" "$paramURL" "-O" "${scratch_dir}$paramFilename" else unzip -o "$archive_file" "$paramFilename" -d "${scratch_dir}" fi touch "${scratch_dir}$paramFilename" }

davcs86 · Accepted Answer · 2015-09-01 02:35:58Z

7

You can take advantage of Wget --continue (to resume broken downloads) and --timestamping (to overwrite successfully downloaded files only when Last-modified attribute has changed, otherwise skips the download)

wget "--continue ‐‐timestamping --timeout=180" "--tries=5" "$paramURL" "-O" "${scratch_dir}$paramFilename"

another option is to use --no-clobber instead of --timestamping, it skips the already downloaded files without checking the Last-modified attribute,

 wget "--continue ‐‐no-clobber --timeout=180" "--tries=5" "$paramURL" "-O" "${scratch_dir}$paramFilename"

edited Sep 1, 2015 at 2:35

answered Sep 1, 2015 at 2:17

davcs86

3,9462 gold badges20 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Woootiness Over a year ago

thanks for the answer, so --continue will skip already present files? I thought it was only for incomplete downloads.

davcs86 Over a year ago

--continue for incomplete downloads, and --timestamping for skip already downloaded files.

Woootiness Over a year ago

tried with --continue and --no-clobber but the problem with no clobber is that you can't continue unfinished downloads

Terje Oseberg Over a year ago

"--no-clobber" should check if the file exists and not bother contacting the server, so how would it know whether or not the file is incomplete? "--continue" needs to check the size of the local file and the size on the server, so it needs to make a request to determine the size on the server in order to continue the download. These options are therefor in conflict with each other. ---- Woootiness needs to somehow record successful and unsuccessful downloads so that Woootiness can --no-clobber successfully downloaded file (or skip them entirely) or --continue if necessary.

dvhh · Accepted Answer · 2015-09-01 03:00:26Z

you could check the WGET exit status code by checking the $?

wget ..... # store the error error=$? if (( $error != 0 )) then #handle error else #handle success fi

Collectives™ on Stack Overflow

Skip successfully downloaded files using Wget

2 Answers 2

4 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Linked

Related