2

I have a db of files to download containing fields for filename and download_url with the format:

"foo-1.23.4.jar", "http://example.com/files/12345678/download" "bar-5.67.8.jar", "http://example.com/files/9876543/download" "baz-3.31.jar", "http://example.com/files/42424242/download"

Where the urls tend to all be the same except for the number. I tried exporting the list of URLs and downloading these with wget -i, but every file gets named download, and I have no way to tell them apart.

Normally I would use the -O parameter to specify the correct output file, but I'm not sure how to combine that with -i

How can I format the input file and command line to wget -i such that each line can specify both a download url and an output filename?

Improve this question

1 Answer 1

Reset to default
4

Concurrently with GNU parallel:

parallel -a files_list.txt -j0 -C ", *" 'wget -q -O {1} {2}'
  • -a input-file - use input-file as input source
  • -j N - run up to N jobs in parallel. 0 means as many as possible
  • -C regex - column separator. The input will be treated as a table with regexp separating the columns. The n'th column can be access using {n} or {n.}. E.g. {3} is the 3rd column
Improve this answer
2
  • I like this. I had not heard of parallel. Does this mean that wget does not directly support what I want to do, and the solution is multiple invocations? Commented Jan 23, 2018 at 21:40
  • @RyanBemrose, this solution provides you with effective and fast downloading process: you are able to download multiple resources in parallel. Your input file does not fit the wget's -i option because it would expect the file to contain only URLs while your file has a more complex structure Commented Jan 23, 2018 at 21:48

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.