I have a csv of JPG url extensions.
http://www.example.com/images/[url_extension] I want to use curl to loop through the CSV and download the jpg at each extension. So far I have the following, but I'm struggling with syntax. Any help is greatly appreciated.
#!/bin/bash file=urlextensions.csv while read line do outfile=$(echo $line | awk 'BEGIN { FS = "/" } ; {print $NF}') curl -o "$http://www.example.com/images/" "$line" done < "$/Users/Me/Documents/urlextensions.csv" There are a couple of errors in your code:
file in line 2, but then you don't use it in the loop.$ in front of things will make the shell try to substitute it which probably isn't what you want when you do $http or $/Users.outfile in your loop, but you don't use it. Maybe you meant to put it after the -o on your curl line.-o argument to curl should be a filename, but you put the URL there.http://www.example.com/images) and the part you add to it need to be in the same argument, not separated by a space which will make the shell think it takes two arguments.So I end up with:
#!/bin/bash filename=./extensions.txt while read line || [[ -n "$line" ]]; do echo downloading $line curl -o $line "http://example.com/$line" done < "$filename" If you put that in a file name read_examp and make it executable you can see it work like:
chicks$ cat extensions.txt foo bar baz chicks$ ./read_examp foo % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1270 100 1270 0 0 41794 0 --:--:-- --:--:-- --:--:-- 42333 bar % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1270 100 1270 0 0 53987 0 --:--:-- --:--:-- --:--:-- 55217 baz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1270 100 1270 0 0 48366 0 --:--:-- --:--:-- --:--:-- 48846 chicks$ ls -l `cat extensions.txt` -rw-r--r-- 1 chicks staff 1270 Oct 7 10:01 bar -rw-r--r-- 1 chicks staff 1270 Oct 7 10:01 baz -rw-r--r-- 1 chicks staff 1270 Oct 7 10:01 foo Note: you mention CSV's but your example code does not seem to deal with that at all. You can extend this with something like this to pull a field out of your CSV instead of using the whole line.
