Timeline for Wget batch download and save failed links
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 23, 2016 at 8:18 | comment | added | Timothy Truckle | @InverseTelecine " I don't know what the script.log part is intended to do in your example " it fullfills your firs replies argument: "I do have a logging function set up in the bash script to create a log,". | " Your method will save a log with information about all the downloads, successful and failed" no, it only logs faild downloads unless the implementation of wget changed meanwhile. " containing just the failed URL lines and nothing else" this is the closesd you can get with standard tooling. You may useawk or sed to shrink wget-fail.log | |
| Nov 23, 2016 at 1:17 | comment | added | InverseTelecine | Thank you for your help @Timothy. I'm sorry if I'm being dense, but I still don't understand. Your method will save a log with information about all the downloads, successful and failed, but what I tried to say I wanted was a text/log file containing just the failed URL lines and nothing else, like the input file. With your method there's a lot of extra text like download progress and HTTP request sent, awaiting response... 200 OK Length: 180524 (176K) and stuff. Unless I am misunderstanding, which is very possible. I don't know what the script.log part is intended to do in your example | |
| Nov 15, 2016 at 22:05 | history | edited | Timothy Truckle | CC BY-SA 3.0 | added 860 characters in body |
| Nov 15, 2016 at 21:35 | comment | added | InverseTelecine | I do have a logging function set up in the bash script to create a log, which saves both stdout and stderr. Just using 2> seems to create a log file with a similarly huge amount of information in it, including successful downloads. I could approach the problem from the angle of parsing the log file, but that seemed like the wrong way to go about it, and figuring out how to parse out just the failed URLs would take me a long time. Do you know of any tricks to parse out just the URLs of the failed downloads easily? Thanks @Timothy | |
| Nov 15, 2016 at 21:09 | history | answered | Timothy Truckle | CC BY-SA 3.0 |