43

I want to download a bunch of files named with ISO-8601 dates. Is there a simple way to do this using bash+GNU coreutils? (Or some trick to make wget/curl to generate the list automatically, but I find that unlikely)

Similar to this question, but not restricted to weekdays: How to generate a range of nonweekend dates using tools available in bash?. I guess that there is a simpler way to do it without that restriction.

Also related to How to generate date range for random data on bash, but not restricted to a single year.

Improve this question
0

5 Answers 5

Reset to default
78

If you have GNU date, you could do use either a for loop in any POSIX-compliant shell:

# with "for" for i in {1..5}; do # ISO 8601 (e.g. 2020-02-20) using -I date -I -d "2014-06-28 +$i days" # custom format using + date +%Y/%m/%d -d "2014-06-28 +$i days" done

or an until loop, this time using Bash's extended test [[:

# with "until" d="2014-06-29" until [[ $d > 2014-07-03 ]]; do echo "$d" d=$(date -I -d "$d + 1 day") done

Note that non-ancient versions of sh will also do lexicographical comparison if you change the condition to [ "$d" \> 2014-07-03 ].

Output from either of those loops:

2014-06-29 2014-06-30 2014-07-01 2014-07-02 2014-07-03

For a more portable way to do the same thing, you could use a Perl script:

use strict; use warnings; use Time::Piece; use Time::Seconds; use File::Fetch; my ($t, $end) = map { Time::Piece->strptime($_, "%Y-%m-%d") } @ARGV; while ($t <= $end) { my $url = "http://www.example.com/" . $t->strftime("%F") . ".log"; my $ff = File::Fetch->new( uri => $url ); my $where = $ff->fetch( to => '.' ); # download to current directory $t += ONE_DAY; }

Time::Piece, Time::Seconds and File::Fetch are all core modules. Use it like perl wget.pl 2014-06-29 2014-07-03.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

The first one can be combined with stackoverflow.com/a/8903280/939108 to use an explicit date. But I guess the second option is better then.
If you want a format other than ISO-8601, you can do date -d $d +"%m/%d/" or whatever formatting string strikes your fancy instead of the echo $d line above.
The until loop is better for "days in a month".
30

Using GNU date and bash:

start=2014-12-29 end=2015-01-03 while ! [[ $start > $end ]]; do echo $start start=$(date -d "$start + 1 day" +%F) done 
2014-12-29 2014-12-30 2014-12-31 2015-01-01 2015-01-02 2015-01-03
Improve this answer

2 Comments

Why not put the test in the while clause? while [[ $start -le $end ]]; do
I mean while [[ ! $start > $end ]]; do. The previous one doesn't work.
12

If you are on macOS, then date works a bit differently from GNU date. Here's a variant to Tom Fenech's date invocation that supports both GNU and Darwin:

if [ $(uname) = 'Darwin' ]; then d=$(date -j -v+1d -f %Y-%m-%d $d +%Y-%m-%d) elif [ $(uname) = 'Linux' ]; then d=$(date -I -d "$d + 1 day") fi
Improve this answer

3 Comments

Or just brew install coreutils, then use gdate
Check the day offset, needs a -v+${i}d for this to work in the @Tom Fenech example. MacOSX 12 zsh: start=2020-01-01; for i in {1..5}; do date -j -v+${i}d -f %Y-%m-%d "$start" +%Y-%m-%d; done
@ijoseph thx 1K for coreutils suggestion!
2

I use this handy function to work with log files in the format yyyymmdd.log.gz: 

function datelist { for dt in $(seq -w $1 $2) ; do date -d $dt +'%Y%m%d' 2>/dev/null ; done ; }

It accepts dates in the format yyyymmdd.

Improve this answer

2 Comments

This can of course be extremely inefficient if you need big date range. It will generate up to 70 useless days per month and 88 useless months per year, which it will feed to date which will give an error.
@mivk: Indeed. Good if you want a couple of years and if you run it on a fat machine. I haven't given it a second thought since I posted it. In my work, back then, I queried the SQL server, which can return a list of dates and does not need checking. But not everybody has an SQL server running and not all servers are as fast as the corporate one I used.
-2

Since I didn’t know the exact date range, only that the last file was today, I ended up using this variation of the top answer, which starts at today’s date and iterates backwards one day at a time until a download fails (presumably with a 404):

d=$(date -I); while wget "http://www.example.com/$d.log"; do d=$(date -I -d "$d - 1 day"); done
Improve this answer

2 Comments

Just fyi, people don't like when you answer your own question with a quick n dirty solution when some research might give you a more refined approach
@pythonian29033 that’s the solution I used after reading the top answer though, since this was a slight AB problem. i didn’t know the exact date range, so fetching until 404 was the best for me

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.