I have sample CSV data file with 1M records with 1 date (14/03/2017 00:11:17) only. I need to generate 6 month data from this sample CSV file. Bash script taking 20 Minutes to generate 1 day data.
DATA SAMPLE
EXPECTED RESULT
'01/01/2017 03:22:17,sampledata,1234,sample,123455,67546464'
'02/01/2017 03:22:17,sampledata,1234,sample,123455,67546464'
to
'30/01/2017 03:22:17,sampledata,1234,sample,123455,67546464'
cat 6months.pl #!/usr/bin/env perl use Text::CSV; use DateTime; use DateTime::Format::Strptime; use autodie qw/ open close /; my $csv = Text::CSV->new({binary => 1, quote_space => 0}); my $dateparser = DateTime::Format::Strptime->new(pattern => "%d/%m/%Y %T", time_zone => "local"); for my $file (@ARGV) { open my $fh, '<', $file; while (my $row = $csv->getline($fh)) { my $datestr = shift @$row; my $date = $dateparser->parse_datetime($datestr)->truncate(to => month); my $end = $date->clone->add(months => 6); while ($date <= $end) { $csv->say(STDOUT, [$dateparser->format_datetime($date), @$row]); $date = $date->add(days => 1); } } close $fh; } Running it:
perl 6months.pl data.csv 01/01/2017 00:00:00,sampledata,1234,sample,123455,67546464 02/01/2017 00:00:00,sampledata,1234,sample,123455,67546464 ... 30/06/2017 00:00:00,sampledata,1234,sample,123455,67546464 01/07/2017 00:00:00,sampledata,1234,sample,123455,67546464 Just noticed this resets the time to midnight. If you want to keep the time, do this instead:
my $date = $dateparser->parse_datetime($datestr)->set(day => 1); # ^^^^^^^^^^^^^ 