Fastest way to read a csv file

Question

I'm looking for a very fast method to read a csv file. My data structure looks like this:

timestamp ,float , string ,ip ,string 1318190061,1640851625, lore ipsum,84.169.42.48,appname

and I'm using fgetcsv to read this data into arrays.

The problem: Performance. On a regular basis the script has to read (and process) more than 10,000 entries.

My first attempt is very simple:

//Performance: 0,141 seconds / 13.5 MB while(!feof($statisticsfile)) { $temp = fgetcsv($statisticsfile); $timestamp[] = $temp[0]; $value[] = $temp[1]; $text[] = $temp[2]; $ip[] = $temp[3]; $app[] = $temp[4]; }

My second attempt:

//Performance: 0,125 seconds / 10.8 MB while (($userinfo = fgetcsv($statisticsfile)) !== FALSE) { list ($timestamp[], $value[], $text, $ip, $app) = $userinfo; }

Is there any way to improve performance even further, or is my method as fast as it could get?
Probably more important: Is there any way to define what columns are read, e.g. sometimes only the timestamp, float columns are needed. Is there any better way than my way (have a look at my second attempt :)

Thanks :)

10,000 lines of CSV is NOTHING. The modifications you're doing are going shave (at most) a few milliseconds off a process that's only taking deciseconds anyways. Ask this kind of micro-optimization question when you reach millions or billions of rows. — Marc B
– Marc B, Commented Oct 9, 2011 at 21:07
Please go through the many existing questions on SO showing how to read CSV files with PHP. Take the examples given in answers and benchmark/profile them under your real world conditions. Then use what is fastest for your scenario. — Gordon
– Gordon, Commented Oct 9, 2011 at 21:11
@Gordon Could you please point out a faster/better example from these questions. I've read them the past couple of days and up there is the result. — Nyoman
– Nyoman, Commented Oct 9, 2011 at 21:17
@Nyoman even if I would be willing to do your work for you, I couldnt because I cannot profile the various possible solutions (fgetcsv, implode, SplFileObject, sscanf) in your real world environment. You have to do it yourself. — Gordon
– Gordon, Commented Oct 9, 2011 at 21:30
$data = array_map("str_getcsv", file($filename)); is the speediest method, as it reads in the whole CSV file at once. — mario
– mario, Commented Oct 9, 2011 at 21:47

Robin Millette · Accepted Answer · 2012-09-13 05:10:37Z

3

How long is the longest line? Pass that as the second parameter to fgetcsv() and you'll see the greatest improvement.

answered Sep 13, 2012 at 5:10

Robin Millette

1941 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user956584 · Accepted Answer · 2011-10-10 02:54:14Z

Check time that PHP read this file:

If is bigg move file to ramdisk or SSD

[..]sometimes only the timestamp

Somthing like that

preg_match_all('#\d{10},\d{10}, (.*?),\d.\d.\d.\d,appname#',$f,$res); print_r($res);

Collectives™ on Stack Overflow

Fastest way to read a csv file

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related