Timeline for Why is iterating over a file twice faster than reading it into memory and computing twice?
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 29, 2014 at 3:23 | history | edited | phunehehe | CC BY-SA 3.0 | deleted 10 characters in body |
| Apr 29, 2014 at 1:55 | history | edited | phunehehe | CC BY-SA 3.0 | deleted 9 characters in body |
| Mar 20, 2014 at 13:58 | history | tweeted | twitter.com/#!/StackUnix/status/446647018229096448 | ||
| Mar 20, 2014 at 12:40 | comment | added | kojiro | Now compare these to tail -n 10000 | fgrep -c '"success": true' and false. | |
| Mar 20, 2014 at 6:16 | vote | accept | phunehehe | ||
| Mar 20, 2014 at 2:37 | answer | added | Gilles 'SO- stop being evil' | timeline score: 12 | |
| Mar 19, 2014 at 14:08 | comment | added | psusi | @LaurentC., no, it is only read one time in the second example. There is only one call to tail. | |
| Mar 19, 2014 at 13:31 | answer | added | terdon♦ | timeline score: 3 | |
| Mar 19, 2014 at 12:55 | comment | added | Matt | In the case of extremely large files, memory constrained machines, or more items to grep for, you might see some speedup using tee so the file is definitely only read once. cat stdout.log | tee >/dev/null >(grep -c 'true'>true.cnt) >(grep -c 'false'>false.cnt); cat true.cnt; cat false.cnt | |
| Mar 19, 2014 at 9:47 | answer | added | mikeserv | timeline score: 3 | |
| Mar 19, 2014 at 9:07 | answer | added | X Tian | timeline score: 26 | |
| Mar 19, 2014 at 8:44 | history | edited | scai | added some tags | |
| Mar 19, 2014 at 8:25 | answer | added | scai | timeline score: 5 | |
| Mar 19, 2014 at 7:46 | comment | added | mikeserv | At least in the second example, your $( command substitution ) is not streamed. All of the rest happens through pipes concurrently, but in the second example you have to wait for the log= to complete. Try it with <<HERE\n${log=$(command)}\nHERE - see what you get. | |
| Mar 19, 2014 at 7:46 | comment | added | Laurent C. | Could it be because the second solution, the file content is read 3 times, and only two times in the first example? | |
| Mar 19, 2014 at 7:17 | history | asked | phunehehe | CC BY-SA 3.0 |