Timeline for Why is head -c (and similar commands) much slower for large files than small files?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 9, 2024 at 7:15 | answer | added | Kaz | timeline score: 0 | |
| Jul 8, 2024 at 18:19 | history | rollback | Jack | Rollback to Revision 2 | |
| Jul 8, 2024 at 18:00 | history | edited | G-Man Says 'Reinstate Monica' | CC BY-SA 4.0 | Tweaked punctuation and capitalization; changed tags (the question is not about bash or C++). |
| Jul 8, 2024 at 17:11 | comment | added | Jack | Thanks very much Marcus Müller that's helpful to know | |
| Jul 8, 2024 at 17:11 | history | edited | Marcus Müller | CC BY-SA 4.0 | added 22 characters in body |
| Jul 8, 2024 at 17:08 | comment | added | Marcus Müller | EOS: yep, that's a userland network file system. It's not head that takes time, it's your file system; and quite likely it's not even the 4 bytes you read, but the metadata that needs to be requested from the MGM instance.. That's a pretty specific tool for a pretty specific (CERN) job. Maybe, if you just happen to need known ranges from files, just going through the XRootD or HTTPS access protocols would be easier. Quite possibly, asking your instance admin whether they offer CIFS/Samba access, S3/Minio or XrdHttp access is clever here. | |
| Jul 8, 2024 at 17:01 | comment | added | Jack | EOS* *I need to enter more characters | |
| Jul 8, 2024 at 16:59 | comment | added | choroba | What file system do you use? | |
| Jul 8, 2024 at 16:57 | comment | added | Jack | I guess if it takes the same amount of time for you for huge files and small files it must just be something to do with my filesystem, I assume it stages the entirety of large files before it reads any part of them or something like that. Thanks | |
| Jul 8, 2024 at 16:43 | comment | added | Jack | Hi @StephenKitt , I don't think so. If I run top while doing e.g. head -c 4 bigFile , as soon as I do the head command appears in top and nothing else, and remains in top until a few minutes later when it finishes | |
| Jul 8, 2024 at 16:33 | comment | added | Stephen Kitt | You wouldn’t happen to have an antivirus running or some other program which scans files you try to read before letting the real caller actually read them, would you? On my system head -c 4 works just as fast on huge files as it does on small ones. | |
| S Jul 8, 2024 at 16:28 | review | First questions | |||
| Jul 18, 2024 at 11:54 | |||||
| S Jul 8, 2024 at 16:28 | history | asked | Jack | CC BY-SA 4.0 |