Timeline for Spreading stdin to parallel processes
Current License: CC BY-SA 3.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 18, 2019 at 16:58 | answer | added | Ole Tange | timeline score: 0 | |
| Sep 12, 2017 at 9:40 | history | edited | Jeff Schaller♦ | edited tags | |
| Dec 3, 2013 at 16:16 | answer | added | peterph | timeline score: 0 | |
| Oct 21, 2013 at 23:16 | answer | added | dannysauer | timeline score: 1 | |
| Oct 18, 2013 at 7:55 | answer | added | kouk | timeline score: 0 | |
| Aug 26, 2013 at 22:07 | answer | added | ash | timeline score: 0 | |
| Aug 26, 2013 at 21:50 | comment | added | ash | @Craig - he wants minimal latency to start every job, so it's important not to feed multiple args into a single job. Every arg takes a different amount of time to process. | |
| Mar 26, 2013 at 17:36 | answer | added | estani | timeline score: 1 | |
| Feb 16, 2013 at 8:48 | answer | added | Johan | timeline score: 0 | |
| Jan 26, 2013 at 7:26 | comment | added | Scott Lamb | I doubt any standard utility will do this; I would roll my own, perhaps using Python. How large in bytes is the list of files? I ask because using the pipe being unblocked to indicate idle means the processes' workloads can be out of balance by one pipe buffer (by default, 64K; always at least 4K), perhaps more if the process buffers internally. If that's not acceptable, you'll need a new mechanism. | |
| Oct 10, 2012 at 14:18 | answer | added | Ole Tange | timeline score: 1 | |
| Oct 10, 2012 at 4:40 | comment | added | cas | why are you using -l 1 in the parallel args? IIRC, that tells parallel to process one line of input per job (i.e. one filename per fork of myjob, so lots of startup overhead). | |
| Oct 10, 2012 at 1:42 | comment | added | Gilles 'SO- stop being evil' | Assuming that the program isn't using bufering on stdin, a FUSE filesystem that reacts to read calls would do the trick. That's a fairly large programming endeavor. | |
| Oct 10, 2012 at 1:21 | comment | added | Gilles 'SO- stop being evil' | I've been thinking about that, and a fundamental problem is knowing that an instance of myjob is ready to receive more input. There is no way to know that a program is ready to process more input, all you can know is that some buffer somewhere (a pipe buffer, an stdio buffer) is ready to receive more input. Can you arrange for your program to send some kind of request (e.g. display a prompt) when it's ready? | |
| Oct 9, 2012 at 22:44 | comment | added | BCoates | @Gilles, it's the GNU one: "split (GNU coreutils) 8.13". Using it as a weird alternative to xargs is probably not the intended use but it's the closest to what I want I've found. | |
| Oct 9, 2012 at 22:27 | comment | added | Gilles 'SO- stop being evil' | Where is that split command from? The name conflicts with the standard text processing utility. | |
| Oct 9, 2012 at 21:48 | answer | added | Bananguin | timeline score: 0 | |
| Oct 9, 2012 at 21:01 | history | tweeted | twitter.com/#!/StackUnix/status/255774963330584576 | ||
| Oct 9, 2012 at 20:46 | history | asked | BCoates | CC BY-SA 3.0 |