So I have a while loop:
cat live_hosts | while read host; do \ sortstuff.sh -a "$host" > sortedstuff-"$host"; done But this can take a long time. How would I use GNU Parallel for this while loop?
Add a comment |
3 Answers
You don't use a while loop.
parallel "sortstuff.sh -a {} > sortedstuff-{}" <live_hosts Note that this won't work if you have paths in your live_hosts (e.g. /some/dir/file) as it would expand to sortstuff.sh -a /some/dir/file > sortedstuff-/some/dir/file (resulting in no such file or directory); for those cases use {//} and {/} (see gnu-parallel manual for details):
parallel "sortstuff.sh -a {} > {//}/sortedstuff-{/}" <live_hosts 
- Is it possible to use
teewithparallelwhen putting the output intosortedstuff? So I can see the output as it goes.Proletariat– Proletariat2015-09-22 14:16:21 +00:00Commented Sep 22, 2015 at 14:16 - 1@Proletariat - you want to output to terminal too ? Just replace
>with| teee.g. the first command becomesparallel "sortstuff.sh -a {} | tee sortedstuff-{}" <live_hostsdon_crissti– don_crissti2015-09-22 15:10:36 +00:00Commented Sep 22, 2015 at 15:10
As an old-school "do one thing and do it well" Unix guy, I'd put the string substitution stuff into a wrapper script:
#!/bin/sh sortstuff.sh -a "$1" > sortedstuff-"$1" If you call it wrapper.sh, the parallel command to call it would be:
parallel wrapper.sh < live_hosts Note that you don't need cat for this kind of thing, which saves an external program invocation.
You don't need parallel, since the body of the loop doesn't depend on previous iterations. Just start a new background process for each host.
while read host; do sortstuff.sh -a "$host" > sortedstuff-"$host" & done < live_hosts wait # Optional, to block until the background tasks are done parallel does make it easier to manage certain aspects, though; you can limit the number of jobs running in parallel more easily.
- 6If
wc -l live_hostsis more than the number of disk spindles or CPU cores — depending on whether the task is I/O or CPU-bound — you're going to eat up a lot of the advantage you get from parallelism with a solution like that. The ability ofparallelto limit the number of jobs isn't just nice, it's near-essential, if processing speed is your goal.Warren Young– Warren Young2015-09-11 20:47:23 +00:00Commented Sep 11, 2015 at 20:47