Can xargs be used to run several arbitrary commands in parallel?

Question

I'd like to be able to provide a long list of arbitrary/different commands (varying binary/executable and arguments) and have xargs run those commands in parallel (xargs -P).

I can use xargs -P fine when only varying arguments. It's when I want to vary the executable and arguments that I'm having difficulty.

Example: command-list.txt

% cat command-list.txt binary_1 arg_A arg_B arg_C binary_2 arg_D arg_E .... <lines deleted for brevity> binary_100 arg_AAA arg_BBB % xargs -a command-list.txt -P 4 -L 1 ** I know the above command will only echo my command-list.txt **

I am aware of GNU parallel but can only use xargs for now. I also can't just background all the commands since there could be too many for the host to handle at once.

Solution is probably staring me in the face. Thanks in advance!

Are any of the reasons for not using GNU Parallel covered on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html — Ole Tange
– Ole Tange, Commented Jun 5, 2013 at 18:04
I don't have permission to copy GNU Parallel (or other software) anywhere onto the network I'm using at this time. — Charlie Hemlock
– Charlie Hemlock, Commented Jun 5, 2013 at 20:35

inthecrossfire · Accepted Answer · 2013-11-22 22:17:52Z

If you don't have access to parallel, one solution is just to use sh with your command as the parameter.

For example:

xargs -a command-list.txt -P 4 -I COMMAND sh -c "COMMAND"

The -c for sh basically just executes the string given (instead of looking for a file). The man page explanation is:

-c string If the -c option is present, then commands are read from string. If there are arguments after the string, they are assigned to the positional parameters, starting with $0.

And the -I for xargs tells it to run one command at a time (like -L 1) and to search and replace the parameter (COMMAND in this case) with the current line being processed by xargs. Man page info is below:

-I replace-str Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character. Implies -x and -L 1.

sh seems to be very forgiving with commands containing quotations marks (") so you don't appear to need to regexp them into escaped quotations.

Collectives™ on Stack Overflow

Can xargs be used to run several arbitrary commands in parallel?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related