13

Here's a basic question. I'm curious as to how do xargs and gnu parallel differ when parallelizing code?

And are there use cases in which you'd use one over the other?

I ask this because I have seen answers to parallelization questions where using either tool has been deemed acceptable by the community.

Improve this question
1
  • 3
    The answer to this might be quite contentious, and everyone else's mileage may vary (wildly), but IMHO GNU Parallel is significantly more powerful, flexible, configurable and capable, but less likely to be present on any given system. POSIX advocates may have differing views. Decide for yourself after reading this google.com/… Commented Jun 22, 2020 at 11:05

1 Answer 1

Reset to default
10

Some of the differences are covered on: https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel

Tl;dr: xargs is faster because there is almost no overhead (~0.3 ms/job compared to GNU Parallel's ~3 ms/job). GNU Parallel is safer because it takes all sorts of precautions so you do not need to worry (e.g. output from two jobs running in parallel will not mix). GNU Parallel has loads of features that xargs does not have. GNU Parallel requires Perl, xargs does not. xargs is everywhere, GNU Parallel requires you to use --embed to make sure it is everywhere.

So in general: If the primary concern is to avoid overhead (e.g. if your jobs take a few ms to run each) or avoid installing Perl (e.g. if your system is embedded and thus resource strained), then use xargs (and take the relevant precautions depending your input/output).

Full disclosure: I have a vested interest in GNU Parallel.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Note that Perl is automatically included in all versions of macOS to date - so Perl is not an onerous requirement on Macs.
Thanks very much. That's a helpful answer. I will look through the differences page. I have a question about one point though, how is 'xargs is everywhere'?
@KleberNoel It is part of the POSIX standard.
I would add that the man page for xargs could be read in several minutes, whereas parallel would require several hours.
@PoundHash A gray version is here gnu.org/software/parallel/parallel_cheat_bw.pdf
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.