6

I know how to run a command in all subdirectories that match a pattern like this (lets say I want to run gh repo sync):

for d in ./*/ ; do (cd "$d" && gh repo sync) ; done

And runs the commands in a subshell in each directory serially.

What I want is to run all the commands in parallel but wait in the main script for them to all finish.

I'm using Ubuntu 22.04. Is that possible?

Improve this question

2 Answers 2

Reset to default
12

GNU Parallel does this excellently. I've been using this structure daily for a long time:

parallel --wd '{}' 'gh repo sync' ::: ./*/.git/..

Similar test on my own system:

$ parallel --wd '{}' 'pwd' ::: ~/my\ projects/*/.git/.. /home/username/my projects/acard /home/username/my projects/acp01 [125 more]
Improve this answer
8
  • This looks awesome! Can I somehow see the output of the commands in the console? Commented Apr 24, 2023 at 16:48
  • A side note about this vs the other answer: If the commands rely on shared resources and you need to limit how many run at one time, parallel has -j to set a maximum number of concurrent jobs. Much more difficult to do that with the simple loop in the other answer, though that one doesn't require an additional dependency. Commented Apr 24, 2023 at 18:27
  • 3
    @traveh, of course parallel preserves command output. Moreover, I think you'll be able to drop cd using --wd option. Commented Apr 24, 2023 at 20:05
  • 1
    You have my vote anyway, but why do you have && cd - at the end, just before the process exits? Commented Apr 25, 2023 at 7:15
  • 1
    @MarkSetchell I was writing on a phone without access to a Linux machine, so I overthunked it :) Commented Apr 25, 2023 at 8:08
12

Run each subshell command in the background and then wait for them all to complete

for d in ./*/ do ( cd "$d" && gh repo sync ) & done wait
Improve this answer
1
  • This is also a good answer, but I'm accepting the other one as it is better suited for my needs (sorry). Commented Apr 25, 2023 at 2:22

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.