Run bash commands in parallel, track results and count

Can I persuade you to use make? This has the advantage that you can tell it how many commands to run in parallel (modify the -j number)

echo -e ".PHONY: c1 c2 c3 c4\nall: c1 c2 c3 c4\nc1:\n\tsleep 2; echo c1\nc2:\n\tsleep 2; echo c2\nc3:\n\tsleep 2; echo c3\nc4:\n\tsleep 2; echo c4" | make -f - -j2 

Stick it in a Makefile and it will be much more readable

.PHONY: c1 c2 c3 c4 all: c1 c2 c3 c4 c1: sleep 2; echo c1 c2: sleep 2; echo c2 c3: sleep 2; echo c3 c4: sleep 2; echo c4 

Beware, those are not spaces at the beginning of the lines, they're a TAB, so a cut and paste won't work here.

Put an "@" infront of each command if you don't the command echoed. e.g.:

 @sleep 2; echo c1 

This would stop on the first command that failed. If you need a count of the failures you'd need to engineer that in the makefile somehow. Perhaps something like

command || echo F >> failed 

Then check the length of failed.

The problem you have is that you cannot wait for one of multiple background processes to complete. If you observe job status (using jobs) then finished background jobs are removed from the job list. You need another mechanism to determine whether a background job has finished.

The following example uses starts to background processes (sleeps). It then loops using ps to see if they are still running. If not it uses wait to gather the exit code and starts a new background process.

#!/bin/bash sleep 3 & pid1=$! sleep 6 & pid2=$! while ( true ) do running1=`ps -p $pid1 --no-headers | wc -l` if [ $running1 == 0 ] then wait $pid1 echo process 1 finished with exit code $? sleep 3 & pid1=$! else echo process 1 running fi running2=`ps -p $pid2 --no-headers | wc -l` if [ $running2 == 0 ] then wait $pid2 echo process 2 finished with exit code $? sleep 6 & pid2=$! else echo process 2 running fi sleep 1 done 

Edit: Using SIGCHLD (without polling):

#!/bin/bash set -bm trap 'ChildFinished' SIGCHLD function ChildFinished() { running1=`ps -p $pid1 --no-headers | wc -l` if [ $running1 == 0 ] then wait $pid1 echo process 1 finished with exit code $? sleep 3 & pid1=$! else echo process 1 running fi running2=`ps -p $pid2 --no-headers | wc -l` if [ $running2 == 0 ] then wait $pid2 echo process 2 finished with exit code $? sleep 6 & pid2=$! else echo process 2 running fi sleep 1 } sleep 3 & pid1=$! sleep 6 & pid2=$! sleep 1000d 

I think the following example answers some of your questions, I am looking into the rest of question

(cat list1 list2 list3 | sort | uniq > list123) & (cat list4 list5 list6 | sort | uniq > list456) & 

from:

Running parallel processes in subshells

There is another package for debian systems named xjobs.

You might want to check it out:

http://packages.debian.org/wheezy/xjobs

If you cannot install parallel for some reason this will work in plain shell or bash

# String to detect failure in subprocess FAIL_STR=failed_cmd result=$( (false || echo ${FAIL_STR}1) & (true || echo ${FAIL_STR}2) & (false || echo ${FAIL_STR}3) ) wait if [[ ${result} == *"$FAIL_STR"* ]]; then failure=`echo ${result} | grep -E -o "$FAIL_STR[^[:space:]]+"` echo The following commands failed: echo "${failure}" echo See above output of these commands for details. exit 1 fi 

Where true & false are placeholders for your commands. You can also echo $? along with the FAIL_STR to get the command status.

Yet another bash only example for your interest. Of course, prefer the use of GNU parallel, which will offer much more features out of the box.

This solution involve tmp file output creation for collecting of job status.

We use /tmp/${$}_ as temporary file prefix $$ is the actual parent process number and it is the same for all the script execution.

First, the loop for starting parallel job by batch. The batch size is set using max_parrallel_connection. try_connect_DB() is a slow bash function in the same file. Here we collect stdout + stderr 2>&1 for failure diagnostic.

nb_project=$(echo "$projects" | wc -w) i=0 parrallel_connection=0 max_parrallel_connection=10 for p in $projects do i=$((i+1)) parrallel_connection=$((parrallel_connection+1)) try_connect_DB $p "$USERNAME" "$pass" > /tmp/${$}_${p}.out 2>&1 & if [[ $parrallel_connection -ge $max_parrallel_connection ]] then echo -n " ... ($i/$nb_project)" wait parrallel_connection=0 fi done if [[ $nb_project -gt $max_parrallel_connection ]] then # final new line echo fi # wait for all remaining jobs wait 

After run all jobs is finished review all results:

SQL_connection_failed is our convention of error, outputed by try_connect_DB() you may filter job success or failure the way that most suite your need.

Here we decided to only output failed results in order to reduce the amount of output on large sized jobs. Especially if most of them, or all, passed successfully.

# displaying result that failed file_with_failure=$(grep -l SQL_connection_failed /tmp/${$}_*.out) if [[ -n $file_with_failure ]] then nb_failed=$(wc -l <<< "$file_with_failure") # we will collect DB name from our output file naming convention, for post treatment db_names="" echo "=========== failed connections : $nb_failed/$nb_project" for failure in $file_with_failure do echo "============ $failure" cat $failure db_names+=" $(basename $failure | sed -e 's/^[0-9]\+_\([^.]\+\)\.out/\1/')" done echo "$db_names" ret=1 else echo "all tests passed" ret=0 fi # temporary files cleanup, could be kept is case of error, adapt to suit your needs. rm /tmp/${$}_*.out exit $ret