Bash script to complete 21 process by running 4 process parallel

Since you are new to scripting, take a look at xargs -P. It is a command worth becoming familiar with even if you are not running processes in parallel and it be somewhat more likely to be pre-installed on your system than parallel mentioned in the answer pointed to by Mat.

There is pbzip2: parallel bzip2 compression for this kind of purpose. Also, 7z has multi-threading options, but I haven't tried them

I use this kind of setup usually:

tar -cp somedir_tobackup/ | pbzip2 | mbuffer -m 180M | netcat <host> <port> 

Obviously, you can use cpio instead, drop the mbuffer, save to a local archive file first, use rsync with it etc. but the idea is there.

Using GNU Parallel you can do:

cat my.txt | parallel /home/cms.sh 

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (http://www.gnu.org/software/parallel/parallel_tutorial.html). You command line with love you for it.

You need a second loop just after your "do" that sleeps repeatedly for 1 second until there are not 4 processes running, and it then breaks out of the loop and starts a new process before reading another line from your control file. Something like this:

#!/bin/bash while read line do while : do run_count=`ps eax | grep ping | grep -v grep | wc -l` [ 4 -gt ${run_count} ] && break; sleep 1 done ./somejob $line & done < my.txt 

I reckon that you got quad cores thus 4 processes. Wrong solution. Instead, count number of cores (this way it can be extended to dual cores, quad cores, 6 cores, etc, without manually changing process variable). This example is the best I can in running bash parallel. It is made by dividing the task in how many cores you got, clone script.cfg in how many cores you got, runs them in background. In this way, each core will compress only a part of those folders.

Create a folder let's say bash. Inside, create other 2 folders (backup and directory). Put those 21 folders inside directory. Create 2 files (make .sh file executable).

start.sh

#!/bin/bash date=$(date '+%d_%B_%Y') mkdir backup/$date cores=$(ls /proc/acpi/processor/ | wc -l) cd directory ls > ../list.result cd .. file_lines=$(cat list.result | wc -l) let "split_lines=$file_lines/$cores" let "split_lines=$split_lines+1" split -d --lines=$split_lines list.result CPU for script in `seq 1 $cores` do let sufix=script-1 cat script.cfg > "CPU0$sufix.script" chmod +x CPU0$sufix.script sed -i 's/filenumber/CPU0'$sufix'/g' CPU0$sufix.script ./CPU0$sufix.script & done exit 

and script.cfg

#!/bin/bash date=$(date '+%d_%B_%Y') lenght=$(cat filenumber |wc -l) counter=1 for x in `seq 1 $lenght` do value="directory/"$(sed "${x}!d" filenumber) tar -zcvf $value.tar.gz $value mv directory/*.tar.gz backup/$date done rm *.script CPU* list.result exit 

All files in directory folder will be compressed using all cores in CPU and moved to backup folder. This technique runs all cores at 100% so take care of CPU temperature.