Limiting the number of subshells spawned

Question

I'm trying to limit the amount of subshells that are spawned in a script that I'm using to sniff our internal network to audit Linux servers in our network. The script works as intended, but due to the way I'm nesting the for loop, it spawns 255 Sub Shells for each network, so therefore it kills the CPU due to the fact that there are over 1000 processes spawned off. I need to be able to limit the amount of processes, and since variables lose their value when in a Sub Shell, I can't figure out a way to make this work. Again, the script works, it just spawns a ton a processes - I need to limit it to, say 10 Processes max:

#!/bin/bash FILE=/root/ats_net_final for network in `cat $FILE`;do for ip in $network.{1..255};do ( SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $ip sysName.0 2>/dev/null | awk '{ print $NF }'` SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $ip sysDescr.0 2>/dev/null | grep -o Linux` if [ $? -eq 0 ];then echo "$SYSNAME" exit 0; else echo "Processed $ip" exit 0 fi ) & done done

This solution I found that works, but not in my case, because no matter what, it still will spawn the processes before the logic of limiting the processes. I think that maybe I've just been looking at the code too long and it's just a simple logic issue and I'm placing things in the wrong area, or in the wrong order.

Answer Accepted:

I've accepted the answer from huitseeker. He was able to provide me with direction of how the logic works, allowing me to get it to work. Final script:

#!/bin/bash FILE=/root/ats_net_final for network in `cat $FILE`;do #for ip in $network.{1..255};do for ip in {1..255};do ( ip=$network.$ip SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $ip sysName.0 2>/dev/null | awk '{ print $NF }'` SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $ip sysDescr.0 2>/dev/null | grep -o Linux` if [ $? -eq 0 ];then echo "$SYSNAME" exit 0; else echo "Processed $ip" exit 0 fi ) & if (( $ip % 10 == 0 )); then wait; fi done wait done

Instead of `cat $FILE` use `<$FILE`. One process start less. Also do not make the loop inlut .255. It can be a broadcast address (in case of C type network) and not a machine address. — TrueY
– TrueY, Commented Jun 21, 2013 at 14:06
@TrueY: While that is true, I'm doing it for processing. In our network, we use /24's and it's true that the .0 is the network and .255 is the broadcast, and .1 is our gateway (for all our networks), I'm still including it. Under normal circumstances, I totally agree. — drewrockshard
– drewrockshard, Commented Jun 21, 2013 at 14:19

Francois G · Accepted Answer · 2013-06-21 14:08:03Z

An easy way to limit the number of concurrent subshells to 10 is:

for ip in $(seq 1 255);do (<whatever you did with $ip in the subshell here, do with $network.$ip instead >) & if (( $ip % 10 == 0 )); then wait; fi done wait

With the last wait being useful not to let the subshells of the last round of the inner loop overlap with those created in first round of the next outer run.

Awesome. This was the direction I needed. I'll date my question with my answer.
Don't use seq in bash. Use either for ip in {1..255}; do or for ((ip=1;i<256;++i)); do.

TrueY · Accepted Answer · 2013-07-15 20:07:24Z

I think I have found a better solution. I implemented using make:

#!/usr/bin/make -f IPFILE := ipfile IPS:=$(foreach ip,$(shell cat $(IPFILE)),$(foreach sub,$(shell seq 1 1 254),$(ip).$(sub))) all: $(IPS) $(IPS): SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $@ sysName.0 2>/dev/null | awk '{ print $$NF }'`; \ SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $@ sysDescr.0 2>/dev/null | grep -o Linux`; \ if [ $$? -eq 0 ]; then \ echo "$$SYSNAME"; \ else \ echo "Processed $@"; \ fi

The first part of the IPs (e.g. 192.168.1) should be placed to ipfile. Then it generates all the IP addresses into the variable IPS (Like 192.168.1.1 ... 192.168.1.254 ...).

These lines can be copied to e.g. test.mak and add execute rights to the file. If one run it as ./test.mak then it will process the IPs in IPS one by one. But if it is run as ./test.mak -j 10 then it will process 10 IPs at once. Also it can be run as ./test.mak -j 10 -l 0.5. It will run maximum 10 processes or until the system load reaches 0.5.

Collectives™ on Stack Overflow

Limiting the number of subshells spawned

Answer Accepted:

2 Answers 2

3 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Answer Accepted:

2 Answers 2

3 Comments

Comments

Linked

Related