5

I have the following script:

echo "$wk" | while read -r a b; do counter=$(($counter+1)) nohup sh -c 'impala-shell -d nielsen -f query.sql --var=dt=$a --var=incre=$b; echo $a,$?>>$week_status_file; if [ $? -ne 0 ]; then echo $a,$b>>$err_week_file fi' if [ $(($counter%5)) == 0 -a $counter -ge 5 ]; then echo "Check" wait fi done

The idea is to kick off the wrapped command impala-shell... with two variables a and b (coming from the variable wk). This should be done in parallel with 5 processes. The next 5 jobs should wait until the previous 5 processes to finish. My current code prints out Check, but it doesn't wait at all. However, if I change wait to sleep 10s, it does sleep.

How do I make wait work here?

Improve this question
2
  • Just glancing at the code not in detail, you're using $? improperly. [ $? -ne 0 ] checks the exit code of the echo statement, not the nohup. Assign $? to something in order to use it more than once. Commented Mar 2, 2018 at 21:08
  • @user1404316 just for your reference $? somehow refers the correct process. I tested using a failing scenario of impala-shell and $? passed 1 correctly. Commented Mar 2, 2018 at 21:25

1 Answer 1

Reset to default
2

You have to supply process identifiers (PIDs) to wait. Also from your description it does not sound like your command is blocking which ultimately would make it harder to gather the PIDs you want to wait for. E.g. if the process is forking and allows you script to continue without knowing new forked processes' PID.

But in case your nohup sh -c ... line is blocking you want to send it to background with &, save the background processes' PID and finally wait for all the PIDs you saved (e.g. in $WAIT_FOR_PIDS). Variable $! gives you the PID of the last process send to background execution.

WAIT_FOR_PIDS=""; echo "$wk" | while read -r a b; do counter=$(($counter+1)) nohup sh -c 'impala-shell -d nielsen -f query.sql --var=dt=$a --var=incre=$b; echo $a,$?>>$week_status_file; if [ $? -ne 0 ]; then echo $a,$b>>$err_week_file fi' & WAIT_FOR_PIDS="$WAIT_FOR_PIDS $!" if [ $(($counter%5)) == 0 -a $counter -ge 5 ]; then echo "Check" wait $WAIT_FOR_PIDS WAIT_FOR_PIDS="" fi done

Also you should think about race conditions while echoing in parallel into $week_status_file. See man flock for easy shell accessible file locking.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.