1

I built-up a kludgey script on the CLI that I can't successfully move into a text file. I know it's poorly-written, but I'm interested in resolving the problem I'm having, not re-writing it. I want to understand why my poorly-written code behaves differently when executed on the command line compared to when it's saved as a file.

I think it's related to something getting interpolated prematurely in the script, because if I substitute the one-liner version into the same file (without formatting) I get identical output. Note the debug line below and how it doesn't print.

CONTENTS OF SCRIPT

 #!/bin/bash totalNodes=0 #initialize sum echo -e "ID\t\t\tNODES"; #title line for n2 in $( #n2 will get populated with the number of nodes (to be added to totalNodes) for n in $( #n is an intermediate string that munges the Resource_list.nodes line (kludgy) for i in $( qstat|grep " ef "|cut -f 1 -d ' ') #i gets the ef queue lines do echo "debug $i" j=$(echo $i|sed 's/(\d+)\..+/\$1/'); #j is the actual ID qstat -f | grep -A43 $j; #extracts the full output from qstat for this job ID done|grep Resource_List.nodes) #closes definition of n over loop do echo ${n};done|grep ppn) #closes definition of n2 over loop do echo "$j ${n2:0:1}" #output line totalNodes=$(($totalNodes+${n2:0:1})) #counting nodes done echo "$totalNodes nodes of 16 running in EF queue"

EXPECTED OUTPUT (what I get at the command line):

 ID NODES 2378512.yaddayadday-adm 4 2378512.yaddayadday-adm 4 2378512.yaddayadday-adm 4 2378512.yaddayadday-adm 2 14 nodes of 16 running in EF queue

CURRENT OUTPUT from the script

 ID NODES 4 4 4 2 14 nodes of 16 running in EF queue

So I'm confused how I can get the right total (meaning $n2 is getting defined correctly) but I can't even print $i on the debug-with-print line (line 8.)

For reference, here's the one-liner. Like I said, this gives the "EXPECTED OUTPUT" shown above, when executed on the command line, but the same output as the above codeblock "CONTENTS OF SCRIPT" when I save it as a file with no additional formatting.

 totalNodes=0;echo -e "ID\t\t\tNODES";for n2 in $(for n in $(for i in $(qstat|grep " ef "|cut -f 1 -d ' ');do j=$(echo $i|sed 's/(\d+)\..+/$1/');qstat -f | grep -A43 $j;done|grep Resource_List.nodes);do echo ${n};done|grep ppn);do echo "$j ${n2:0:1}";totalNodes=$(($totalNodes+${n2:0:1})); done;echo "$totalNodes nodes of 16 running in EF queue"
Improve this question
6
  • I would start by replacing for n{2} in $(... with a |while read LINE; do case "$LINE in" ... you can do the same with the output of qstat and operate on streams instead of ballooning the memory with unneeded large variables that is nearly as painful to read as recursion but with no benefit ... the read command can replace cut as well by using read dummy0 dummy1 needed_var dummy2; Commented Sep 26, 2013 at 16:45
  • Your variable j is used iteratively internally before the outer loop parses n2. How would it fit to be placed in echo "$j ${n2:0:1}"? Commented Sep 26, 2013 at 17:31
  • 1
    To answer your question: it's caused by subshells, as detailed in konsolebox his answer. bash is not well suited for stuff like this - a scripting language like python would be a better fit. To address your problem, a quick and dirty option to pass data back from a child to parent is to write the output to a file. e.g echo "debug $i" >> tmp.out or "your_oneliner" > tmp.out; cat tmp.out This can also be used for intermediate results, to iterate over with read while. Commented Sep 30, 2013 at 18:35
  • This does not answer my question. My question is, in terms of subshells, why do the subshells cause problems only when the script is called from a file? Commented Oct 1, 2013 at 0:01
  • 1
    Perhaps j is already set to 2378512.yaddayadday-adm in your shell before you execute the one-liner? Do an echo $j to find out. You could add some sample input to your question. Commented Oct 1, 2013 at 0:21

2 Answers 2

Reset to default
4
+50

Command substitution method $() runs commands in it on a subshell so if you define a variable inside it the value of that variable would be lost outside.

j=1 : "$(j=2)" echo "$j" # => 1
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

This seems like a good clue but I think I don't understand the full significance of 'subshell.' If I insert an "echo $variable-in-scope" within a $(yadda), it gives output on the command line but not in the script. Why does the subshell's behavior (or scoping) seem different in the two cases?
@Ryan Subshells simply run as a different process. If you do echo "$(pstree)" you'd notice that pstree is running under another instance of bash. Processes normally can't share their own variables along each other.
The recommended solution actually would be to store your output to an array for every step but perhaps a pseudocode would be helpful for that since I don't exactly know what your code wants i.e. I can't even follow why you access $j on the outside. Even if it's not processed inside a subshell it would still be odd since $j is an iterating variable on the inner part.
I have to emphasize; this works in the one-liner form and not when it's saved in a file and executed as a script. Are you explaining why that's happening, and I'm just not absorbing it, or are you explaining why it shouldn't work (and therefore missing my observation that it does work on the CLI)?
@Ryan I think I missed your point. The thing is you're actually reading the output of the echo message as well as part of the iteration since it's naturally under command substitution. If you like you could redirect your output to stderr instead e.g. echo "debug $i" >&2. Yet again it's still recommended to not recurse your commands in nested command substitutions like that. The redirection probably wouldn't be necessary if you're storing output by arrays instead, and many other unexpected troubles could be avoided as well.
0

several bugs :

1- qstat|grep " ef "|cut -f 1 -d ' ' is returning an empty list

2- qstat -f | grep -A43 $j is returning en empty list

before running the loop make sure #1 and #2 generates a valid list

since qstat|grep " ef "|cut -f 1 -d ' ' is constant , consider moving it outside the loops

$x1 = $(qstat|grep " ef "|cut -f 1 -d ' ')

use $x1 instead of above

Improve this answer

3 Comments

I don't see how this explains why the bugs appear only when I'm calling the script from a file. Do you see how this is happening? Also, 'qstat -f | grep -A43 $j' can't be returning an empty list because $n2 is accumulating the correct total.
its possible qstat is running from a different path (or other scripts ). also some variable might be set in session and not in script. add a set-x at the beginning of script and put some echo in between forloops
I don't think you've identified any bugs. qstat|grep " ef "|cut -f 1 -d ' ' is not a constant. Also your definition of x1 is not valid Bash.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.