398

How does one properly iterate over lines in bash either in a variable, or from the output of a command?

Simply setting the IFS variable to a new line works for the output of a command but not when processing a variable that contains new lines.

For example

#!/bin/bash list="One\ntwo\nthree\nfour" #Print the list with echo echo -e "echo: \n$list" #Set the field separator to new line IFS=$'\n' #Try to iterate over each line echo "For loop:" for item in $list do echo "Item: $item" done #Output the variable to a file echo -e $list > list.txt #Try to iterate over each line from the cat command echo "For loop over command output:" for item in `cat list.txt` do echo "Item: $item" done

This gives the output:

echo: One two three four For loop: Item: One\ntwo\nthree\nfour For loop over command output: Item: One Item: two Item: three Item: four

As you can see, echoing the variable or iterating over the cat command prints each of the lines one by one correctly. However, the first for loop prints all the items on a single line. Any ideas?

Improve this question
1
  • Just a comment for all answers: I had to do $(echo "$line" | sed -e 's/^[[:space:]]*//') in order to trim the newline character. Commented Sep 4, 2019 at 10:48

7 Answers 7

Reset to default
479

With bash, if you want to embed newlines in a string, enclose the string with $'':

$ list="One\ntwo\nthree\nfour" $ echo "$list" One\ntwo\nthree\nfour $ list=$'One\ntwo\nthree\nfour' $ echo "$list" One two three four

And if you have such a string already in a variable, you can read it line-by-line with:

while IFS= read -r line; do echo "... $line ..." done <<< "$list"

@wheeler makes a good point about <<< adding a trailing newline.

Suppose the variable ends with a newline

list=$'One\ntwo\nthree\nfour\n'

Then the while loop outputs

while IFS= read -r line; do echo "... $line ..." done <<< "$list" 
... One ... ... two ... ... three ... ... four ... ... ...

To get around that, use a redirection from a process substitution instead of a here-string

while IFS= read -r line; do echo "... $line ..." done < <(printf '%s' "$list") 
... One ... ... two ... ... three ... ... four ...

But now this "fails" for strings without a trailing newline

list2=$'foo\nbar\nbaz' while IFS= read -r line; do echo "... $line ..." done < <(printf '%s' "$list2") 
... foo ... ... bar ...

The read documentation says

The exit status is zero, unless end-of-file is encountered

and because the input does not end with a newline, EOF is encountered before read can get a whole line. read exits non-zero and the while loop completes.

The characters are consumed into the variable though.

So, the absolutely proper way to loop over the lines of a string is:

while IFS= read -r line || [[ -n $line ]]; do echo "... $line ..." done < <(printf '%s' "$list2")

This outputs the expected for both $list and $list2

Improve this answer
18
  • 43
    Just a note that the done <<< "$list" is crucial Commented Sep 26, 2012 at 7:53
  • 32
    The reason done <<< "$list" is crucial is because that will pass "$list" as the input to read Commented Mar 11, 2015 at 21:00
  • 31
    I need to stress this: DOUBLE-QUOTING $list is very crucial. Commented Jun 10, 2015 at 7:44
  • 12
    echo "$list" | while ... could appear more clear than ... done <<< "$line" Commented May 14, 2018 at 10:07
  • 23
    There are downsides to that approach as the while loop will run in a subshell: any variables assigned in the loop will not persist. Commented May 14, 2018 at 14:13
151

You can use while + read:

some_command | while read line ; do echo "=== $line ===" done

Btw. the -e option to echo is non-standard. Use printf instead, if you want portability.

Improve this answer
4
  • 26
    Note that if you use this syntax, variables assigned inside the loop won't stick after the loop. Oddly enough, the <<< version suggested by glenn jackman does work with variable assignment. Commented Oct 3, 2013 at 4:51
  • 12
    @Sparhawk Yes, that's because the pipe starts a subshell executing the while part. The <<< version does not (in new bash versions, at least). Commented Oct 21, 2013 at 13:53
  • This fails if the line contains spaces. Commented Jun 26 at 20:23
  • @tobiasBora That's a reason for always quoting the variable when using it, like this: "$line". Commented Aug 27 at 1:40
63 
#!/bin/sh items=" one two three four hello world this should work just fine " IFS=' ' count=0 for item in $items do count=$((count+1)) echo $count $item done
Improve this answer
4
  • 2
    This outputs all items, not lines. Commented Aug 13, 2015 at 15:05
  • 12
    @TomaszPosłuszny No, this outputs 3 (count is 3) lines on my machine. Note the setting of the IFS. You could also use IFS=$'\n' for the same effect. Commented May 10, 2016 at 12:33
  • This and @jmiserez proposal both work beautifully, thank you! Commented Jul 16, 2020 at 14:34
  • This is the best option if your text already has a trailing newline. Commented Apr 12, 2023 at 7:22
25

Here's a funny way of doing your for loop:

for item in ${list//\\n/ } do echo "Item: $item" done

A little more sensible/readable would be:

cr=' ' for item in ${list//\\n/$cr} do echo "Item: $item" done

But that's all too complex, you only need a space in there:

for item in ${list//\\n/ } do echo "Item: $item" done

You $line variable doesn't contain newlines. It contains instances of \ followed by n. You can see that clearly with:

$ cat t.sh #! /bin/bash list="One\ntwo\nthree\nfour" echo $list | hexdump -C $ ./t.sh 00000000 4f 6e 65 5c 6e 74 77 6f 5c 6e 74 68 72 65 65 5c |One\ntwo\nthree\| 00000010 6e 66 6f 75 72 0a |nfour.| 00000016

The substitution is replacing those with spaces, which is enough for it to work in for loops:

$ cat t.sh #! /bin/bash list="One\ntwo\nthree\nfour" echo ${list//\\n/ } | hexdump -C $ ./t.sh 00000000 4f 6e 65 20 74 77 6f 20 74 68 72 65 65 20 66 6f |One two three fo| 00000010 75 72 0a |ur.| 00000013

Demo:

$ cat t.sh #! /bin/bash list="One\ntwo\nthree\nfour" echo ${list//\\n/ } | hexdump -C for item in ${list//\\n/ } ; do echo $item done $ ./t.sh 00000000 4f 6e 65 20 74 77 6f 20 74 68 72 65 65 20 66 6f |One two three fo| 00000010 75 72 0a |ur.| 00000013 One two three four
Improve this answer
8
  • Interesting, can you explain what is going on here? It looks like you are replacing \n with a new line... What is the difference between the original string and the new one? Commented May 16, 2011 at 13:15
  • @Alex: updated my answer - with a simpler version too :-) Commented May 16, 2011 at 13:24
  • 1
    I tried this out and it doesn't appear to work if you have spaces in your input. In the example above we have "one\ntwo\nthree" and this works but it fails if we have "entry one\nentry two\nentry three" as it also adds a new line for the space too. Commented Nov 8, 2012 at 18:15
  • 4
    Just a note that instead of defining cr you can use $'\n'. Commented Dec 21, 2013 at 19:12
  • 1
    This is the most correct answer. The other answers will get into the loop even if variable is not set or empty. Commented Apr 23, 2020 at 10:45
15

You can also first convert the variable into an array, then iterate over this.

lines="abc def ghi" declare -a theArray while read -r line do theArray+=("$line") done <<< "$lines" for line in "${theArray[@]}" do echo "$line" #Do something complex here that would break your read loop done

This is only usefull if you do not want to mess with the IFS and also have issues with the read command, as it can happen, if you call another script inside the loop that that script can empty your read buffer before returning, as it happened to me.

Improve this answer
2
  • thank you very much for this! Your mentioned problem was exactly the issue I was facing with the accepted answer... your approach works perfectly! Commented Aug 23, 2024 at 13:25
  • Instead of read loop you can readarray -t theArray <<<$lines (or synonym mapfile) Commented Oct 26, 2024 at 2:36
0 
list="One\ntwo\nthree\nfour" for line in $(echo ${list} | xargs -L1 echo) do echo ${line} done
Improve this answer
1
  • 2
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. Commented Aug 1, 2024 at 16:18
0

Using a string var with carefully crafted newline-delimited substrings:

tmp1=' M file*4.bin'$'\n''AM file5\0345.txt'$'\n''M file6a$\t.rtf'

Note the concatenation via $'\n', in order to actually insert newline control chars, instead of just '\n' substrings and single-quoted substrings, to preserve special chars/avoid escaping.

The code below should work with arrays as well:

tmp2=(' M file*1.bin' 'AM file2\0345.txt' 'M file3a$\t.rtf')

Iterating the string in a for loop now can be achieved via unquoted expansion to a single string ([*]) and IFS restriction to newline (the printf variant can give more control, but invokes a subshell:

_IFS="$IFS"; IFS=$'\n' #for e in $(printf '%s\n' "${tmp1[*]}"); do for e in ${tmp1[*]}; do printf 'ELEM: [%s]\n' "$e" done IFS="$_IFS"

Note the missing double quotes around ${tmp1[*]}, in order to expand and split at the newline control chars, instead of expanding to just a single string here.

The surrounding IFS modification and reset should prevent accidential splitting at spaces or tabs, but might also affect commands inside the loop.

Also consider the hint on requiring to disable globbing from 'dave_thompson_085' below, in case the input strings may contain escaped or special chars (not single-quoted).

A while loop over a quoted expanded Here String may be the better choice, depending on the input. Narrowing the IFS to only newlines for just the loop is possible, but shouldn't be necessary:

while IFS=$'\n' read -r; do printf 'ELEM: [%s]\n' "$REPLY" done <<<"$tmp1"

When we have an array, it can be simply iterated with a for loop, from a quoted array Parameter Expansion ([@]) - note the double-quotes here:

for e in "${tmp2[@]}"; do printf 'ELEM: [%s]\n' "$e" done

or with while loop:

_IFS="$IFS"; IFS=$'\n' while read -r; do printf 'ELEM: [%s]\n' "$REPLY" done <<<"${tmp2[*]}" # done < <(printf '%s\n' "${tmp2[*]}") IFS="$_IFS" # done <<<"${tmp2[@]@E}"

The usage of @E as operator for the array Parameter Expansion ([@]), is required to expand the newline control chars, but this will also expand other special chars end escape sequences, which may be undesired.

To avoid this, restricting the IFS to newline (and restoring it) and expanding the array to a single string ([*]) is used instead.

Alternatively, the array splitting can be done via Command Substitution and printf.

This code should work for newline-delimited strings, as well as for arrays.

Improve this answer
2
  • For the first method (for e in $tobesplit) you also need to turn off (and afterwards turn back on) globbing, or a line consisting of * will have very wrong results, and so will many other values if failglob or nullglob is (nondefaulty) set Commented Oct 26, 2024 at 2:43
  • Thanks for pointing this out, I added a hint. Commented Oct 29, 2024 at 14:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.