6

I want to invoke a command for every line of the standard input, much like xargs, but the line should be passed as standard input, not as a command-line argument:

cat some-file.txt | <magic> | wc -c 
  • this should print the number of characters in each line of the file.

What are my options?

5
  • It's unfortunate that the actual issue only is hinted at at the end of the question. It would be more interesting to work with the code that you were actually trying to run, together with an example of the input file. The given code does not say much about where the problem lies. It might be a buffering issue, which would be easily fixed. Commented Aug 24, 2022 at 22:30
  • @Kusalananda I disagree, I think linewise execution as a general technique is much more interesting (and something I was able to use for a number of other problems once I learned about it) than some specific application of jq (which has an --unbuffered flag these days but didn't back then). Commented Aug 25, 2022 at 19:26
  • In that case, I'll remove the mentioning of jq at the end of the question, as it implies that the data is in fact not line-based at all but structured. Any answer using line-based tools would therefore be inadequate in various ways. Also note that although the question is almost ten years old, answers may be added that solves it using up-to-date tools. Commented Aug 25, 2022 at 19:29
  • Would not awk '{ print length }' as the second and final stage of the pipeline solve the particular problem here? Or were you stuck having to call wc -c specifically? The question is whether the issue is philosophical (and the commands shown are examples) or actual (and you actually want to count the length of each line in a particular way). Commented Aug 25, 2022 at 19:36
  • FWIW, single-line JSON is a common log format. I'm pretty sure wc was just a random example. Commented Aug 26, 2022 at 20:25

4 Answers 4

8

How about a plain loop

while IFS= read -r line ; do printf "%s" "$line" | wc -c done < some-file.txt 
1
  • For my use case of processing json log format, the trick in the above solution was to use read -r line, not just read line. Commented Jun 25 at 11:05
4

A while-read loop is the most clear. If you want to use xargs to do something for each line, you may end up with a monstrosity like this:

printf "%s\n" "foo bar" one " 1 2 3" | xargs -d '\n' -n 1 -I LINE bash -c 'wc -c <<< "LINE"' 
8 4 7 

Pretty expensive since you have to spawn a bash process for each line.

3
cat file.txt | while IFS= read -r i; do echo -n "$i" | wc -c; done ## or (better): while IFS= read -r i; do echo -n "$i" | wc -c; done < file.txt 

However, this will just print the number of characters on a line, one line at a time. If you want something more readable, you might want one of the following:

## Prints the contents of each line underneath the character count: while IFS= read -r i; do echo -n "$i" | wc -c; echo "$i"; done < file.txt ## Prints the line number alongside the character count: n=0; while IFS= read -r i; do n=$((n+1)); echo -n "line number $n : "; echo -n "$i" | wc -c; done < file.txt 

For greater portability you could use printf '%s' "$i" instead of all the echo -ns

1

Z shell

Storing lines into a variable can be avoided using option -e

setopt pipefail cat some-file.txt | while read -re | wc -c do done 

-r reads \s as they are

-e echoes the input instead of assigning it to any variable

pipefail makes read -re | wc -c exit with non-zero status code as soon as so does read -re, which it will at the end of the file

Unfortunately, this option isn't available in Bash

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.