4

I've got a command that outputs a few different lines constantly (a stream), I'd like a live summary of the number of times each line has occurred (with a few seconds latency being acceptable).

for example if my command outputs the following:

apple apple apple apple banana orange banana

I'd like something like:

4 apple 2 banana 1 orange

and for the output to refresh every few seconds.

How might I achieve this? (rereading an entire log file will take too long, it has to be the output of a live pipe)

Improve this question

3 Answers 3

Reset to default
3

You could use awk:

... | awk '{seen[$0]++} !(NR % 10) {print "======"; for (i in seen) print seen[i], i}'

This will keep a count of duplicates (seen[$0]++), and print them every ten input lines (!(NR % 10)). For example:

% % while sleep 1; do echo $((RANDOM % 10)); done | awk '{seen[$0]++} !(NR % 10) {print "======"; for (i in seen) print seen[i], i}' ====== 1 0 1 1 3 3 1 6 2 7 2 9 ====== 3 0 3 1 1 2 3 3 1 4 2 5 2 6 3 7 2 9 ====== 3 0 3 1 2 2 3 3 2 4 5 5 4 6 3 7 2 8 3 9 ...
Improve this answer
0
2

Could be done with a short perl script like:

#! /usr/bin/perl system qw(tput sc); # save cursor $rc = `tput rc; tput ed`; # restore cursor and erase down sub report { print $rc; print "$_: $c{$_}\n" for sort { ($c{$b} <=> $c{$a}) || ($a cmp $b) } keys %c; STDOUT->flush; alarm 1; } $SIG{ALRM} = \&report; alarm 1; while (<>) { chomp; $c{$_}++; } report;
Improve this answer
1
  • Thanks for this, I accepted the awk answer because it's just a bit more readable than perl, but your answer is great too. Commented May 13, 2020 at 14:43
1

watch -n <seconds> "sort <file> | uniq -c"

Should be about what you want. Runs sort | uniq every <seconds>.

Improve this answer
2
  • 2
    Note the (rereading an entire log file will take too long, it has to be the output of a live pipe) in the OP's question. Commented May 12, 2020 at 11:37
  • hmm, that makes matters more complex. you will need to read the file in chunks and keep a rolling count of entries to be updated. Commented May 13, 2020 at 13:25

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.