3

I would like to have a script that would prepend each line of a stdin with information how long it took to generate it. Basically for input:

foo bar baz

I would like to have

0 foo 10 bar 5 baz

Where 10 is 10 seconds passed between printing foo and printing bar, similar for 5, it took 5 seconds after printing bar to print baz.

I know there is a utility ts that shows timestamps and I know about https://github.com/paypal/gnomon, but I would prefer not to use javascript to do that.

Is there a standard tool for that or should I use awk and do processing?

Improve this question
1
  • The time command does this but I'm not quite sure how to extract it's output. Commented Sep 8, 2017 at 19:01

5 Answers 5

Reset to default
7

Let's suppose that the script generating the output is called generate. Then, to display the number of seconds it takes for it to generate each line:

$ generate | ( t0=$(date +%s); while read -r line; do t1=$(date +%s); echo " $((t1-t0)) $line"; t0=$t1; done ) 2 foo 0 foo 5 foo 3 foo

The same commands spread out over multiple lines looks like:

generate | ( t0=$(date +%s) while read -r line do t1=$(date +%s) echo " $((t1-t0)) $line" t0=$t1 done )

Alternatively, for convenience, we can define a shell function that contains this code:

timer() { t0=$(date +%s); while read -r line; do t1=$(date +%s); echo " $((t1-t0)) $line"; t0=$t1; done; }

We can use this function as follows:

$ generate | timer 0 foo 2 foo 4 foo 3 foo

How it works

  • t0=$(date +%s)

    This captures the current time at the start of the script in seconds-since-epoch.

  • while read -r line; do

    This starts a loop which reads from standard input

  • t1=$(date +%s)

    This captures the time in seconds-since-epoch at which the current line was captured.

  • echo " $((t1-t0)) $line"

    This prints out the time in seconds that it took for the current line.

  • t0=$t1

    This updates t0 for the next line.

  • done

    This signals the end of the while loop.

Improve this answer
5 
trickle () { awk 'BEGIN { t = systime(); OFS="\t" } { print systime() - t, $0 } { t = systime() }' }

This little shell function simply wraps an awk script (compatible with both GNU awk, and mawk -Wi, but not with BSD awk due to it missing systime()). It will output each line of input prefixed with the number of whole seconds since the last line of output. The first line will be prefixed with a zero.

Testing:

$ { echo hello; sleep 2; echo world } | trickle 0 hello 2 world

The awk script could obviously be run stand-alone as well:

$ some_command | awk 'BEGIN{OFS="\t";t=systime()}{print systime()-t,$0;t=systime()}

E.g.

$ { echo hello; sleep 2; echo world; } | awk 'BEGIN{OFS="\t";t=systime()}{print systime()-t,$0;t=systime()}' 0 hello 2 world
Improve this answer
1
  • 1
    (note that it's @ThomasDickey's mawk, not the old one on Debian or derivatives). Commented Sep 8, 2017 at 22:24
4

Well, moreutils ts seems to be a perfect tool for that:

for seconds since the start:

$ seq 5 | pv -qL 1 | ts -s %s 2 1 4 2 6 3 8 4 10 5

Or with more precision:

$ seq 5 | pv -qL 1 | ts -s %.s 1.949228 1 3.933483 2 5.917923 3 7.902231 4 9.886618 5

Or with zsh or ksh93 ($SECONDS is integer by default but you can make it float with typeset -F; pdksh and bash have $SECONDS as well, but integer only):

$ typeset -F SECONDS=0; seq 5 | pv -qL 1 | while IFS= read -r line; do printf '%.3f %s\n' "$SECONDS" "$line"; done 1.987 1 3.972 2 5.956 3 7.941 4 9.925 5

for seconds between lines

$ seq 5 | pv -qL 1 | ts -i %.s 1.947814 1 1.984442 2 1.984456 3 1.984151 4 1.984195 5

zsh/ksh93:

$ typeset -F SECONDS; seq 5 | pv -qL 1 | while SECONDS=0; IFS= read -r line; do printf '%.6f %s\n' "$SECONDS" "$line"; done 1.985298 1 1.984322 2 1.984212 3 1.984240 4 1.984441 5
Improve this answer
4
  • Where's ts from? GNU moreutils? Commented Sep 8, 2017 at 22:12
  • @Kusalananda, yes, moreutils (not GNU's though). Note that the OP mentions it in his question. Commented Sep 8, 2017 at 22:14
  • 2
    ts -i is what I was looking for Commented Sep 9, 2017 at 6:13
  • On centos 7 moreutils package, ts -i "%.s" provides relative times, with sub-second values. Commented Sep 10, 2017 at 12:54
2

You can use time (note: a separate binary, not the shell built-in) and combine it with a while loop. To get the execution time for each command:

while read -r line; do /usr/bin/time -f "%e %C" $line 2>&1 done

Example output:

0.1 sleep 0.1 0.2 sleep 0.2 1.0 sleep 1

In case you want the total execution time from the beginning, you need to sum the execution time for each step. Precision for %e format is 1/100th of second, so you can't use shell integer arithmetic directly (depending on the shell you using).

total=0.0 while read -r line; do t=$(/usr/bin/time -f "%e" $line 2>&1) total=$(awk "BEGIN{print $total+$t}") echo "$total $line" done

Example output:

0.1 sleep 0.1 0.3 sleep 0.2 1.3 sleep 1
Improve this answer
2
  • It doesn't look correct, I want o have information how long it took to generate the line, not a time from the beginning. In your case it should be 0.2 sleep 0.2 0.2 sleep 0.2 1.0 sleep 1 Commented Sep 9, 2017 at 6:12
  • I've edited the answer, the first example works as you described. Commented Sep 9, 2017 at 9:32
1

First, create a simple function leveraging the bash builtin SECONDS. Notice we initialize it to zero on each invocation.

_time() { SECONDS=0; out="$($@)"; echo "[${SECONDS}] ${out:-no output}"; }

Next, use this _time function to run your command(s) from a list of commands could be a file of them, or whatever.

while read cmd; do _time $cmd; done < <(echo -e "sleep 5\nsleep 2\nsleep 1")

Here, sleep has no output, but your commands likely will. This is simply to show the use of SECONDS as it pertains to this problem. This example will output the default output of no output:

[5] no output [2] no output [1] no output
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.