0

I am using the following code to download streaming financial data

curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output '[.time, .bids[0].price, .asks[0].price] | @csv'

which is successfully streaming in the terminal thus

"2020-05-05T10:02:37.060299264Z","1.08472","1.08481"

However, what I would really like is to further process this, using sed, to get

2020,05,05,10,02,37.060299264,1.08472,1.08481

and then append this to a file, let's call it "output," but when I try piping to sed I no longer see the streaming data although I'm sure my sed syntax is correct as I've checked it against a static test file.

So my question is: how do I further pipe to sed and then append to the file "output?"

Improve this question
8
  • 3
    Its probably just being buffered. Commented May 5, 2020 at 10:45
  • @muru How would I check if this is the case? Commented May 5, 2020 at 11:25
  • 1
    Use stdbuf, unbuffer or similar (unix.stackexchange.com/a/25378/70524) and see if you get output Commented May 5, 2020 at 11:27
  • 1
    @babelproofreader: jq itself has an option to unbuffer with --unbuffered flag Commented May 5, 2020 at 11:35
  • A combination of stdbuf, --unbuffered flag in jq and -u flag in sed accomplishes what I want. Thanks to muru and Inian. Commented May 5, 2020 at 12:53

1 Answer 1

Reset to default
1

In response to Glenn Jackman's comment requesting that I post an answer, the syntax that I find works for me is:

stdbuf -oL -eL curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output --unbuffered '[.time, .bids[0].price, .asks[0].price] | @csv' | sed -u 's/["Z]//g' | sed -u 's/[-T:]/,/g' >> ~/path/to/append/to/output

which appends to the file in the desired format described in my original question. I can also look at the "output" file from time to time and see that the streaming data is continually being appended.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.