1

I've been dealing with a weird issue that I can't find a way to solve.

My situation is as follows.

I have an application in python called "app1", that requires a file for outputting the results of it's execution.

I have a secondary application, called "app2"; a binary, that gets the input from stdin.

I want to pipe what "app1" is generating directly into "app2" for processing, what in an ideal situation would be like this:

app1 | app2

But, as I said, there are some restrictions, like the fact that app1 requires a file to be the output. The first solution I found for "fooling" app1 into outputting to stdout is to use mkfifo and create a pipe, so I can pipe it into stdin in app2. Like this:

pipe='/tmp/output_pipe' mkfifo "$output_pipe" python app1 -o "$output_pipe" & app2 < $tmp_pipe

The problem is that eventually, during the execution, app1 will generate more output than what app2 can handle as an input, and due to the buffer size restrictions on the pipe, the pipe will fill up and everything will stop working.

Then I used this other approach:

python app1 -o /dev/stdout | app2

But the situation is the same as stdout has buffer size restrictions too.

Anyone has any idea on how can I solve this specific scenario?

TL;DR: I need a "dummy" file that will act as stdout but without the standard size restrictions of the pipes.

Improve this question
3
  • It sounds like you have a deadlock issue that needs to be resolved. Creating an larger pipe is not going to be the best solution. Commented Jan 14, 2018 at 21:36
  • 3
    In particular, the statement "the pipe will fill up and everything will stop working" doesn't make much sense. If the pipe fills up, then app1 will block on a write, but app2 should continue working. When it clears data from the pipe, app1 will start writing more data. Unless you have some other deadlock you're not describing. Commented Jan 14, 2018 at 21:54
  • My deduction was because if I try to use a buffered pipe, after a couple of hours running, the app2 will stop working. If I use an unbuffered pipe, like stderr, the application keeps running for hours without problem. That was the only reason for my (maybe wrong) deduction. Commented Jan 16, 2018 at 23:14

3 Answers 3

Reset to default
1

There are several utils designed to handle similar situations:

  1. buffer: python app1 -o /dev/stdout | buffer | app2

  2. stdbuf: python app1 -o /dev/stdout | stdbuf app2

  3. unbuffer: python app1 -o /dev/stdout | unbuffer app2

  4. mbuffer (buffer with more options): python app1 -o /dev/stdout | mbuffer | app2

  5. bash process substitution: python app1 -o >(app2)

The utils have various options, some of which may be required here, (that depends on what app1 and app2 are doing). Some options set the size of the buffer, or add delays, or show diagnostic info.

Pixelbeat.org has some diagrams to help visualize how buffering works, (or fails to).

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Plus there's a socat trick here, but I'm not quite sure what it does...
0

You have a few options:

  1. Use a file. Instead of reading from stdin have the consumer read from a file and implement the file following code from "tail -f"
  2. Write a pipe buffer program. This option is kind of silly but works if you cannot change either of the others. I wrote one in Perl a while ago, sorry can't share it, but basically use non-blocking IO to read from a pipe and write to a pipe, holding all the data in memory. Probably good to log a complaint if memory use goes too high.
  3. Modify the reader or writer to use non-blocking IO and buffer the output or input.
Improve this answer

Comments

0

Well. My bad.

It was not a buffer problem, as some people suggested here.

It was a CPU cap problem. Both applications were consuming 100% for the CPU and RAM when running and that's why the application crashed.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.