Lets say I know that a file descriptor fd is open for reading in my process. I would like to pipe data from this fd into a fifo that is available for reading outside my of process, in a way that avoids calling poll or select on fd and manually reading/forwarding data. Can this be done?
Add a comment |
1 Answer
You mean ask the OS to do that behind the scenes on an ongoing basis from now on? Like an I/O redirection?
No, you can't do that.
You could spawn a thread that does nothing but read the file fd and write the pipe fd, though. To avoid the overhead of copying memory around in read(2) and write(2) system calls, you can use sendfile(out_fd, in_fd, NULL, 4096) to tell the kernel to copy a page from in_fd to out_fd. See the man page.
You might have better results with splice(2), since it's designed for use with files and pipes. sendfile(2) used to require out_fd to be a socket. (Designed for zero-copy sending static data on TCP sockets, e.g. from a web server.)
Linux does have asynchronous I/O, so you can queue up a read or write to happen in the background. That's not a good choice here, because you can't queue up a copy from one fd to another. (no async splice(2) or sendfile(2)). Even if there was, it would have a specific request size, not a fire-and-forget keep copying-forever. AFAIK, threads have become the preferred way to do async I/O, rather than the POSIX AIO facilities.
6 Comments
iggy
thanks for the explanation, i guess
splice in a separate thread is a good choice. sendfile has a limit on the number of bytes it can send.
Peter Cordes
@iggy: splice and sendfile both have a
size_t length argument. IDK if it's better to call them with a small-ish size like 64kiB, or if it's good to use SIZE_MAX so your user-space loop only happens if/when the system call is interrupted (by a signal, for example). Actually transfering 2^64-1 bytes is unlikely to say the least, but you shouldn't assume that the call will never return.iggy
quoting doc for sendfile
sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.), but thanks for suggesting splice, that would definitely workPeter Cordes
@iggy: Which doc did you find that in? It's not in the current man page (Linux man-pages 3.74, sendfile man page last edited 2011-09-14). So IDK if it's an old limitation that's been removed, or if the documentation was previously incomplete.
iggy
you are right, this is probably not the most up-to date resource man7.org/linux/man-pages/man2/sendfile.2.html
|