Add new API for annoymous pipe and named fifo #375
Description
Proposal
Problem statement
Annoymous and named pipe are widely used, there is crate os_pipe which just provides an abstraction for pipe on unix and has 15 million download.
While having a third-party crate for it is enough for many crates, I think it'd be better if it's in stdlib so that it can be used without having to another dependency for this.
It would also enable better integration with the std::process API, since users might want to pipe output of multiple processes to one pipe and read them.
Motivating examples or use cases
jobserver-rs, for example, is used by cc-rs and pulled in as build-dependencies quite often.
It internally implements all kinds of API for annoymous pipe and named fifo, contains a bunch of unsafe code for this and quite some code for just managing the pipe/fifo.
It'd be great if we could move them to stdlib and make jobserver-rs easier to maintain.
It might also speedup jobserver-rs compilation since it could've drop the libc dependency.
tokio, the widely used async executor, already provide pipe support in mod tokio::net::unix::pipe.
Solution sketch
I suppose we can use os_pipe as basis and then add in more functions used by jobserver-rs, and functions provided by rustix:
The basic API for annoymous pipe:
mod pipe { // Put under std::io // Create annoymous pipe that is close-on-exec and blocking. pub fn pipe() -> io::Result<(PipeReader, PipeWriter)> { PipeBuilder::new().build() } #[derive(Debug, Default, Clone)] pub struct PipeBuilder { ... } impl PipeBuilder { pub fn new() -> Self; /// Enable user to share this pipe across execve. /// /// NOTE that the fd itself is shared between process, not the file descriptor, /// so if you change its non-blocking mode, it would affect every process using it. pub fn cloexec(&mut self, cloexec: bool) -> &mut Self; pub fn build() -> io::Result<(PipeReader, PipeWriter)>; } #[derive(Debug)] pub struct PipeReader(/* private fields */); impl PipeReader { pub fn try_clone(&self) -> Result<Self>; } #[cfg(unix)] impl AsFd for PipeReader { ... } #[cfg(unix)] impl AsRawFd for PipeReader { ... } #[cfg(windows)] impl AsHandle for PipeReader { ... } #[cfg(windows)] impl AsRawHandle for PipeReader { ... } // Use TryFrom here, because not every owned fd is a valid pipe #[cfg(unix)] impl TryFrom<OwnedFd> for PipeReader { ... } #[cfg(windows)] impl TryFrom<OwnedHandle> for PipeReader { ... } #[cfg(unix)] impl From<PipeReader> for OwnedFd { ... } #[cfg(windows)] impl From<PipeReader> for OwnedHandle { ... } impl From<PipeReader> for Stdio { ... } #[cfg(unix)] impl FromRawFd for PipeReader { ... } #[cfg(unix)] impl IntoRawFd for PipeReader { ... } #[cfg(windows)] impl FromRawHandle for PipeReader { ... } #[cfg(windows)] impl IntoRawHandle for PipeReader { ... } impl<'a> Read for &'a PipeReader { ...} impl Read for PipeReader { ... } #[derive(Debug)] pub struct PipeWriter(/* private fields */); impl PipeWriter { pub fn try_clone(&self) -> Result<Self>; } #[cfg(unix)] impl AsFd for PipeWriter { ... } #[cfg(unix)] impl AsRawFd for PipeWriter { ... } #[cfg(windows)] impl AsHandle for PipeWriter { ... } #[cfg(windows)] impl AsRawHandle for PipeWriter { ... } // Use TryFrom here, because not every owned fd is a valid pipe #[cfg(unix)] impl TryFrom<OwnedFd> for PipeWriter { ... } #[cfg(windows)] impl TryFrom<OwnedHandle> for PipeWriter { ... } #[cfg(unix)] impl From<PipeWriter> for OwnedFd { ... } #[cfg(windows)] impl From<PipeWriter> for OwnedHandle { ... } impl From<PipeWriter> for Stdio { ... } #[cfg(unix)] impl FromRawFd for PipeWriter { ... } #[cfg(unix)] impl IntoRawFd for PipeWriter { ... } #[cfg(windows)] impl FromRawHandle for PipeWriter { ... } #[cfg(windows)] impl IntoRawHandle for PipeWriter { ... } impl<'a> Write for &'a PipeWriter { ...} impl Write for PipeWriter { ... } }The basic API for named fifo:
// Under std::fs mod fifo { #[derive(Debug, Default, Clone)] pub struct FifoOpenOptions { ... } impl FifoOpenOptions { pub fn new() -> Self; pub fn create(&mut self, create: bool) -> &mut Self; pub fn create_new(&mut self, create_new: bool) -> &mut Self; pub fn write(&mut self, write: bool) -> &mut Self; pub fn read(&mut self, read: bool) -> &mut Self; pub fn open<P: AsRef<Path>>(&self, path: P) -> io::Result<Fifo>; } #[derive(Debug)] pub struct Fifo(/* private fields */); impl Fifo { pub fn try_clone(&self) -> Result<Self>; } #[cfg(unix)] impl AsFd for Fifo { ... } #[cfg(unix)] impl AsRawFd for Fifo { ... } #[cfg(windows)] impl AsHandle for Fifo { ... } #[cfg(windows)] impl AsRawHandle for Fifo { ... } // Use TryFrom here, because not every owned fd is a valid pipe #[cfg(unix)] impl TryFrom<OwnedFd> for Fifo { ... } #[cfg(windows)] impl TryFrom<OwnedHandle> for Fifo { ... } #[cfg(unix)] impl From<Fifo> for OwnedFd { ... } #[cfg(windows)] impl From<Fifo> for OwnedHandle { ... } impl From<Fifo> for Stdio { ... } #[cfg(unix)] impl FromRawFd for Fifo { ... } #[cfg(unix)] impl IntoRawFd for Fifo { ... } #[cfg(windows)] impl FromRawHandle for Fifo { ... } #[cfg(windows)] impl IntoRawHandle for Fifo { ... } impl<'a> Read for &'a Fifo { ...} impl Read for Fifo { ... } impl<'a> Write for &'a Fifo { ...} impl Write for Fifo { ... } }Extension methods for unix:
// Under std::os::unix pub const PIPE_BUF: usize = c::PIPE_BUF; // 4_096usize trait PipeBuildExt: Sealed { fn non_blocking(&mut self, non_blocking: bool) -> &mut Self; } impl PipeBuildExt for PipeBuild { ... } trait PipeExt: Sealed { fn set_non_blocking(&mut self, non_blocking: bool) -> io::Result<()>; } impl PipeExt for PipeReader { ... } impl PipeExt for PipeWriter { ... } impl PipeExt for Fifo { ... } trait FifoOpenOptionsExt: Sealed { fn non_blocking(&mut self, non_blocking: bool) -> &mut Self; fn mode(&mut self, mode: u32) -> &mut Self; fn custom_flags(&mut self, flags: i32) -> &mut Self; } impl FifoOpenOptionsExt for FifoOpenOptions { ... }Extension methods for Linux:
// Under std::os::linux::pipe #[derive(Debug, Clone, Copy)] pub struct SpliceFlags { ... } impl SpliceFlags { pub const MOVE: Self; pub const NONBLOCK: Self; pub const MORE: Self; pub const GIFT: Self; pub const fn combine(self, other: Self) -> Self; } impl BitOr for SpliceFlags { ... } impl BitOrAssign for SpliceFlags { ... } trait PipeReaderExt: Sealed { fn non_blocking_read(&self, bufs: &mut [io::IoSliceMut<'_>]) -> io::Result<()>; fn splice_to<FdOut: AsFd>( &self, fd_out: FdOut, off_out: Option<&mut u64>, len: usize, flags: SpliceFlags ) -> io::Result<usize>; /// convenient method for splice_to or splice_from, when both in/out /// are pipes/fifos. fn splice( &self, fd_out: &impl PipeWriterExt, len: usize, flags: SpliceFlags ) -> io::Result<usize>; fn tee( &self, fd_out: &impl PipeWriterExt, len: usize, flags: SpliceFlags ) -> io::Result<usize>; } impl PipeReaderExt for PipeReader { ... } impl PipeReaderExt for Fifo { ... } trait PipeWriterExt: Sealed { fn non_blocking_write(&self, bufs: &[io::IoSlice<'_>]) -> io::Result<()>; fn splice_from<FdIn: AsFd>( &self, fd_in: FdIn, off_in: Option<&mut u64>, len: usize, flags: SpliceFlags ) -> io::Result<usize>; } impl PipeWriterExt for PipeWriter { ... } impl PipeWriterExt for Fifo { ... } trait PipeExt: Sealed { pub fn get_pipe_size(&self) -> io::Result<usize>; pub fn set_pipe_size(&self, size: usize) -> io::Result<()>; } impl PipeExt for PipeWriter { ... } impl PipeExt for PipeReader { ... } impl PipeExt for Fifo { ... } // Under std::os::linux::fs trait FifoOpenOptionsExt: Sealed { fn open_at<P: AsRef<Path>>(&self, path: P, dirfd: BorrowedFd<'_>) -> io::Result<Fifo>; } impl FifoOpenOptionsExt for FifoOpenOptions { ... }Alternatives
Alternatively, we could first implement a subset of the API I proposed here to reduce the scope.
Then we could consider adding more methods that are needed.
Or we could just leave them up to third-party crates, which is the current status-quo, which is OK-ish but not good enough
for users who need pipe/fifo, they would have to grab a third-party crate for this.