Pipe woes (not DF specific I guess)

[Joerg Sonnenberger]

On Thu, Feb 05, 2004 at 02:48:41PM +0100, Simon 'corecode' Schubert wrote:
> Heyas,
> 
> While doing some shell-fu (with grep and tee) I once again noted that 
> pipes don't really work the way I'd like them to work: Be line buffered 
> (not fully buffered) when the final output is a terminal.
> 
[snip]
> 
> Joerg wrote an approach with nonblocking IO and polling, but actually 
> I'd like to have some more generic thing, where not every involved 
> program needs to be changed.

There are aspects suboptimal and related to this topic:
- reading from a pipe does not block, to support pipes the polling is not
  necessary. This is not true e.g. for sockets. I'm not sure about FIFOs.
- stdio does line buffering for stdout if the output goes to a terminal,
  otherwise it is fully buffered
- some tools are switching between stdio for stdin and read/write for all
  other output

IMO we should really discuss this. At least all the tools meant for
streaming should do only a minimal amount of buffering. E.g. grep
should only do line buffering, tee and cat no buffering at all.

Other tools which might be reasonable called and used in pipes should
be changed to add some fsync at strategic places. Alternatively an
environmental flag could be added to force libc to use line buffering
by default for all streams, instead of full buffering.

Joerg

> cheers
>   simon
> 
> -- 
> /"\   http://corecode.ath.cx/#donate
> \ /
>  \     ASCII Ribbon Campaign
> / \  Against HTML Mail and News