[darcs-devel] http pipelining

David Roundy droundy at darcs.net
Mon Dec 17 17:23:33 UTC 2007

On Mon, Dec 17, 2007 at 04:30:47PM +0000, Duncan Coutts wrote:
> On Mon, 2007-12-17 at 11:15 -0500, David Roundy wrote:
> > On Mon, Dec 17, 2007 at 10:36:08AM -0500, Tim Chevalier wrote:
> > > I spent a few hours trying to implement this in darcs a few months
> > > ago, by using multiple Haskell threads in combination with
> > > Network.HTTP to simulate pipelining. I don't remember where I got
> > > stuck anymore, and I don't even know if that approach makes any sense
> > > to begin with.
> > 
> > I saw this on the bug tracker.  Any chance you'd be interested in taking
> > another look at it? At a minimum, the same infrastructure would be needed,
> > in that we would in either case need to figure out how to spark downloads
> > in parallel.  I think some sort of "par" approach to parallelization of
> > pure code ought to be able to handle this, as almost all our http requests
> > are wrapped in unsafeInterleaveIO wrappers (which should actually be safe,
> > since we always download straight to disk, and *when* we download a given
> > file should never matter... nor should it matter if we download them
> > needlessly).
> I would suggest doing proper HTTP pipelining rather than doing parallel
> downloads. HTTP pipelining will always be faster, it is a well supported
> standard and many http servers respond most unfavourably to parallel
> downloads even to the point of considering it abusive and responding in
> kind.

In either case, the sanest pure interface that I can imagine would involve
*darcs* seeing itself as doing parallel downloads, while the HTTP library
(whichever is used) queues these parallel downloads into an HTTP pipeline.
Otherwise we'd be reduced to moving things into the IO monad that don't
belong there, which would be rather ugly indeed.  Or alternatively, we
could restrict pipeline to special cases (e.g. darcs get), and use the
existing mechanism the rest of the time.  But it'd be nicer to have a nice
interface that optimizes all network access.

> How about adding proper pipelining to the HTTP package instead?

Yes, that would be a good idea for someone to do.
David Roundy
Department of Physics
Oregon State University

More information about the darcs-devel mailing list