[darcs-devel] http pipelining

[Duncan Coutts]

On Mon, 2007-12-17 at 11:15 -0500, David Roundy wrote:
> On Mon, Dec 17, 2007 at 10:36:08AM -0500, Tim Chevalier wrote:

> > I spent a few hours trying to implement this in darcs a few months
> > ago, by using multiple Haskell threads in combination with
> > Network.HTTP to simulate pipelining. I don't remember where I got
> > stuck anymore, and I don't even know if that approach makes any sense
> > to begin with.
> 
> I saw this on the bug tracker.  Any chance you'd be interested in taking
> another look at it? At a minimum, the same infrastructure would be needed,
> in that we would in either case need to figure out how to spark downloads
> in parallel.  I think some sort of "par" approach to parallelization of
> pure code ought to be able to handle this, as almost all our http requests
> are wrapped in unsafeInterleaveIO wrappers (which should actually be safe,
> since we always download straight to disk, and *when* we download a given
> file should never matter... nor should it matter if we download them
> needlessly).

I would suggest doing proper HTTP pipelining rather than doing parallel
downloads. HTTP pipelining will always be faster, it is a well supported
standard and many http servers respond most unfavourably to parallel
downloads even to the point of considering it abusive and responding in
kind.

How about adding proper pipelining to the HTTP package instead?

Duncan