[darcs-users] solution to the "slow get" problem?

[Juliusz Chroboczek]

>> Use pipelining instead.

Pipelining is a goodness.  And the sort of traffic that Darcs
generates would benefit a lot from it.

> I have to learn about pipelining then...

I've spent a few months implementing pipelining (in Polipo), and it's
tricky.  It's fairly easy to do an inefficient implementation
server-side (you just have to make sure you don't flush buffers
between requests).  Making best use of it on the server side --
opening the receive TCP window before you've sent the reply -- is more
difficult.

On the client side, it's absolute hell.  Pipelining becomes rather
tricky when you try to implement error recovery and timeouts, e.g. if
you've pipelined n requests and the first one fails, you should
restart the latter n-1.  Oh, and remember that TCP buffers are finite,
and hence you cannot just push a zillion requets and then wait for all
the replies, you'd deadlock.

(Plug: chapter 1 of the Polipo manual.)

> It's probably similar to what rsync does, isn't it? 

Indeed, rsync does pipelining.  I've never seen it deadlock :-)

> I see that it should be supported by all HTTP/1.1 servers,

You wish.  Polipo has some IMHO rather sophisticated heuristics to
detect broken servers and automagically disable pipelining, but still
I keep getting bug reports that are about server bugs.  It's getting
better since Mozilla started implementing pipelining, but there's
still a lot of those around.

Apache is fine, in all versions.  IIS 4 is broken.  IIS 5 is
officially fixed, but I've had bug reports against it.  Thtpd is
officially okay, but it seems to have some timing-related bug that can
only be triggered across the Atlantic and hence is somewhat difficult
to track down.

Back on-topic: the built-in web server of BitKeeper is broken beyond
belief ;-)

                                        Juliusz