[darcs-users] state of HTTP pipelining in Darcs?

Eric Kow kowey at darcs.net
Thu Jun 24 13:33:55 UTC 2010

Hi Dmitry,

I just read a note from myself asking me to follow-up on the HTTP pipelining
work.  Since we're coming up to a release, I thought I'd better do it now :-)

Could I ask what is the state of HTTP pipelining in Darcs?  Are there any
outstanding issues you think should still be fixed?

If I understand correctly, all your recent pipelining patches were applied:

Sun Apr 18 23:10:13 BST 2010  Dmitry Kurochkin <dmitry.kurochkin at gmail.com>
  * Simplify libcurl pipelining configuration.

Fri Apr 16 12:21:05 BST 2010  Dmitry Kurochkin <dmitry.kurochkin at gmail.com>
  * Pass -DCURL_PIPELINING to C compiler when HTTP pipelining is enabled.

Sun Apr 18 16:03:02 BST 2010  Dmitry Kurochkin <dmitry.kurochkin at gmail.com>
  * Darcs.Repository: use pipelining when copying patches.

In 2010-04-16 (GSoC Project: Improving darcs' network performance),
you said:

> 1. I was able to reproduce the "no running handles" bug reported by
> many people recently using HTTP server at localhost.

That's http://bugs.darcs.net/issue1770 fixed by your
'Simplify libcurl pipelining configuration' patch

> 2. Pipelining works only if darcs makes several parallel requests. I
> added few debug prints to understand why no pipelining is done for
> http://darcs.net repo. copyFileUsingCache tries to download patches to
> /home/darcs-unstable/darcs/_darcs/pristine.hashed/ directory. There is
> no such directory on my system, so createDirectoryIfMissing fails. And
> speculateFileOrUrl is never called.
> 3. If I create this directory on my system, some pipelining starts to
> work. After several requests that reuse one connection, curl starts to
> open parallel connections to darcs.net. This seems a bug in curl to
> me, but it needs more investigation. Later darcs hangs in waitUrl.
> This is fixed in another patch that I would send shortly.

In a reply you added:
> Turns out that when built with cabal -DCURL_PIPELINING flag is never
> passed to C compiler. Fixed by patch208. Looks like in cabal builds
> pipelining was always broken because of this.

...and patch268 was applied.  But does this address all of #2 and #3
above or is there more work to do?
> Summary: pipelining is tricky. There are problems on darcs side (both
> in network code and patch fetching code in general) and, perhaps, curl
> side. So it can be not trivial to investigate and fix. But I think
> this is one of the things that should be done to improve darcs network
> performance. AFAIK there were never a proper pipelining benchmarks
> done, but during my first tests with libwww and darcs-1 I got very
> nice results.

Maybe a summary of what work (if any) needs doing?  Also benchmarking the http
stuff would be great.  We have this darcs-benchmark tool now... not sure if we
can apply it effectively for this.

> I believe that pipelining combined with patch packs will resolve all
> darcs network performance problems. Smart server, IMHO, is not the way
> to go - too complex (from both user and development POV) for little
> benefit (compared to patch packs and proper pipelining).

Oh, by the way... Alexey's packing work is going to be applied quite
soon.  Exciting times!

Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20100624/6725eab6/attachment.pgp>

More information about the darcs-users mailing list