[darcs-users] darcs-benchmark (0.1.8.3, leak fixed?)

[Eric Kow]

Hi Zooko,

Many thanks for the report.

Reordering slightly.

> 3. Comparing *optimized* 2.4.4 to 2.5 with the "mean + sd" value shows
> that there is a significant performance regression for "darcs record".

Now that's surprising, given the fix in http://bugs.darcs.net/issue1106
In our other benchmarking efforts, darcs record improves rather
dramatically:

* http://wiki.darcs.net/Benchmarks/Dewdrop
* http://wiki.darcs.net/Benchmarks/GhLinux

Are you using the canned tahoe-lafs repository that one fetches with
darcs-benchmark, or are you using a more recent copy.

If the latter, how many patches since the last tag, please?  For what
it's worth, here's the output in our canned tahoe-lafs

% darcs changes --count --from-tag .
10

> 2. I like to look at mean + sd or mean + 2 * sd instead of just mean.
> My intuition is that I'd rather have something that's sort of fast
> almost every time than something that's really fast half the time and
> slow half the time, even if the latter thing has a better mean.

I don't know what the best practices are, but it does sound like this is
a more conservative and informative way to report results, at least in
the graphs.  If nobody complains, I may tweak darcs-benchmark so that
its graphs use the mean+sd figure.

One disadvantage for tables is that we'd have to explain that we do
this, and that people are unlikely to notice the explanation.

> Folks, I benchmarked darcs-2.4.4 vs. the current 2.5.0 pre-release on
> one of my linux servers with the Tahoe-LAFS repo. I reran the
> benchmark over and over until it stopped having any "?" or "~" in the
> output. Here is the final results:

By the way, I hope that a future version of darcs-benchmark will include
a feature to just run the benchmarks with '?' or '~'.

> Overall: way to go on making useful benchmarks that are easy to
> install and run! And way to go on making a new release of darcs that
> is mostly better and only possibly a little bit worse than 2.4.4!

More progress to go!  We still don't have automated benchmarking down pat
yet (or reporting of automated benchmarks).

Next up on the performance hacking front:

 - fast darcs annotate - I hope Benedikt will be able to push for getting
   this into Darcs HEAD as soon as the 2.5 release is out the door

 - fast darcs get over networks - Alexey's Summer of Code project will
   be refined over time.  One aim is to reduce the number of files that
   darcs has to download.

 - fixing Mac-related performance regressions?  Maybe clues from Florian's
   dtracing will lead us to some simple fishy thing that we can just poke
   ...

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
For a faster response, please try +44 (0)1273 64 2905.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20100730/f2dc4a57/attachment.pgp>