[darcs-users] darcs-benchmark (0.1.8.3, leak fixed?)

Zooko O'Whielacronx zooko at zooko.com
Wed Aug 4 03:58:56 UTC 2010


On Fri, Jul 30, 2010 at 12:05 PM, Eric Kow <kowey at darcs.net> wrote:
>
>> 3. Comparing *optimized* 2.4.4 to 2.5 with the "mean + sd" value shows
>> that there is a significant performance regression for "darcs record".
…
> * http://wiki.darcs.net/Benchmarks/Dewdrop
> * http://wiki.darcs.net/Benchmarks/GhLinux

I added

http://wiki.darcs.net/Benchmarks/TahoeLAFSOrg

> Are you using the canned tahoe-lafs repository that one fetches with
> darcs-benchmark, or are you using a more recent copy.

The canned one, to make the results more comparable to other people's
benchmarks.

> I don't know what the best practices are, but it does sound like this is
> a more conservative and informative way to report results, at least in
> the graphs.  If nobody complains, I may tweak darcs-benchmark so that
> its graphs use the mean+sd figure.

I think this is a good way to go. See for example the Amazon Dynamo
paper [1]. If I recall correctly, established practice within Amazon
server engineering is to measure and optimize for the 99.9th
percentile. So, if your service would respond within 100 ms on mean,
but would, one in 1000 times, respond in 500 ms, then you would fail
the "300 ms 99.9 percentile" requirement.

Using mean + sd or mean + 2 * sd is a reasonable approximation of a
percentile requirement.

[1] http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

> By the way, I hope that a future version of darcs-benchmark will include
> a feature to just run the benchmarks with '?' or '~'.

I was thinking maybe a command to run everything at least 20 times so
that no '?' or '~' would appear (and the results would be more
statistically reliable).

> More progress to go!  We still don't have automated benchmarking down pat
> yet (or reporting of automated benchmarks).

I do think automating the execution of benchmarks and collection of
reports. We already have this:

http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/77/steps/shell_2/logs/stdio
http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/78/steps/shell_2/logs/stdio
http://buildbot.darcs.net/builders/6.12.1%20Debian/builds/79/steps/shell_2/logs/stdio

Hm... These are interesting results here. The build 77 had these benchmarks:

get (full) 6.2s d=0.1
get (lazy) 1.5s d=1.1
pull 100 3.2s d=0.2

> Next up on the performance hacking front:
>
>  - fast darcs annotate - I hope Benedikt will be able to push for getting
>   this into Darcs HEAD as soon as the 2.5 release is out the door


Yay! :-)

>  - fast darcs get over networks - Alexey's Summer of Code project will
>   be refined over time.  One aim is to reduce the number of files that
>   darcs has to download.

Yay! :-)

This should have the accidental side-effect of making darcs work much
better over Tahoe-LAFS.

>  - fixing Mac-related performance regressions?  Maybe clues from Florian's
>   dtracing will lead us to some simple fishy thing that we can just poke

Yay! :-)

Regards,

Zooko


More information about the darcs-users mailing list