[darcs-users] Improving tests
tux_rocker at reinier.de
Mon Mar 23 21:14:09 UTC 2009
Lately there was a discussion on #darcs about improving the automated tests
for darcs. I intend to do some work on this at the darcs hacking sprint next
month, and I seek some input on what sort of improvement you'd like to see the
Currently, there are two testing systems in darcs. There are the functional
tests, that live as shell scripts in the tests/ directory. These simulate user
interactions with darcs and check that darcs does the right thing. Such
scripts are often written in response to a bug report, as regression tests to
assure that a bug never comes back.
Secondly, there are some quickcheck properties that can be run by doing a
"cabal configure -ftest && cabal build && dist/build/unit/unit". These are
known as "the unit tests" or "the slowtest". The latter name reflects the fact
that this test takes a couple of hours on current hardware. In contrast to the
functional tests, this test does targeted, rigorous tests of individual
What can be improved about this scheme?
* It should be faster. The functional tests take minutes to run, the unit
tests take hours. This does not have to be the case: xmonad's unit tests run
in 3 seconds on my machine. I have two ideas to make things faster. First, we
could rewrite the unit tests in Haskell. Haskell is faster than sh, of course,
and by calling libdarcs instead of the executable we can avoid lots of
creation of processes and starting and stopping the haskell runtime system. I
have no educated estimate of how much we can save by that, though.
Secondly we could carefully look at the quickcheck tests and write them in a
faster way. Many of the tests make quickcheck generate lots of complicated
data structures, only to discard them because they don't have a certain
property needed for the test. We could improve this by rewriting some of those
slow tests to use custom generators or pose less strict requirements on the
If we want to make things faster, we must also decide how fast it should be:
must we have a test target that runs in seconds? Or is minutes OK?
* It should be easier to make unit tests. There should be more documentation
on how to add unit tests, and it should be clear when and where you should add
a unit test. Currently, unit tests are thrown into a big file in src/unit.lhs
that also contains the test driver. We could craft a hierarchical file tree
for the tests that reflects the module tree of the actual code, and use a
third-party driver from hackage. testpack  and test-framework  on
hackage look promising. It would also be nice if you could run a meaningful
subset of the tests by using command-line flags, so that you can quickly test
the module you're messing with without running all the other stuff.
* Effectiveness. Of course, we have tests to catch bugs. But do they? I don't
see it happening that often. Is it because the coverage is too low? I don't
even know what the coverage is. Is it because we are covering the wrong
things? Perhaps we can start measuring test coverage. Has anyone ever done
that with a Haskell project?
Now, please let me know what your wishes are :-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 197 bytes
Desc: This is a digitally signed message part.
More information about the darcs-users