[darcs-users] Improving tests

[Reinier Lamers]

Hi all,

Lately there was a discussion on #darcs about improving the automated tests 
for darcs. I intend to do some work on this at the darcs hacking sprint next 
month, and I seek some input on what sort of improvement you'd like to see the 
most.

Currently, there are two testing systems in darcs. There are the functional 
tests, that live as shell scripts in the tests/ directory. These simulate user 
interactions with darcs and check that darcs does the right thing. Such 
scripts are often written in response to a bug report,  as regression tests to 
assure that a bug never comes back.
Secondly, there are some quickcheck properties that can be run by doing a 
"cabal configure -ftest && cabal build && dist/build/unit/unit". These are 
known as "the unit tests" or "the slowtest". The latter name reflects the fact 
that this test takes a couple of hours on current hardware. In contrast to the 
functional tests, this test does targeted, rigorous tests of individual 
functions.

What can be improved about this scheme?

* It should be faster. The functional tests take minutes to run, the unit 
tests take hours. This does not have to be the case: xmonad's unit tests run 
in 3 seconds on my machine. I have two ideas to make things faster. First, we 
could rewrite the unit tests in Haskell. Haskell is faster than sh, of course, 
and by calling libdarcs instead of the executable we can avoid lots of 
creation of processes and starting and stopping the haskell runtime system. I 
have no educated estimate of how much we can save by that, though. 
Secondly we could carefully look at the quickcheck tests and write them in a 
faster way. Many of the tests make quickcheck generate lots of complicated 
data structures, only to discard them because they don't have a certain 
property needed for the test. We could improve this by rewriting some of those 
slow tests to use custom generators or pose less strict requirements on the 
input data.
If we want to make things faster, we must also decide how fast it should be: 
must we have a test target that runs in seconds? Or is minutes OK?

* It should be easier to make unit tests. There should be more documentation 
on how to add unit tests, and it should be clear when and where you should add 
a unit test. Currently, unit tests are thrown into a big file in src/unit.lhs 
that also contains the test driver. We could craft a hierarchical file tree 
for the tests that reflects the module tree of the actual code, and use a 
third-party driver from hackage. testpack [0] and test-framework [1] on 
hackage look promising. It would also be nice if you could run a meaningful 
subset of the tests by using command-line flags, so that you can quickly test 
the module you're messing with without running all the other stuff.

* Effectiveness. Of course, we have tests to catch bugs. But do they? I don't 
see it happening that often. Is it because the coverage is too low? I don't 
even know what the coverage is. Is it because we are covering the wrong 
things? Perhaps we can start measuring test coverage. Has anyone ever done 
that with a Haskell project?

Now, please let me know what your wishes are :-)

Regards,
Reinier

[0]: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/testpack
[1]: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/test-framework
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20090323/46fbf6db/attachment.pgp>