[darcs-devel] [issue1283] run tests in separate subdirs

[Trent Buck]

New submission from Trent Buck <trentbuck at gmail.com>:

Currently, all the tests in tests/ run directly in tests-foo/ and many
have a prelude along the lines of

    rm -rf test                 # cleanup after a previous run
    mkdir test
    cd test

This is stupid for two reasons: firstly, it is duplication of code.
Secondly, screw-ups are not obvious because a previous test script
might be inadvertently "fixing" the broken script.  We found two such
bugs merely by reversing the order in which test scripts ran!

Instead, the script harness should create a completely clean scratch
directory for each test, and cd into it before running the test.  It
should also automatically remove this scratch directory at the end of
the test -- UNLESS the test failed, in which case the scratch
directory might be left around as a diagnostic tool.

I propose that "cabal test" continue to use tests.<repo format>/
directories as it does now, but within that directory create a <script
name>/ directory for each test.

PS: when a test fails, perhaps its stderr and stdout could also be
placed in this test-specific directory, rather than printing it to
standard output (which tends to be overwhelming when you initially
just want a list of failing tests).

----------
assignedto: twb
messages: 6893
nosy: dmitry.kurochkin, kowey, simon, thorkilnaur, twb
status: unread
title: run tests in separate subdirs

__________________________________
Darcs bug tracker <bugs at darcs.net>
<http://bugs.darcs.net/issue1283>
__________________________________