[darcs-devel] Re: rerecord.pl test (was: darcs patch)

[Mark Stosberg]

On 2004-10-17, David Roundy <droundy at abridgegame.org> wrote:
>
> I'm flexible.  The most important thing is to have the test suite as
> complete as possible.  As long as I can understand and modify the tests,
> I'm all right with it.  Also, the barrier to writing new tests should be
> low, which I think precludes standardizing on perl.  I've been using
> autotest at work for some time now, and it's not bad, but is harder to
> use than plain shell.

Just to clarify: I'm not suggesting the whole test suite should be in
Perl, but just that it should be an acceptable alternative. 

> I don't really care whether the output looks nice, since most often the
> test is run without humans watching, as long as it's sufficiently verbose
> when there is a problem. 

With the Perl testing system, you have the option to run the tests in a
quiet mode or a verbose mode. With an up-to-date installation, a tool
called 'prove' will be installed. You can run tests with 'prove' for
minimal output or 'prove -v' for verbose output. 

With verbose output, you are given precise line numbers of the code
where the test failed.

The system produces nice summary reports like this:

Failed Test                   Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
base/data_integrity.pl           6  1536    15    6  40.00%  3 5-6 8 10 13
m1/shelter_add_edit_func.pl      8  2048   150    8   5.33%  88-89 91-95 150
m3/shelter_add_edit_func.pl      9  2304   105    9   8.57%  71-76 78-79 104
m3/shelter_volunteers_func.pl   12  3072    22   12  54.55%  9-20
m5/shelter_add_func.pl           8  2048   100    8   8.00%  90-93 95-97 100
m7/opt_out_func.pl               2   512     7    2  28.57%  4 6

You can then review the details to see what line numbers need to be reviewed.
I use this system to manage a project that has over 3000 tests in dozens of
test scripts, and I think it works quite well. 

> I didn't like the fact that when there was an
> error in the rerecord.pl script the test kept running, and ended up
> removing the directory.  It's nice to have it exit on the first failure, so
> I can go in and see the state of the repository, and preferably also print
> the output of the command that failed, so I can see what went wrong.

This is a difference with how the shell tests would work. With the perl
system, it's true that a failing test doesn't generally stop the test
script. (Although you could always have it die if the test failed). This
allows you to get summary reports about what failed, and then
prioritize what to work on first. 

>>   I believe I adjusted the Makefile appropriately so this will get
>>   run. However, I'm having trouble getting my environment set up so that
>>   "make test" works at all, so I wasn't able to test this.
>
> Works for me, except that the "successfully initialized" message has been
> removed, so the first test failed.

	Mark

-- 
http://mark.stosberg.com/