[darcs-users] Why are tests done in reverse order by default?

Trent W. Buck trentbuck at gmail.com
Fri Oct 24 09:45:31 UTC 2008


Eric Kow <kowey at darcs.net> writes:

> On Fri, Oct 24, 2008 at 16:46:52 +1100, Trent W. Buck wrote:
>> twb> Why are tests done in reverse order by default?
>> twb> Because without that requirement,
>> twb>
>> twb>     printf %s\\n | sort -r | xarg $(SHELL_HARNESS)
>> twb> twb> reduces to twb> twb>     $(SHELL_HARNESS) *.sh
>
> No good reason that I can think of.
>
> We need the tests to be sorted though (for ease of use), but it seems
> like the sorting should be done in tests/shell_harness.

I don't agree; running the tests in a deterministic order (and more
importantly, in the same working directory) means that bugs in one
test can hide brokenness in another test.

When I removed the sort -r, resulting in pure lexicographic sorting
(because I use LC_COLLATE=C), I already found two bugs where scripts
assumed that temp1 didn't already exist.

In fact, I could make a case that randomizing the test order (sort -R)
is the best approach, because it allows us to check that there are no
implied dependency orderings.

I also wondered why if it might be worth simply generating a makefile
that can run each test independently, because make does set -ev by
default, also supports simple multiplexing with -jN and can aggregate
target failures using -k (which is basically what shell_harness does,
except it aggregates them).

> Care to look into it?

The patch bundle I've submitted just now strips out explicit sorting in
addition to all the other stuff.



More information about the darcs-users mailing list