[darcs-users] darcs patch: make test-running targets & docs a littl... (and 1 more)

[David Roundy]

On Thu, Sep 4, 2008 at 8:13 PM, Simon Michael <simon at joyful.com> wrote:
> Hi David, thanks for the reviews, it's appreciated. I don't want to
> spend much more time on these patches, but being interested in how we
> should encourage and grow more and better darcs contributors I'll pursue
> this a little more.
>
>> I find your refactored targets more confusing, sorry.  make check or
>> make test seems perfectly intuitive to me.  In fact, they're standard
>> targets.
>
> Agreed, and my proposed step forward wasn't perfect. Here are some
> reasons I didn't like the status quo:
>
> - make test runs the functional tests. On most projects I've worked on,
> the unit tests are the ones you run by default and frequently,
> functional tests are more expensive and run less frequently. With darcs
> this seems to be reversed, which felt and feels a little odd.

But in darcs the unit tests are only helpful when hacking on the
underlying darcs theory, while the  functional tests are both faster
and more useful,  and thus should be run by default and more
frequently.  They are the ones that check that darcs actually works.

> - make unit builds but does not run the unit tests; you have to do that
> first to make sure the binary is current, then run ./unit. I made the
> makefile do both of these things, so that you don't need to worry about
> whether you are running current unit tests and or have to remember how
> to run them. We say make suchandsuch for most tasks, let's make that
> convention work for all of them.

Sure, we could change  make unit to both  build it and run it.

> - make bugs sounds like something I don't want to run. I didn't do
> anything about that.

That's fine.

>>> This patch addes a "bench" target and trivial harness for running the
>>> tests in bench/. There's only one, but this seems like something we want
>>> to increase.
>>
>> I don't see this as a particularly good direction to go.  Let's not
>> codify this approach to creating a benchmark suite unless someone
>> decides it's worth working on.  As I understand things, there are
>> people working on other benchmarking projects.
>
> Is it the crappy test harness, or the fact that it's in our makefile at
> all ? People have been calling for more organized darcs performance and
> regression testing for a long time. Isn't it better to start with
> something, even a single number you can compare, than nothing ? And
> isn't our makefile a good place to control this thing, evolve as it may ?

It's the fact that no one is working on it, or seems likely to do so.
Nor is it necessarily a good  idea for them to do so.   Anyone
interested in working on darcs benchmarking really needs to be in
contact with the other people who are  doing the same thing, both to
reduce duplicated effort, and  to learn from  each other's mistakes.

> Stepping back: have you read Richard Gabriel's articles at
> http://en.wikipedia.org/wiki/Worse_is_Better ?
> My feeling has been that we want to increase the number of patches
> submitted and accepted (in non-sensitive areas), we want to increase the
> number and activity of patch contributors, increase the pool of
> reviewers and coders and thereby reinforce the darcs success train. We
> don't want to increase your workload, we do want to increase the
> project's capacity to handle more and better change. Happily that has
> happened a bit recently. To be most effective it needs to be part of our
> culture. So if a patch brings any improvement, it's often better to
> accept and improve it later, than to reject entirely. Would you agree
> with this principle generally ? I think you will, since I've seen you do
> that a bunch of times. So this is not to argue for this patch but to
> hear your view and clarify our policy.

Sometimes worse is just worse, and darcs has suffered too much from
accepting incomplete changes from folks  who then stopped working on
the project (or on that particular change).  As the  Worse is Better
article points out, simpler is often  better, even if it's less
featureful.

David