[darcs-users] managing change - lets have more tests
kowey at darcs.net
Sat Feb 14 13:02:19 UTC 2009
Thanks for all the comments on this thread! The short summary is that I
now want us to do more testing (but still feeling conservative about the
Now, I'm going to try and summarise what you've written so far, my
responses and then some suggestions concrete things we need to do if we
want to move forward on this. (And if you don't like my responses, at
least hear out the part about concrete actions?)
1. Sunset helpful or harmful?
2. Will test driven development scare off darcs developers?
3. How to switch to test driven development?
Summary of the discussion
With apologies to anybody I may have omitted or misread:
- trunk should always be good; never accept something broken with
the aim to fix it later
- this needs a test driven development context
- a lot of 'real world' and integration issues are a lot easier to
test automatically than one would initially expect (AIX example)
- policy suggestions
* mandate that new patches have tests or they don't go in
Stephen J. Turnbull says:
- empirically, the sunset procedure wins [i.e. does more good than harm]
- external modules may not have same degree of platform support, so old
code needs to stick around until they catch up
- test driven development not a panacea
- bugs take a while to appear
- policy suggestions:
* stick of enforcement needs to be paired with carrot of faster progress
* Eric: start asking for tests, fast tracking patches with tests
Zooko replies to SJT:
- don't worry about turning off contributors
- culture shift will happen: we will just get used to fact that tests
are required and adjust our behaviour accordingly
SJT and Dan Pascu not so sanguine:
- this may only work in projects big projects, i.e. which already have a
lot of momentum or a large pool of potential contributors
- would it be acceptable to just tell users who really care about
stability to stick to the old version of darcs?
- policy suggestions:
* by-attrition (see below)
- agreement with old version for stability
I'll just point out that darcs already does some amount of unit testing
in the form of QuickCheck tests on patch properties. Indeed, it would
be good for us to extend this practice to other parts of darcs.
Also, I have noted a sort of general agreement that the way to "switch"
is by attrition, i.e. requiring that all new code be tested (without
worrying about the old code). Thanks!
1. Even if we adopt a test-driven culture, I think we should maintain
our insistence on reviewing every patch before it enters the darcs
code base. I realise it causes delays (which can be discouraging to
developers) and that is prone to scalability issues, but I still feel
that this is the way to go.
Look at this way: there is something we all agree on, which is that
we should be doing more testing. So let's move forward on that
first. Over time we can hope that testing becomes a fixture in the
darcs community, and we all gain confidence in the process. As a
result of this, I may "see the light" and change my mind.
But until that happens, let's continue reviewing patches. (By the
way, I would love to have more volunteers for the review team!).
2. Likewise, I'm still reluctant to abandon the sunset process, but I
think we can negotiate some sort of agreement on what kinds of things
we will try to sunset and what sort of things we will let ourselves
move faster on.
As with #1, I am willing to change my mind, but slowly and
cautiously. Rather than talking about doing less review, or
abandoning the sunset, let us focus on the thing we all agree on,
which is doing more testing. I realise that it can be frustrating
when even effecting meta-change is slow, not to speak of actual
change. If there is anything I can do to mitigate this frustration,
or if we can find more things that we can agree on (like testing!)
and build upon, please let me know. So, please be stubborn and keep
pushing things forward. Focus on getting what you want for darcs
(testing) without worrying about what you don't want (sunsets),
because if you get what you want, maybe what you don't want will
wither away on its own accord!
3. As for asking people who care about stability to stick to older
versions of darcs, my fear is that this will pretty much just cause
people to generally use old versions of darcs (in which case, we
still suffer the sunset problem, nobody testing the new features).
Put yourself in the shoes of somebody who just uses the darcs,
doesn't care about patch theory or whatever. How much do you care
about stability? I would like us to be more successful at persuading
people to upgrade darcs fairly frequently, because this really is in
their best interests. It seems that telling people to "just use an
older darcs" is sending them the wrong message. If we fix something
(non-controversial and "safe"), then we want as many people to be
using the fixed version. So, my preference would still be for us to
consistently encourage users to move forward, whilst holding up our
end of the bargain in giving them reasonable stability.
Does that make sense?
Let's do more testing: Concrete actions
I am not promising any policy changes, but I think it would be useful
for us to work on testing infrastructure, fulfilling the concrete
prerequisites needed to shift darcs over to a test-driven culture.
Three ways of thinking of this: one we're doing it just in case; two,
we're doing it so that it's easier for us to make the switch (less
inertia) if the debate comes up again and three maybe we don't have a
testing revolution, but just start doing more and more testing over
time, which is still better than nothing.
Perhaps we should work on clarifying the following questions?
1. What are we using as a test driver? Right now we have src/unit.lhs
which I confess that I am not running. Does src/unit.lhs need to be
transformed into a module and incorporated into Setup.lhs, along the
lines of Distribution/ShellHarness.hs
2. What is the concrete policy on writing tests? I'm talking
nitty-gritty here. For example, do you put tests in the same
module as the code you are testing? Should each module have
a tests section (say on the bottom of the file), or below
each function? Don't answer this here -- no bikesheds -- please just
start a draft on the wiki and work it out in another thread if you
3. What documentation do we need on our testing requirements? Do we
need a darcs testing tutorial? Such a tutorial should not repeat
what is already said in other Haskell testing tutorials, but focus on
testing as it applies to darcs.
4. Can you volunteer to mentor people in writing tests? Can I have one
or two people that I can point my finger at when a contributor
protests that they don't know how to write tests, or that it would
take them too much effort?
If the darcs community provide concrete solutions to the 4 issues above,
I will start requesting more tests. I won't refuse patches for lack of
tests yet (still testing the test waters), but I definitely want to try
this thing out!
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 197 bytes
Desc: Digital signature
More information about the darcs-users