[darcs-users] managing change - lets have more tests

Eric Kow kowey at darcs.net
Sat Feb 14 13:02:19 UTC 2009


Hi everybody,

Thanks for all the comments on this thread!  The short summary is that I
now want us to do more testing (but still feeling conservative about the
rest).

Now, I'm going to try and summarise what you've written so far, my
responses and then some suggestions concrete things we need to do if we
want to move forward on this.  (And if you don't like my responses, at
least hear out the part about concrete actions?)

Basic issues
------------
1. Sunset helpful or harmful?

2. Will test driven development scare off darcs developers?

3. How to switch to test driven development?

Summary of the discussion
-------------------------
With apologies to anybody I may have omitted or misread:

Zooko said:
- trunk should always be good; never accept something broken with
  the aim to fix it later
- this needs a test driven development context
- a lot of 'real world' and integration issues are a lot easier to
  test automatically than one would initially expect (AIX example)
- policy suggestions
  * mandate that new patches have tests or they don't go in

Stephen J. Turnbull says:
- empirically, the sunset procedure wins [i.e. does more good than harm]
- external modules may not have same degree of platform support, so old
  code needs to stick around until they catch up
- test driven development not a panacea
  [TODO: reasons]
- bugs take a while to appear
- policy suggestions:
  * stick of enforcement needs to be paired with carrot of faster progress
  * Eric: start asking for tests, fast tracking patches with tests

Zooko replies to SJT:
- don't worry about turning off contributors
- culture shift will happen: we will just get used to fact that tests
  are required and adjust our behaviour accordingly

SJT and Dan Pascu not so sanguine:
- this may only work in projects big projects, i.e. which already have a
  lot of momentum or a large pool of potential contributors

Trent:
- would it be acceptable to just tell users who really care about
  stability to stick to the old version of darcs?
- policy suggestions:
  * by-attrition (see below)

Petr:
- agreement with old version for stability

Some responses
--------------
I'll just point out that darcs already does some amount of unit testing
in the form of QuickCheck tests on patch properties.  Indeed, it would
be good for us to extend this practice to other parts of darcs.

Also, I have noted a sort of general agreement that the way to "switch"
is by attrition, i.e. requiring that all new code be tested (without
worrying about the old code).  Thanks!

1. Even if we adopt a test-driven culture, I think we should maintain
   our insistence on reviewing every patch before it enters the darcs
   code base.  I realise it causes delays (which can be discouraging to
   developers) and that is prone to scalability issues, but I still feel
   that this is the way to go.
   
   Look at this way: there is something we all agree on, which is that
   we should be doing more testing.  So let's move forward on that
   first.  Over time we can hope that testing becomes a fixture in the
   darcs community, and we all gain confidence in the process.  As a
   result of this, I may "see the light" and change my mind.
   
   But until that happens, let's continue reviewing patches.  (By the
   way, I would love to have more volunteers for the review team!).

2. Likewise, I'm still reluctant to abandon the sunset process, but I
   think we can negotiate some sort of agreement on what kinds of things
   we will try to sunset and what sort of things we will let ourselves
   move faster on.
   
   As with #1, I am willing to change my mind, but slowly and
   cautiously.  Rather than talking about doing less review, or
   abandoning the sunset, let us focus on the thing we all agree on,
   which is doing more testing.  I realise that it can be frustrating
   when even effecting meta-change is slow, not to speak of actual
   change.  If there is anything I can do to mitigate this frustration,
   or if we can find more things that we can agree on (like testing!)
   and build upon, please let me know.  So, please be stubborn and keep
   pushing things forward.  Focus on getting what you want for darcs
   (testing) without worrying about what you don't want (sunsets),
   because if you get what you want, maybe what you don't want will
   wither away on its own accord!

3. As for asking people who care about stability to stick to older
   versions of darcs, my fear is that this will pretty much just cause
   people to generally use old versions of darcs (in which case, we
   still suffer the sunset problem, nobody testing the new features).
   Put yourself in the shoes of somebody who just uses the darcs,
   doesn't care about patch theory or whatever.  How much do you care
   about stability?  I would like us to be more successful at persuading
   people to upgrade darcs fairly frequently, because this really is in
   their best interests.  It seems that telling people to "just use an
   older darcs" is sending them the wrong message.  If we fix something
   (non-controversial and "safe"), then we want as many people to be
   using the fixed version.  So, my preference would still be for us to
   consistently encourage users to move forward, whilst holding up our
   end of the bargain in giving them reasonable stability.
   Does that make sense?

Let's do more testing: Concrete actions
---------------------------------------
I am not promising any policy changes, but I think it would be useful
for us to work on testing infrastructure, fulfilling the concrete
prerequisites needed to shift darcs over to a test-driven culture.
Three ways of thinking of this: one we're doing it just in case; two,
we're doing it so that it's easier for us to make the switch (less
inertia) if the debate comes up again and three maybe we don't have a
testing revolution, but just start doing more and more testing over
time, which is still better than nothing.

Perhaps we should work on clarifying the following questions?

1. What are we using as a test driver?  Right now we have src/unit.lhs
   which I confess that I am not running.  Does src/unit.lhs need to be
   transformed into a module and incorporated into Setup.lhs, along the
   lines of Distribution/ShellHarness.hs

2. What is the concrete policy on writing tests?  I'm talking
   nitty-gritty here.  For example, do you put tests in the same
   module as the code you are testing?  Should each module have
   a tests section (say on the bottom of the file), or below
   each function?  Don't answer this here -- no bikesheds -- please just
   start a draft on the wiki and work it out in another thread if you
   need.
   
3. What documentation do we need on our testing requirements?  Do we
   need a darcs testing tutorial?  Such a tutorial should not repeat
   what is already said in other Haskell testing tutorials, but focus on
   testing as it applies to darcs.

4. Can you volunteer to mentor people in writing tests? Can I have one
   or two people that I can point my finger at when a contributor
   protests that they don't know how to write tests, or that it would
   take them too much effort?

If the darcs community provide concrete solutions to the 4 issues above,
I will start requesting more tests.  I won't refuse patches for lack of
tests yet (still testing the test waters), but I definitely want to try
this thing out!

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20090214/04263468/attachment.pgp>


More information about the darcs-users mailing list