[darcs-users] managing change

[Eric Kow]

Hi Petr,

So following up on my previous reply: I want us to come to some stable
agreement on how we can manage change smoothly.

Addressing darcs-users in general...

Ugly sunsets
============
What we are talking about here is making any sort of major change which
has a risk of breaking darcs.  Mostly it means getting rid of some old
code and replacing it with some external module.  The approach I am
defaulting to, is a conservative Sunset Procedure, rolling things out in
stages (major darcs releases), where at each stage

 - make the new code available through conditional compilation
   defaulting to the old code

 - default to the new code code (deprecating the old code)
   
 - remove the old code (hooray!)

The purpose of the sunset procedure is not to slow down change, but to
make sure it happens in a controlled fashion.  We cautiously expose the
new stuff to a gradually widening audience, and provide a known safe
fall-back until we're fairly confident that we are going to be alright.

As Petr points out, this approach has its disadvantages:
1. Slow:... it takes us one whole year to get rid of anything, as
    major darcs releases are every 6 months;
2. Time-consuming: because it causes us to split our time between
   maintaining the old stuff and working on the new stuff and
3  Potentially bad engineering: we're increasing the number of possible
   code paths (yuck! conditional compilation!) thereby reducing the
   amount of time that each path is explored.
So despite my claims, it's not even completely clear that the so-called
"conservative" sunset procedure is the sort of responsible engineering
practice that it aspires to be.

So why do we do it this way?

First, I hope that it allows us to discover flaws in the new code early
on before a lot of people start depending on it.  It's not clear that
this effective; maybe I have it all backwards, in the sense that they
only way we are going to discover the flaws (test-driven-development
aside) is to expose the new code to a wide audience as early as
possible.  For example, as Petr might suggest, instead of doing things
in three 6 month stages, we should compress it to one stage and count on
people using the unstable version to catch errors.

Second, by forcing ourselves to maintain the old code, we give ourselves
the option of easily switching back (tail between our legs) in case
something does go wrong.  The mere fact of using revision control, nay
the most advanced revision control system of the lot, doesn't really
cover this need.  The point is to do the "semantic" maintenance, always
making sure that the old stuff continues working as the rest of darcs
evolves.

Third, we want to make sure that we never break darcs, because sometimes
Life Just Happens: deadlines pile up at work, buses hit people, hackers
get girlfriends, babies are born.  The cautious approach is meant as a
sort of defensive mechanism to protect us from being left in a lurch in
case life ever happens to one of us.

Basically, the point is not to rush.  The sunset procedure is just a
concrete manifestation of a risk-averse, methodical and unhurried
attitude which I feel to be useful in this project.  I suspect
that there are really two debates to be had here, (A) if the attitude
itself is wrong ("if you want to make an omelette, you have to break
some eggs") and (B) if the concrete itself is mistaken.

Test-driven development : a new way forward?
============================================
> Instead of keeping things around in case "something breaks", we should ask new
> code to come with test coverage, and maybe correctness reasoning. I am not
> asking proofs of correctness, because nobody is going to supply them. But I do
> believe that a good testsuite is crucial for our success. Whenever manpower is
> in short supply, automated testing can be of great value, since it lets people
> worry less about introducing bugs, and keeps their hands free from superfluous
> testing and re-testing their code manually. Of course, bugs will happen, but if
> we are forced to live on our own dogfood, and on the bleeding edge of it, even,
> I don't think bugs will take a long time to get fixed. We should be forced to
> fix our own bugs, instead of giving ourselves easy ways to avoid them by
> compiling with different options.

I would also welcome the idea of using placing a heavier emphasis on
automated testing as an alternative means of managing change.  (We
already do a lot of automated testing, but we could really do a whole
lot more).

Now turning this into a culture shift is going to take a lot work.
There is a bit of static inertia and general fear of the unknown
(especially since despite my respect for test-driven development, I
don't have much practical experience putting it into practice) that
you're going to need to overcome.  So Petr, if you want something like
this to happen, you're going to need to show a little leadership on
this.  Don't just walk away because I throw a little resistance on you;
show me a better way!

Show me concretely how you use automated testing to manage the
transition from this slurp code we've been using for years to the new
shinier code.  Or maybe, take something relatively well-understood and
easy to test, like the date parsing code and show us a test plan for
that.  Show us what kind of patches a person might want to submit and
the kinds of tests it would be reasonable for us to expect.

For what it's worth: three concerns I might have are

1. The kinds of things the sunset procedure aims to catch are
   integration errors (unexpected interaction between different parts of
   darcs), and also real-world errors (e.g. HTTP not working behind
   proxies) that seem tricky to capture in laboratory conditions.
   I don't mean to say "we shouldn't do automated testing because it
   can't cover everything".  Of course we should do more automated
   testing.  But how should we catch the real world errors?

2. I'm not sure how to go about testing IO-intensive stuff (I guess
   our functional tests, i.e. the shell scripts, are a good example)

3. It seems that for a heavy reliance on testing to work, we are going
   to need to have much much wider test coverage.  How do we break out
   of this chicken and egg?  Do we put everything on hold and launch a
   massive darcs testing initiative?  Do we compromise and spend half
   our energy on testing, and the other half on doing new stuff?  It
   might be sensible.  More tests could just mean faster progress.

Anyway, that's the second batch of thoughts I have on this.  As you can
see, I'm not dead-set on sunsetting everything (for example, the LCS
stuff could go quite easily since it looks like something we can just
test), and I'm definitely willing for us to change the way we do things.
But I need help seeing how to go about it...

Thanks!

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20090206/350d74c7/attachment-0001.pgp>