Discussing free software syncronicity

I've been quiet for the last week as I've had a house guest here during that time and it's been rather distracting. In a good way, but still, it took up the time I'd usually spend obsessively blogging and doing similar such things. It's back to normal come this week, however, and I'm looking at Monday as the hard feature freeze day with a bit of excitement and a bit of trepidation. (It's always like that for me =)

Which brings my thoughts back to release cycles ("oh no!" you say, "not that again!"), so it was timely to read Mark's post that was, at least in part, a reply to my previous blogs on this matter. (Thanks, Mark, for taking the time to read and reply =)

The good news is that I think we're finding common ground: Mark sees the logic behind staggering upstream to reflect dependencies. This makes a lot more sense than the overly simplified "everyone synchronize!" message that was getting bandied about. It will still be insanely tricky to get these cycles timed out right so that dependencies are met, but it'll be interesting to perhaps try.

I do hope that as the projects start to look at each other's cycles more closely that they also start to look at each other's integration points more closely. There are obvious points of integration between certain projects that just haven't materialized in very compelling ways largely because of a lack of value put on such things in Free software projects.

I did notice that Mark also didn't bring up the "it's better for marketing!" argument this time. As a blogger on internetnews.com put it, "From a selfish journalist point of view - Shuttleworth's version of syncronicity would also be terribly boring. I mean instead of being able to write about Ubuntu, Fedora and OpenSUSE releases on their own specific release dates and give each their due - I'd have one release day for all and lump them all together." I really hope that we have put aside the whole marketing canard in this discussion for good, because it actually is a weak point in the argument for syncronicity rather than a strong point.

Getting back to release cycles ... Mark talked about punting features being a strength; he's approaching it from the perspective of discipline. I agree, but that's just half the picture. The other half is that if you set up your cycles such that you are punting more than you should, you end up with a jilted development system that is in a constant state of hurk-and-jerk with features not making open windows and getting delayed. You end up slowly marching into the molasses as features have dependencies, too, and so delaying one changeset will end up delaying other changesets. Rinse, repeat, watch development spread out thinner and thinner on the timeline.

Mark talked about missing the opportunity to be in a down stream release (and, btw, mistakenly says it would be another 6 months; that would imply a near miss, which is usually not the case meaning it's almost always rather less than 6 months), but he didn't talk about individual features and fixes missing releases either which is essentially the same thing.

I suggest that true syncronicity takes into consideration the total life-cycle of software, from inception as an idea to design to draft code to releasable code to release to system integration to user. To tie a release cycle to a development cycle will hurt one or the other in the sense of making one of them less efficient. This is needless waste.

Mark suggests doing development in branches and merging them in "when they are ready". This obviously favors the release process which is a point in time event that requires a fraction of the resources involved in the actual development process. Given that the creative process is ongoing and consumes the bulk of resources, the release process should flow from the development process not the other way around. Decoupling of to-market-release from development would let development happen unabated in a mainline branch with release points being syncronized from the actual streams of development. As long as there is some discipline in the development stream (such that known-good-points can be identified), release should be quite possible.

This does not imply a wild west development scenario where things are in constant flux. That would be Bad(tm). It instead lets each component pick its pace and sync at regular intervals as it makes sense to. Hopefully those cycles are nice and short, such as a few weeks at most.

The benefit of doing it this way, which is really sort of the opposite direction that Mark suggests (with development out of mainline and release being mainline instead), is that instead of N different branches happening everywhere it's possible to once again work together on the code base. Some things do benefit from being done in a branch: experimental or highly disruptive changes, for example. Then again, there are few things as frustrating as fixing multiple branches to work together as work in one diverges things. Been there, done that.

Generally, however, having people working in their own copies (microbranches?) with them pushing up to mainline keeps things moving really well. There's a lot to be said about being able to test everyone else's work nearly constantly in such an environment; problems get caught sooner, improvements are suggested earlier. There's less effort wasted in general. It's also a bit inane to branch to work on smaller changes, which tend to dominate most development cycles. Mark talked about lean processes, but only from the integration point of view; there are lean process in development as well, and defining the development cycle by the release cycle, particularly the wrong one, erodes the leanness of the development process. Since development is the bulk of resources ... it should be obvious where to pay attention to leanness more.

There's also the social/psychological aspect of not working together. When working alone in a branch, it moves only as fast as you do. In fact, when others move faster the integration of their changes into your branch can actually slow you down if you don't do it often enough. This is about more than just conflicts in lines of code changes, but also things like regressions in features that interact with or depend on each other. When working together in a common area with a minimal (but no less) amount of separation (something I'm really liking git for, btw; it seems to really "get it right" in this respect) other people's work moves your code base forward, too. You get a sense of momentum, of things happening, of excitement.

It is not overly dramatic to say that if we make Free software development overly sterile via choice of process, there will be a commensurate diminishment in participation and momentum.

Remember that it is the development process that delivers nearly all the value in a Linux distribution. The distributions make that value accessible to the masses and create new kinds of value on top of it (support, marketing, etc) but it is the development not the integration that is primary source of the value. It should then be obvious that the development process is not something you screw with lightly; fortunately I think both Mark and I can have our cake and eat it too by decoupling the two processes into coordinated, parallel efforts.

I think Mark understands this on some level, because he says, "There’s a lot of discussion about the exact length of cycle that is “optimal”, with some commentary about the windows of development, freeze, QA and so on. I think that’s a bit of a red herring, when you factor in good branching, because feature development absolutely does not stop when the trunk is frozen in preparation for a release. Those who prefer to keep committing to their branches do so, they scratch the itch that matters most to them."

So he gets it that the two processes should be decoupled, but maybe he gets the solution a bit backwards. Reality is that development does slow down during freezes; reality is that freezes happen at times that aren't convenient for people (remember that a good 70% of KDE development is done in people's own time!); reality is that as fixes happen in mainline, they then need to trickle back out to other branches.

So I'd much rather see release be done in a branch with development in a rapidly cycling mainline.

What about bug fixes and stabilization in releases? We've done exactly this for years in KDE: we backport bug fixes from trunk/ to branches/ so that fixes trickle out to the next minor release (e.g. 4.0.x) as we work on the next major release (e.g. 4.1.x).

So now here's a really radical idea (so hold on): given that there is a hunger for synchronized release cycles for downstream .... why doesn't downstream take on producing releases? Why not cease waiting for tarballs to be delivered to your doorstep and start working on a release engineering service!

Why not have the system integration community (mostly the OSVs, really) come together and branch things for release at a certain point in time, defined by them, and work with upstream on stabilization of that branch? Instead of hoping that upstream does what they want, why don't they rally a bunch of cohorts from Novell, Red Hat, Debian, Mandriva, the MacOS and Windows communities, Canonical and whomever else wishes to engage and start offering a real, serious release process for upstream development to filter into? Upstreams that adopt a compatible method of development (such that branching could be done with at least semi-predictable results) could candidate for this service.

Just imagine how much efficiency there is to be gained by each project not having to build it's own entire release team and infrastructure but share a common one that is riding on the pulse of the downstream integrators! This would incentivate upstream (time and energy savings), free upstream development from release constraints and give integrators a direct influence over release points.

This work would be right within the core compentencies of the OSVs (release engineering) and would be scratching their itch at the same time. Heck, you'd even manage to get the same version of software in all the participating distribution releases "for free" by virtue of them preparing the releases in a common arena together.

There would be a number of details to work out, but having thought about it I think it's really doable.

I know this would be a non-trivial investment, but if Mark is truly serious about what he suggests this would be a very compelling way to put his currency where his mouth is, so to speak. Stop trying to convince the world and just start doing it.