kaizen and kakushin

I read Mark Shuttleworth's blog entry about meta-cycles this evening and was taken back in my mind to an entry I wrote in September of 2007 about Toyota.

My issue with the "6 month cycle" has always been three-fold:

1. the number 6 is a project specific detail and should not be pushed on every project because it won't fit every project (Mark notes he works on projects with 1, 3 and 6 month cycles; other cycle are also possible, of course); what we're really talking about it timed releases made with a relatively high level of consistency


2. The discussion is often coupled with the idea of coordinating different projects to the same cycle, which is really a different sort of conversation altogether with "length of the resulting release cycle" being an implementation detail within it


3. it's only one half of the equation, with the missing half being the non-high-level-of-consistency-time-interval releases; too often it was an either/or discussion where if you were doing one you couldn't have been doing the other

Both halves together make up the kaizen (continual incremental improvement) and the kakushin (continual leaps in innovation), and Mark notes that both are needed and can work together well. In fact, it seems that as the discussion continues, Mark and I see things increasingly similarly in general.

(Odd/intersting side point: I recently came across the concept of a "kaizen blitz" which happens to be exactly what our "developer sprints" are. Neat.)

Mark asks some questions, and since he asked I thought I'd share my personal answers to some of them:

'Is it true that the “first release of the major cycle” (KDE 4.0, Python 3.0) is best done as a short cycle that does not get long term attention?'

It's not he only way of doing it, but if the starting of the major cycle represents the end of a period of major innovation, it's only sensible to expect there to be a period of cooling off, dust settling and refinement. The point of making such a release is to draw a line in the sand saying "from here, be engage in the process of kaizen with earnest".

One can also do first major releases that don't bear these hallmarks very strongly at all (KDE 3.0 leaps to mind as an example of that), but then it isn't a kakushin release.

Being able to discern the difference between a release that is the result of kaizen versus one that more embodies kakushin is the key to whether that initial release should be "sat on" for a while or immediately pushed to production. The people to ask are the people doing the work.

Wait, though .. what does that mean, "people doing the work"?

In a FOSS project it's a combination of the people laying down the code / documentation / translations / art / etc and the people who rely on those technologies in their own projects. So KDE 4, as a concrete example, wasn't "ready" until application developers were confident with the platform, proven by basing releases of their own apps on it, and until production environment needs were met. So depending on who you are, 4.1 or 4.2 marked that point of "readiness".

With KDE 3, 3.0 was "ready" for all intents and purposes. 3.5 was miles better, but that's the point of kaizen and should not be confused for previous releases not being "ready".

This all implies that there is communication which is heard, listened to and delivered without unreasonable expectation. With 4.0, our communication was too often ignored, which erodes trust in both directions, and communication we received in return was often pejorative. The pejorative nature of it was mostly due to the lack of realization that 4.0 was the start of another period of kaizen (something we did for 6 years with KDE3) and the mid-cycle of a kakushin period.

There needs to be a way to allow releases like this to happen without it become a hail storm for all involved, and I personally believe that means generating an understanding of these two different but complimentary cycles and trusting people when they say "let's get busy with this, but it's going to be shaky in the short term" without freaking out about it.

We also need to not leave our users and developers high and dry: that's why we announced a year of KDE 3 releases at the KDE 4.0 release event right in the keynote. Back to Mark's questions:

'How do short term (3 month, 6 month) cycles fit into a broader meta-cycle?'

I think that really depends on the goals of the meta-cycle and the resources you can bring to bear to it. In general, however, there should be a lot more kaizen than kakushin type releases. Software needs revision to approach greatness, and in FOSS we do our revision in the open rather than behind closed doors. How many short term releases are "needed" and how they fit into a broader cycle is a bit like asking "how many licks to the center of a tootsie-pop". Well it really depends on how hard you lick it and how patient you are. :) This varies from project to project, and even within a project over time.

Compare the position of "GNOME is decadent" to "GNOME is all about 6 month cycles forever!"; they were both "right" for the project given its context when those positions were taken. It will shift, and to deny that is to deny the nature of existence itself.

This isn't a nice easy answer, I know. It means we can't draw cute little boxes on paper from here to eternity and have it all "figured out"; it means we need to do continual assessment. This is harder and requires more trust (or, to put it another way, is a more anxiety inspiring aproach ;), but it fits reality, one that is based on "continuousness of process".

Some people will need to plan long into the future, though. Distributions are one such group, needing to say things like "here's when our next long term, enterprise release will be" well in advance (this is implied by giving a life span to the current enterprise release, in fact).

So how do we balance those two needs? Personally, I think it can be solved by communicating with an appropriately long horizon. What does that mean?

Well, let's take the made up example of Frubagge. It's a hot new piece of technology that solves a really important gap in the Linux stack: the ability to read Dr. Seuss books with proper cooperative support from the kernel. Now, maybe we can already read books, but Dr. Seuss books with their anapestic tetrameter are a bit odd and we can do better. Frubbage just might be the key to unlocking it all. It also might not be. Distros need to know.

So maybe one or even a number of distros could say something like, "We're going to make Frubbage the default Dr. Seuss reader in one year's time in our main line releases. We're adding it as an experimental, will-eat-your-babies-and-steal-your-shoes, option." Then they would need to keep to that distinction and not get over enthused and confuse people by implying that the the experimental configuration is the main line stable release. Clarity and consistency, like good parenting.

Then all the book reader app devs out there need to pay heed and should start poking Frubbage and adding support in their apps. Maybe one group does a release with Frubbage support in 3 months time, and another will take 9 months to do the same. With a year's horizon, it's all details and none of it matters.

More importantly, those app developers now have a responsibility to speak up about Frubbage if it's somehow broken and then those distros need to pay heed to that. If the brokenness can be addressed and then is addressed, things go ahead as planned. If they don't, then the Frubbage horizon is either extended or dropped altogether.

This way the Frubbage project gets to "force" those persnickety app devs to kick its tires, the distros get what ends up looking like a coordinated release and the user doesn't get Frubbage unless it's "ready". Nobody loses, except maybe our proprietary competition as we start hitting on all cylinders and stop screwing up our platform on a regular basis. ;)

Most interesting, this doesn't require coordinating release cycles or punishing kakushin just because someone else is looking for kaizen that week.

My approach, as outlined above, is certainly not the only one and I admit it's probably a bit more Sun Tzu Planning than Gantt Chart Heaven, but I think it would work and work a lot better than what we have now.

Back to Mark:

'If it’s 2 years or 3 years, which is better for you?'

My answer: yes.

Honestly, from this high in the stack I don't care, just pick one and communicate it and then let us communicate back as to which release should hit that. Give me access to your spreadsheet so I can pencil in the KDE release version for you with reasoning and rational. If you want two years, awesome. If you want three years, awesome. KDE will be doing more kaizen than kakushin in that time period anyways, as we gracefully coast down the backide of our current kakushin cycles.

(To clarify: when I refer to the "backside process" means taking full advantage of the big initial arcs of the Pillars such as Nepomuk. Finally getting network transparency fully completed for Plasma components is another backside-of-the-initial-kakushin; not to be confused with the kaizen of making panel popups animate, for instance.)

As long as I know which it is, I can offer guidance. If our releases don't line up perfectly, that's OK, nobody's going to die. If your release concept is compelling, I might try and make my line up for that event. Cooperation as friendly negotiation.

Back to Mark:

'Does it make sense to have low-level, hardware-related things on a different cycle to high-level, user visible things? Or does it make more sense to have a rhythm of life that’s shared from the top to the bottom of the stack?

Would it make more sense to stagger long term releases based on how they depend on one another[?]'

I don't see how I can take advantage of new hardware features in my application until I can get my hands on them, nor can I see how KDE application developers could port to KDE 4 until they could get their hands on it.

There is some reality stuff that gets in the way here:

• there's too much going on for me to build everything from source all the time, so I do rely on binary packages for a lot of stuff which means waiting on actual releases that can then be packaged for me.


• I don't have the time to bleed on everyone's project's cutting edge, so except for very critical things to me I wait until there's at least a pack of bandages that comes with the release. I don't need stability, I just need the first aide kit to come included. ;)


• I don't want to wait for everyone to "be ready", because if the hardware guy can't release until I support it in my software that's going to suck for him: he needs to get on with his life too without me holding up his show because of coordinated releases; and the "show" is all about getting releases out in the wild

So if we had a constantly safe-enough, built from sources on demand base system and we were happy not doing quick releases of our stuff except to each other within our own little circle of happy coder friends, I don't see a non-staggered system working.

A non-staggered system could work, but we'd likely see Microsoft-esque releases: few and far between, and still half-baked when they first come out anyways.

'Is there any established theory or practice for this?'

There's lots said in business management and the social sciences about these kinds of processes. It's why I try and go to at least a few non-software events per year and read as much as I can afford to in these areas: they are solving similar problems and typically have a lot more experience (in both the good and bad directions) than we do. Software is a young industry, and modern FOSS is the baby brother in the family. The good news is we can learn from older siblings and don't have to reinvent it all.

We'll still screw up, I'm sure: we're learning, together, after all; but we can look at what others do. The ideas of kaizen and kakushin and how to combine them were not born from the software industry, let alone, FOSS, for example.

So, executive summary of my personal opinion on it all:

• We do need regular releases of continuous improvement


• We also need regular releases of major innovation, which will likely cause bruising that will be healed in time with the regular releases of continuous improvement


• We can blunt the impact of major innovation bruising on the user and vastly limit platform risk through processes of check/balance coordination (the Frubbage example)


• Staggered releases: realistic; lock-step releases: not realistic


• We can 'fake' lock-step releases by coordinating sufficiently openly, cooperatively and in advance as to what we (as a whole production line, top to bottom) are going to deliver as final products to market

It's mignight now .. so I'm going to take my tired butt to bed and dream of our next kaizen laden KDE release, 4.3.

Cheers...