Edge of Chaos

Many interesting posts and discussions about Scrum and Kanban published last weeks. Someone even called this a war :) I don’t think it is a war, but some posts indeed may drive such feeling.

• Ken Schwaber bashed Kanban. “I was told that Kanban is frequently used when an organization cannot readily adopt Scrum. Many of Scrum most difficult aspects are then sidestepped. Managers are still in charge of telling people what to do. People can be interrupted at any time. People are still work in functional silos, preserving the jobs of functional managers. People are not allowed to work in containers, sharing skills and knowledge to bring complexity into solutions – instead they are worked on a pull (more sophisticated than push) production line.”
• Karl Scotland discusses Scrum and Kanban difference as Intentional vs. Implementational approaches. Interesting perspective in fact. Karl thinks that Kanban can be used with Scrum to reveal even more problems in development process.
• David J. Anderson posted the most rational article about Scrum and Kanban difference. I like it a lot. Some phrases are true gems: “Kanban is not a project management or software development lifecycle method. It is an approach to change management – a framework for catalyzing change in an organization.” and “Kanban uses a WIP limit as a change agent and Scrum uses commitments. This is a fundamental difference in approach.” David also posted some reflections later with interesting thoughts about anarchy and science.
• A year ago Tobias Mayer didn’t believe that it is possible to use Kanban at all. “I fundamentally disbelieve that there is any such thing as a “value stream” when you are working in a complex environment, in a creative process, building new products or generating new ideas.”

I think Ken is wrong. His arguments against Lean and Kanban quite ridiculous. Sure there is no intention to treat software development as a factory and apply lean manufacturing principles blindly. Of course there are people who will try (or already tried) that and fail. But vast majority of lean community in software development do understand the difference and working on concrete practices. Lean philosophy is great, but tools and practices can’t be taken from manufacturing directly.

Pull system is more complex than Push? C’mon!

David has wise arguments and perfect position. I think Kanban will be more popular than Scrum in long-term perspective, but it will take time. Visualization is a key thing to manage complexity, and software development is a complex system.

We live in the age of added value. It’s everywhere. Value-added services, value-added products, value-added goods etc. etc. etc.  Actually, so much value has been pumped up in our life, that it’s even strange that this value is not protruding from us like clothes from an overly packed vacation suitcase.

If we take a closer look at the back side of added value, a huge surprise is waiting there. The example I find most notorious is cell phones. What if I want a simplistic cell phone with NO Internet, no camera, even no voice mail, just live calls and SMS?  You’ll never ever find such a phone.

I bet that a phone manufacturer who stops the rush for more new features, would make a fortune in an instant selling the “new frugal” cell phones. In this case, the added value is content which comes from Internet, capabilities to exchange content. Why should someone want a phone without Internet?  My word, very soon we will see such phones on the market. The niche for them is already there. Here’s why:

More content and more various channels to produce and exchange content is now commonly presented as added-value. Hence, a communication device which happens to be a humble phone is supposed to deliver this value. But as we’re oversaturated with content, no buzz is a huge deal. The luxury of focusing on one thing at a time is something that only a few can afford. Besides, it’d be very frugal (frugal is actually the new buzzword :) to buy a phone for a reasonable price, and sell a used iPhone to some geek. Oh, pardon me. It’s all about iPads now :)

There’re plenty of such examples. Another one: the added value of having a car means lack of natural movement, the necessity to pay for gym etc. It actually brings along the whole array of more added value  goods and services that turn out to be not of added value but of less value, since you pay for what you could do naturally.

Take organic products. Now they’re added value. 100 years ago who could have thought that something natural adds value? Now it’s a rollback. What is simple and natural costs more, but has less value compared to the original added value concept for the matter, and the cycle goes on and on.

And we’re nailing it down to our favorite: project management tools. Versatility and too many features now have bounced back to the simplistic Kanban board.

It looks like it’s time to not only practice lean production, but to produce lean products. Gaining focus requires focused tools, one way or another.

Cumulative Flow diagram is a very good starting point for stop-the-line or retrospective meeting. Here is a real example from TargetProcess development.

You can see in the chart that we had a bottleneck in the beginning of December. It was caused by a quite complex user story. In theory a single user story should not affect cycle time significantly and should not create bottlenecks, but if you violate some rules it might be the case.

The user story was about replacing jQuery javascript framework with ExtJS framework. It took 1 month to implement by 1 developer. During this month we’ve made several releases and all went smooth. Then this user story was verified by testers and all existing acceptance tests were passed. So we’ve made a decision to merge this story to the main code line.

Unfortunately, after the merge quite many bugs were found in the build during smoke testing. These bugs took more than a week to fix, and during this time we were unable to release anything, since the merge was already done and the rollback was quite complex as well.

The lesson learned is to put more effort into testing of user stories that are more complex in nature. This particular story affected many places in the application and usual smoke testing was not enough. So we are going to introduce a new class of service (let’s call it  “technically complex story” so far :) which would mean more in-depth testing and verification before the merge.

In general, Cumulative Flow chart is a valuable tool to analyze historical data, but for emergency bottleneck identification Kanban Board is even better. If you have limits on Kanban Board, you see problems immediately.

You Kanbanized your development process and setup a perfect flow. You see the flow and feel its power. You even measure cycle time and want to reduce it. But it is not as easy as expected. Some problems in your development process are hard to find and are revealed on team level. Here I’ll describe an interesting technique that may help you.

The idea is quite simple. Take a single user story and visualize its states transition from start to finish, thus you see the whole flow of a single story. Draw all available states on Y-axis and days on X-axis. As a result, you will have a chart like that (click to enlarge).

It is a real user story from TargetProcess project. It was quite complex and its cycle time was 19 days. Let me provide some comments.

User story has been in Planned state during just 1 day, then developers took it and moved to In Progress state. So far so good. 4 days of development and user story is moved to Coded state, but it sits there for 2 days. Why? Why testers did not take it immediately? Clearly, we found 2 days of delay and the reason of the delay should be investigated, since it increases story cycle time.

Then testers took the user story and tested it for 2 days. Then the  story was moved to In Progress state again and developers worked on it for 2 more days (including Saturday). Why? It is a half of the initial development time. It is a clear sign of a waste, but we need more details. It appeared, that testers found 11 bugs. Half of the bugs were caused by unclear user story specification, other are subject to investigate. We should apply root-cause analysis here to find the real problems.

OK. Then testers have verified the user story for 2 days and moved it to Ready to Merge state. It was merged quite fast (in 1 day), but still there may be a waste to eliminate. Finally, the user story have lived in Merged state for 5 days and was released Jan-29.

In total (and as a minimum!) we discovered about 9 days of waste. It means we can reduce cycle time significantly by eliminating this wait time (waste). If you check several user stories using this technique, I believe you will find comparable results for simple and complex stories.

Here is another example for a quite simple user story. Still we clearly see wait times in our flow! We can reduce cycle time to 2 days instead of 6 by just eliminating waits in the flow. But it is not easy ;)

This chart is a really powerful tool. It is easy to create and you could use it to find waste in the development flow.

Kanban looks so simple. In theory. Map your flow, set limits, draw some lines and stick stories on the wall. There you go. Is that it? Obviously, no. One of the hardest things is to change your behavior, shift from “push mindset” to “pull mindset”. And that may be tricky.

You have Kanban board and there are no free slots in “In Progress” column. It means new stories should not be started — you’ve reached the limit. Developer comes and says “I know that there are no free slots, but I want to start this new small user story, since all the people are already working on the items and my help is not required”. You might think “Hmm, indeed we are moving forward with a good pace, this small story should not take much time…” and you say “OK, John, take it. I don’t think this would be a problem to exceed the limit right now.”

Two hours later Pete comes to you with a similar question: “I just completed my user story, testers will take it soon and I want to start something new.” You say “Well, we just took one story above the limit…”. Pete replies “Please, boss, I don’t want to sit and do nothing, I want to add some value, I am so much into work today!” Can you say no? I bet in most cases you say something like “OK, let’s do it!” to not discourage your teammate.

What happens then? Testers find some bugs in just implemented user stories, but there are no free developer to fix them. John and Pete are working on new stories. They may switch to bugs, but they’d lose focus and you have 2 stories in progress that can’t be passed to other developers easily (it will take time to make the transition and it looks like waste, isn’t it?). As a result, you deliver some stories later. Yes, later!

I understand that it is very hard initially to say something like “No, you’d better go and read something interesting about new technology. We can’t start new story right now, because if testers find bugs in some running stories, we should fix them ASAP, so we need a free developer”. Moreover, you should train your team to not ask such questions. WIP limit is a rule that should be followed. You may change it, but not arbitrarily. This change should have something behind (better productivity, more developers, etc). And this change should be discussed in retrospective meeting or during a Kaizen event.

I should say it is really, really hard to accept the fact, that sometimes developers and testers should do nothing (I mean do not start new stories when they are free). But it’s absolutely required to do so, otherwise pull system will not work.

It is always interesting to look behind the scene. In previous post I  described our development process, now I will show how we use Kanban Board in TargetProcess. Maybe it will give you some new ideas to try.

The most interesting part of every Kanban Board is its flow. So here is it:

Planned: contains all stories and bugs that are ready for development. It means they have full details/specification, so developers may start working on any item from this list anytime. Items are prioritized in this columns, first item should be taken first. Only product owner can put stories and bugs into Planned state from Backlog.

WIP Bugs: this is quite a special column. When testers find bug in just coded user stories, these bugs are drag-and-dropped in WIP bugs column. It means developers should take them ASAP to make this story Done-Done ASAP. Sometimes we put important bugs from production here as well. This state was introduced just several weeks ago. Works fine.

In Progress: shows stories and bugs that are in development now. When development pair takes a new story, the story moves here.

Coded: shows that story is implemented by developers. It is a queue for testers in fact.

Testing: shows stories that are in testing now.

Ready to Merge: we implement each story into a separate branch. If user story is in this state, it means it is implemented and tested, so may be merged into master. Every day someone merges all the items from this column.

Merged: this column contains all the stories and bugs that will be included into the next public build/release. We normally do weekly releases.

Done: shows several items included into the latest build.

We have a large TV in the office that displays Kanban Board all the time. Daily meetings are near the TV, so everybody can see it and understand what we are talking about.

Inspired by Kanban at Lonely Planet. I think it is a good idea to share software development processes from different companies. Maybe even it would be great to create a repository of the processes somewhere on a popular agile web site/blog.

Context

• Product development (quite large web application)
• Two development teams: Core (5 developers, 1 scrum master, 3 testers) and Integration (3 developers – one of them plays scrum master role, 1 tester)
• Quite mature product: 4 years
• Technology: C#, ASP.NET, ExtJS, NHibernate

Process

• Development team has all the power to try new practices, keep working practices and remove bad practices. There’s a little push from scrum master side.
• We do not use iterations.
• We do use releases, but often it is just a set of all the features implemented last month.
• We do not have iteration planning meeting, but discuss every user story JIT before implementation on a meeting with PO + Testers + Developers.
• We do not estimate stories and bugs, just a very brief estimate like (“oh, this will be a large story, let’s split it”).
• We split stories, but sometimes not as aggressively as required.
• We release new builds every week (sometimes 2 builds per week). The goal is to release every day, but we do not have a good automated functional tests coverage so far.
• Smoke testing of the build takes 3 hours (3 testers).
• We have a limit of 3 user stories or bugs in progress.
• We have retrospective meetings every 2 weeks.
• We sometimes have technical user stories (for example, remove jQuery from application), but that is rare.
• We use Kanban Board in TargetProcess heavily and have a large screen in development room.
• We have a flat hierarchy in the company. There are only two layers and no boundaries between the layers. No formalism at all. Also we do not have positions like Senior Developers, Junior Developers, etc. All people are equal (however, there are informal leaders).
• Average Cycle time for user story is 13 days and 5 days for bug.
• We track spent/remaining time on tasks and stories.

Development practices

• We develop each user story or bug in a separate branch. Master is always ready for release (after smoke testing). We use Git (switched from Subversion).
• Pair programming on all user stories and complex bug fixes. Sometimes people work alone on smaller/simpler problems.
• Developers switch pairs every day, even if user story is not completed yet. The goal is to a have fresh look, knowledge spreading and real collective code ownership.
• Before story implementation, pairs discuss possible solutions. It is quite formal.
• TDD. All the new code is covered by unit tests. Situation is worse on JavaScript side, we just started TDD adoption for UI.
• We use Cruise Control and have quite many different setups. Smoke builds, release builds, etc. Functional tests run in parallel and it takes 1 hour to run them all.
• Selenium. Automated functional tests are based on Selenium. We’ve created a framework in C# especially for our application to simplify tests creation. Tests coverage is not good so far, but we are working on it (there are 2 automation testers in our team).

ToDo

• We want to add performance tests into automated builds.
• We want to have 80% functional tests coverage to be able to release new builds every day
• We are incorporating User Experience into development process. This is our main goal for the next year. It includes Customers UX Community creation, interactive prototypes for all user stories, usability testing.

See what changed 1.5 years later.

In previous post I mentioned 5 wrong reasons to apply Kanban. Tobias Mayer asked to blog about 5 right reasons :) Obviously there should be right reasons, otherwise Kanban adoption would fail everywhere. So there they go:

#1. Ability to release anytime

In Scrum or XP you can’t release in the middle of an iteration. In Kanban you may release anytime. When user story is ready, you may release it. Definitely it is a challenge to setup development process this way. It is required to have “branch-by-feature” source control policy, merge often, integrate and test often. But it gives you an ability to release often. That is something worth to fight for.

As a PO I like this ability very much. An important user story implemented? OK, let’s release it tomorrow in v.2.15.2. Our customers may benefit from this release as soon as possible. Lead time is faster — customers are happier.

One thing to mention — you need to have a good automated test suite, otherwise it will be problematic to make small releases with a good quality.

#2. Ability to change priorities on the fly

In Scrum you can’t add stories on the fly into sprint (usually). At least it is not an easy process and development team often resists to replacing a story from Sprint backlog. The story has been discussed, estimated etc. A new story may be discussed in a hurry, some details will be missed and as a result significant re-work will be required. So in general iteration or sprint should be frozen.

In Kanban if you have an urgent request to implement or a really important user story, you can just put it on top of the queue. It will be taken as soon as a free slot is available. It is simply a Product Owner’s dream :)

#3. No need in iterations

Why you need iterations? Initially iterations help to reveal real problems in the development process quickly. Further on they establish a project rhythm and rituals. However at some point in project I think you no longer need them.

My vision is that you need to iterate first, then flow. Our backlog is fuzzy now, plans change often. We learn how to do agile in general and iterations are not helpful there anymore. Without iterations there is no need in iteration planning and iteration demo meetings. They are waste. Instead we have more smaller just-in-time meetings with people in a feature-team before starting the development for each user story.  It works perfect.

Some people are missing the rhythm, but I think it is more to habit. Kanban establishes more complex rhythm and it may take time for development team to recognize it. Still stable rhythm may be the only reason to keep iterations in the late phase of product development, in my opinion.

#4. No need in estimates

Why you need estimates? In iterative development you need them to predict how many stories you may take in the next iteration. You may even predict a release date based on estimated backlog and iteration velocity. The other reason is that PO wants to know how big the user story is. If it is big enough, PO  may consider moving it to the next release. If it is small, PO may decide to put it into the next iteration.

Obviously, iterative development is hardly possible without estimates. But if you drop iterations, it is not a problem anymore. Can Product Owner live without estimates? Well, yes. In our company we don’t estimate user stories. Why not? Because I as a PO don’t need them and we don’t use iterations. All I may ask is a very rough estimate like  normal, large, very large.

In our situation (I stress that, in our situation), estimates are waste. We don’t spend time on estimates. We have a prioritized backlog and just take the most important user story and implement it. It may not work in some conditions, but we have quite a mature product and hundreds requests from customers. There is no clean field development, so normally we don’t have any defined release date. We release when it is ready (I understand that many companies can’t do that, but we have this favor).

#5. Perfect flow visualization

Kanban Board provides a very clear view on current work in progress. It visualizes flow and enables fast planning and tracking. It is really a great tool, we love it.

Kanban is becoming a popular agile tool. Indeed it is very good for software projects of certain types. However, there is a danger of false reasons behind Kanban adoption.

#1. User Stories Diversity

“Our stories vary in size a lot from 1 point to 40 points. Large stories just do not fit into an iteration”

It is very easy to say you can’t split user stories, thus iterations should be abandoned. An easy solution is not the best one. There is something important behind the fact you can’t split stories. Most likely you don’t know how to split stories right. It is quite hard from the beginning and demands creativity.

Moreover, according to Queueing Theory, it is better to have small stories with roughly equal size. Small batches with equal size improve flow (and you have a better and more predictive cycle time). So in Kanban you still have to split stories to smaller stories and try to make them equal size.

#2. Failed Iterations

“We can’t complete most stories in a single iteration”

There may be many, many reasons behind. Mini-waterfall approach, velocity greed, large stories, manual testing, poor stories description, etc. You may even have a wrong iteration length. For example, in large projects 1 week iteration may be an overhead with a huge transaction cost. So 1 month iteration may be much better in this case.

You need to analyze true reasons, the roots of the problem.

#3. Failed Retrospective Meetings

“Retrospective meetings are waste, they do not help in process improvement and we want to remove them”

You just don’t do these meetings right. One popular failure is no Action Items after the meeting. Without action items a Retrospective Meeting is just a kind of informal chat that you may have over a glass of beer. You meet, chat about your problems, and get back on the same track next day. Rule #1 is to collect and write the list of Action Items that you will try to accomplish during the next Sprint [or any other time box].

One more  common pitfall is to skip action items execution. You may collect them, but not really try. You may even try them, but abandon too quickly. Almost all new rules or practices put you off the comfort zone. It takes time to learn them, use them and like them (or dislike), but it should be an expert opinion, not just a gut feeling.

If you expect that you will replace failed retrospective meetings with a nice and simple “stop the line” rule, you’ll fail, since it is even harder than scheduled regular retrospectives and demands even higher level of self-discipline.

#4. Shared People / Functional Departments

“We have a single pool of developers and share them between projects. We can’t form stable project teams”

Do you really think that Kanban will solve your social and organizational problems? C’mon! It helps to visualize flow and find bottlenecks, but if you don’t have a cross-functional team — you have hard times. It is proven in so many sources and researches that cross-functional teams perform better, produce better results and better software.

If you are experiencing difficulties planning sprints with shared pool of developers, try to fix the root of the problem first – switch to cross-functional teams and eliminate multi-tasking.

#5. Simplicity

“Kanban is so simple! No plans, no estimations, no iterations, no overhead”

Indeed Kanban looks simple. But it provides nothing interesting by itself. To ensure a successful Kanban adoption, you need to apply Lean principles first. This new buzzword may sound like a silver bullet, but obviously it  is not. Hard work, discipline, target on perfection and constant improvements – all that is required to apply any agile methodology.

Don’t get me wrong. I am not saying that Kanban is bad. We use it within TargetProcess and I personally believe it works way better than iterative development (for us).  I just want to make a point that you’d hardly resolve all your underlying problems as you switch to Kanban.

P.S. I usually don’t read posts like “10 reasons to …” Can’t believe I wrote such post myself! :)
P.P.S. Read next post 5 right reasons to apply Kanban.

There are several known areas where Kanban software development works fine: maintenance projects, game development, multi-media projects. There is no clear data whether it is good to use Kanban for product development or not.

I have my own vision of Kanban role in software development projects.  I think that

When you start a new product, you should use iterations first, but then may switch to pure flow development using Kanban.

Reasons are quite simple. Initially you do not have any burden. You have quite a large set of features that you want to push into Release #1. Definitely it’s better to have a small backlog, but it depends, you know. Sometimes  a “minimal marketable product” still has quite many features. So you have many user stories to implement, you do not have any bugs from production release, you do not have change requests, etc. It means you have a pretty stable environment with a quiet and stable backlog. And it means you don’t need to branch features and be able to release anytime. You may define clear goals for your iterations, estimate stories, use velocity and predict Release #1 date with acceptable accuracy.

However, things change with time. Suddenly you have large customers base, you have many enhancements, you have urgent bugs to fix and release. Now the environment is not so stable, you have unexpected events in the development process. Then I think it’s a good time to switch to Kanban. Urgent bugs may be fixed and released in 1-2 days, enhancements may be planned just in time with good release date prediction, velocity is no longer critical, since you do not need to predict large release dates, cycle time works fine for small features.

We’ve had this experience at TargetProcess. We’ve iterated for over 3 years, but several months ago switched to Kanban. Normally we do small releases each week, but are able to release anytime if required. We do not estimate stories anymore (maybe we will get back to story estimates, but doing without it so far). We are using Git and we’ve got a branch for each feature or bug.

Ironically, we stopped using some features in TargetProcess like release/iteration planning and progress tracking ourselves. It was a clear sign that Kanban support should be added to the tool. And it is on the way.