Edge of Chaos | Agile Development Blog

Inspired by Kanban at Lonely Planet. I think it is a good idea to share software development processes from different companies. Maybe even it would be great to create a repository of the processes somewhere on a popular agile web site/blog.

Context

• Product development (quite large web application)
• Two development teams: Core (5 developers, 1 scrum master, 3 testers) and Integration (3 developers – one of them plays scrum master role, 1 tester)
• Quite mature product: 4 years
• Technology: C#, ASP.NET, ExtJS, NHibernate

Process

• Development team has all the power to try new practices, keep working practices and remove bad practices. There’s a little push from scrum master side.
• We do not use iterations.
• We do use releases, but often it is just a set of all the features implemented last month.
• We do not have iteration planning meeting, but discuss every user story JIT before implementation on a meeting with PO + Testers + Developers.
• We do not estimate stories and bugs, just a very brief estimate like (“oh, this will be a large story, let’s split it”).
• We split stories, but sometimes not as aggressively as required.
• We release new builds every week (sometimes 2 builds per week). The goal is to release every day, but we do not have a good automated functional tests coverage so far.
• Smoke testing of the build takes 3 hours (3 testers).
• We have a limit of 3 user stories or bugs in progress.
• We have retrospective meetings every 2 weeks.
• We sometimes have technical user stories (for example, remove jQuery from application), but that is rare.
• We use Kanban Board in TargetProcess heavily and have a large screen in development room.
• We have a flat hierarchy in the company. There are only two layers and no boundaries between the layers. No formalism at all. Also we do not have positions like Senior Developers, Junior Developers, etc. All people are equal (however, there are informal leaders).
• Average Cycle time for user story is 13 days and 5 days for bug.
• We track spent/remaining time on tasks and stories.

Development practices

• We develop each user story or bug in a separate branch. Master is always ready for release (after smoke testing). We use Git (switched from Subversion).
• Pair programming on all user stories and complex bug fixes. Sometimes people work alone on smaller/simpler problems.
• Developers switch pairs every day, even if user story is not completed yet. The goal is to a have fresh look, knowledge spreading and real collective code ownership.
• Before story implementation, pairs discuss possible solutions. It is quite formal.
• TDD. All the new code is covered by unit tests. Situation is worse on JavaScript side, we just started TDD adoption for UI.
• We use Cruise Control and have quite many different setups. Smoke builds, release builds, etc. Functional tests run in parallel and it takes 1 hour to run them all.
• Selenium. Automated functional tests are based on Selenium. We’ve created a framework in C# especially for our application to simplify tests creation. Tests coverage is not good so far, but we are working on it (there are 2 automation testers in our team).

ToDo

• We want to add performance tests into automated builds.
• We want to have 80% functional tests coverage to be able to release new builds every day
• We are incorporating User Experience into development process. This is our main goal for the next year. It includes Customers UX Community creation, interactive prototypes for all user stories, usability testing.

See what changed 1.5 years later.

In previous post I mentioned 5 wrong reasons to apply Kanban. Tobias Mayer asked to blog about 5 right reasons :) Obviously there should be right reasons, otherwise Kanban adoption would fail everywhere. So there they go:

#1. Ability to release anytime

In Scrum or XP you can’t release in the middle of an iteration. In Kanban you may release anytime. When user story is ready, you may release it. Definitely it is a challenge to setup development process this way. It is required to have “branch-by-feature” source control policy, merge often, integrate and test often. But it gives you an ability to release often. That is something worth to fight for.

As a PO I like this ability very much. An important user story implemented? OK, let’s release it tomorrow in v.2.15.2. Our customers may benefit from this release as soon as possible. Lead time is faster — customers are happier.

One thing to mention — you need to have a good automated test suite, otherwise it will be problematic to make small releases with a good quality.

#2. Ability to change priorities on the fly

In Scrum you can’t add stories on the fly into sprint (usually). At least it is not an easy process and development team often resists to replacing a story from Sprint backlog. The story has been discussed, estimated etc. A new story may be discussed in a hurry, some details will be missed and as a result significant re-work will be required. So in general iteration or sprint should be frozen.

In Kanban if you have an urgent request to implement or a really important user story, you can just put it on top of the queue. It will be taken as soon as a free slot is available. It is simply a Product Owner’s dream :)

#3. No need in iterations

Why you need iterations? Initially iterations help to reveal real problems in the development process quickly. Further on they establish a project rhythm and rituals. However at some point in project I think you no longer need them.

My vision is that you need to iterate first, then flow. Our backlog is fuzzy now, plans change often. We learn how to do agile in general and iterations are not helpful there anymore. Without iterations there is no need in iteration planning and iteration demo meetings. They are waste. Instead we have more smaller just-in-time meetings with people in a feature-team before starting the development for each user story.  It works perfect.

Some people are missing the rhythm, but I think it is more to habit. Kanban establishes more complex rhythm and it may take time for development team to recognize it. Still stable rhythm may be the only reason to keep iterations in the late phase of product development, in my opinion.

#4. No need in estimates

Why you need estimates? In iterative development you need them to predict how many stories you may take in the next iteration. You may even predict a release date based on estimated backlog and iteration velocity. The other reason is that PO wants to know how big the user story is. If it is big enough, PO  may consider moving it to the next release. If it is small, PO may decide to put it into the next iteration.

Obviously, iterative development is hardly possible without estimates. But if you drop iterations, it is not a problem anymore. Can Product Owner live without estimates? Well, yes. In our company we don’t estimate user stories. Why not? Because I as a PO don’t need them and we don’t use iterations. All I may ask is a very rough estimate like  normal, large, very large.

In our situation (I stress that, in our situation), estimates are waste. We don’t spend time on estimates. We have a prioritized backlog and just take the most important user story and implement it. It may not work in some conditions, but we have quite a mature product and hundreds requests from customers. There is no clean field development, so normally we don’t have any defined release date. We release when it is ready (I understand that many companies can’t do that, but we have this favor).

#5. Perfect flow visualization

Kanban Board provides a very clear view on current work in progress. It visualizes flow and enables fast planning and tracking. It is really a great tool, we love it.

Kanban is becoming a popular agile tool. Indeed it is very good for software projects of certain types. However, there is a danger of false reasons behind Kanban adoption.

#1. User Stories Diversity

“Our stories vary in size a lot from 1 point to 40 points. Large stories just do not fit into an iteration”

It is very easy to say you can’t split user stories, thus iterations should be abandoned. An easy solution is not the best one. There is something important behind the fact you can’t split stories. Most likely you don’t know how to split stories right. It is quite hard from the beginning and demands creativity.

Moreover, according to Queueing Theory, it is better to have small stories with roughly equal size. Small batches with equal size improve flow (and you have a better and more predictive cycle time). So in Kanban you still have to split stories to smaller stories and try to make them equal size.

#2. Failed Iterations

“We can’t complete most stories in a single iteration”

There may be many, many reasons behind. Mini-waterfall approach, velocity greed, large stories, manual testing, poor stories description, etc. You may even have a wrong iteration length. For example, in large projects 1 week iteration may be an overhead with a huge transaction cost. So 1 month iteration may be much better in this case.

You need to analyze true reasons, the roots of the problem.

#3. Failed Retrospective Meetings

“Retrospective meetings are waste, they do not help in process improvement and we want to remove them”

You just don’t do these meetings right. One popular failure is no Action Items after the meeting. Without action items a Retrospective Meeting is just a kind of informal chat that you may have over a glass of beer. You meet, chat about your problems, and get back on the same track next day. Rule #1 is to collect and write the list of Action Items that you will try to accomplish during the next Sprint [or any other time box].

One more  common pitfall is to skip action items execution. You may collect them, but not really try. You may even try them, but abandon too quickly. Almost all new rules or practices put you off the comfort zone. It takes time to learn them, use them and like them (or dislike), but it should be an expert opinion, not just a gut feeling.

If you expect that you will replace failed retrospective meetings with a nice and simple “stop the line” rule, you’ll fail, since it is even harder than scheduled regular retrospectives and demands even higher level of self-discipline.

#4. Shared People / Functional Departments

“We have a single pool of developers and share them between projects. We can’t form stable project teams”

Do you really think that Kanban will solve your social and organizational problems? C’mon! It helps to visualize flow and find bottlenecks, but if you don’t have a cross-functional team — you have hard times. It is proven in so many sources and researches that cross-functional teams perform better, produce better results and better software.

If you are experiencing difficulties planning sprints with shared pool of developers, try to fix the root of the problem first – switch to cross-functional teams and eliminate multi-tasking.

#5. Simplicity

“Kanban is so simple! No plans, no estimations, no iterations, no overhead”

Indeed Kanban looks simple. But it provides nothing interesting by itself. To ensure a successful Kanban adoption, you need to apply Lean principles first. This new buzzword may sound like a silver bullet, but obviously it  is not. Hard work, discipline, target on perfection and constant improvements – all that is required to apply any agile methodology.

Don’t get me wrong. I am not saying that Kanban is bad. We use it within TargetProcess and I personally believe it works way better than iterative development (for us).  I just want to make a point that you’d hardly resolve all your underlying problems as you switch to Kanban.

P.S. I usually don’t read posts like “10 reasons to …” Can’t believe I wrote such post myself! :)
P.P.S. Read next post 5 right reasons to apply Kanban.

• What is Cadence? Great post explaining cadence using nice pictures. “The purpose of a cadence is to establish a reliable and dependable capability which demonstrates a predictable capacity. Cadence gives some confidence in the upcoming work when we are triggering rather than scheduling work.”
• Series of amazing Danilo Sato posts about Velocity anti-patterns: Done is not done, Making up points, Used as a performance measure.
• Alan Shalloway has interesting and thought provoking post “Is Part of the Agile Community Acting Like the Waterfall Community of Old?” “The pattern is unfortunate – it seems that there are those who don’t like discussing some thing have taken one of two approaches. If they can intimidate people who are using Scrum Alliance certifications for their livelihood, they do so. If they can’t, they attack the integrity of those people they don’t like (e.g., David and myself and others I haven’t mentioned). “

There are several known areas where Kanban software development works fine: maintenance projects, game development, multi-media projects. There is no clear data whether it is good to use Kanban for product development or not.

I have my own vision of Kanban role in software development projects.  I think that

When you start a new product, you should use iterations first, but then may switch to pure flow development using Kanban.

Reasons are quite simple. Initially you do not have any burden. You have quite a large set of features that you want to push into Release #1. Definitely it’s better to have a small backlog, but it depends, you know. Sometimes  a “minimal marketable product” still has quite many features. So you have many user stories to implement, you do not have any bugs from production release, you do not have change requests, etc. It means you have a pretty stable environment with a quiet and stable backlog. And it means you don’t need to branch features and be able to release anytime. You may define clear goals for your iterations, estimate stories, use velocity and predict Release #1 date with acceptable accuracy.

However, things change with time. Suddenly you have large customers base, you have many enhancements, you have urgent bugs to fix and release. Now the environment is not so stable, you have unexpected events in the development process. Then I think it’s a good time to switch to Kanban. Urgent bugs may be fixed and released in 1-2 days, enhancements may be planned just in time with good release date prediction, velocity is no longer critical, since you do not need to predict large release dates, cycle time works fine for small features.

We’ve had this experience at TargetProcess. We’ve iterated for over 3 years, but several months ago switched to Kanban. Normally we do small releases each week, but are able to release anytime if required. We do not estimate stories anymore (maybe we will get back to story estimates, but doing without it so far). We are using Git and we’ve got a branch for each feature or bug.

Ironically, we stopped using some features in TargetProcess like release/iteration planning and progress tracking ourselves. It was a clear sign that Kanban support should be added to the tool. And it is on the way.

Mind maps are cool. They provide a very good overview of quite complex structures and things. You may quickly scan the mind map and find interesting relations. Also it helps to create/maintain good picture of the process/structure in your head.

Agile development processes are good candidates for mind maps.

Scrum

This mind map represents Scrum checklist (PDF). Looks neat and focuses on right things like Definition of Done, correct roles definition,etc. In short it is a mind map of best practices in Scrum.

More formal mind map (PDF) of Scrum process. Plenty of details and good nodes separation. And finally a small and funny Scrum mind map.

Extreme Programming

Two very similar mind maps of Extreme Programming, but with different details.

Lean

Small Lean mind map. Interestingly, I did not find any Kanban mind map. Seems like it is time to create one :)

Lean and Kanban software development adoption is growing. More and more companies setup Kanban Boards, limit WIP and eliminate Muda.

This collection of links will help you understand all that buzz around Lean/Kanban and decide whether it is worth trying. I’ve read all the articles and posts below, so this list is a truly selected thing ;).

Articles and Blog Posts

• Lean Software Development. Wikipedia summary about lean software development. It is a good start to digg into the topic (as usual).
• Kanban Development Oversimplified. Most likely the best article to start with Kanban. Very clear, very detailed. Good work!
• Kanban, Flow and Cadence. This blog post with many nice pictures describes three important properties of Lean: Kanban – Controlled Work, Flow – Effective Work, Cadence – Reliable Work.
• Scrum-ban. Interesting attempt to mix Scrum and Kanban, taking the best from both worlds. Kanban with iterations is possible.
• Beyond Scrum: Lean and Kanban for Game Developers. Article describes real Lean/Kanban implementation for game development industry. The section on how to improve The Flow (3 strategies: Time-boxing, Levelling workflow, Reduce waste) is especially good.
• Adventures In Lean. Series of posts about Lean approach with focus on real problems solving (handling bugs and emergency fixes in Kanban, setup pipeline, bottlenecks, etc.).
• Lean and Kanban. Several posts on the topics in this blog.

Presentations

• Kanban for Software Engineering . Detailed presentation about Lean and Kanban. Focuses on many problems and solutions in a software development process.
• Kanban: A Process Tool . Another good presentation that emphasizes the simplest rule in kanban: Limit the number of things in work to a fixed number. Maybe a good start indeed.
• Practical Experiences and Tools Applied to a Kanban Sustaining Engineering System . Fantastic presentation about Lean/Kanban implementation with real examples by Alisson Vale.
• Kanban vs. Scrum. Rich and lengthy presentation that compares Scrum and Kanban. A very good visualization (which is a key in Kanban :). Shows why Kanban fits better for multiple development teams working on the same project. I tend to share this opinion as well.
• A Kanban System for Sustaining Engineering on Software Systems . Very interesting presentation of Lean implementation at Corbis. As I know, it is one of the first successful attempts to run Kanban in a software shop. Must read.

Lean/Kanban Blogs

• Agile Management Blog. Lots of interesting posts from David J. Anderson (well known engine of Lean software development :)
• Richard Durnall Blog. Pull and Push systems, interviews, lean roots and principles. Nice reading with hand-drawn diagrams.
• Lean Software Engineering. Corey Ladas and Bernie Thompson are blogging about Lean, Scrumban and Kanban, Theory of Constraints, software development and other topics you did not even hear about.
• AvailAgility. Karl Scotland’s posts are very interesting (and helpful) to read. Isn’t Kanban just a Task-board? Check the blog to get an answer.
• The Agile Executive. Many insights into Kanban and summaries from the first lean conference.
• Software Just in Time. Lean concepts and real lean applications posts by Alisson Vale.

Lean/Kanban People in Twitter

• David J. Anderson. Lean/Kanban software development pioneer.
• Corey Ladas. Product development methodologist. Author of Scrum-ban book.
• Henrik Kniberg. Optimize, debug & refactor IT companies. Author of Scrum vs. Kanban presentation (which is very good!)
• Karl Scotland. Agile Coach. He runs AvailAgility blog with great insights into Lean and Kanban.
• Rob Lally. Renaissance Technologist.
• Alisson Vale. Alisson implemented outstanding Kanban process in his company.

Tools

There are just several Kanban tools on the market. To be honest, I don’t like TRICHORD UI. LeanKit: Kanban looks much better, but it can work for small teams only on my opinion. Anyway, it seems Kanban tools vendors’ race just began.

If you know other tools that support Kanban, drop a comment and I’ll happily include them into the list.

• TargetProcess. Customizable Kanban Board and other vanilla.
• Zen. Good tool for small teams.
• LeanKit: Kanban. In beta so far, but looks quite neat. Maybe useful for small teams.
• TRICHORD. Desktop project management application with Kanban boards.
• Radtrack. Registration does not work, but I found the screenshot via Google. Looks like LeanKit so far.

Did I miss something interesting? Drop a comment!