Edge of Chaos | Agile Development Blog

Almost 2 years ago I published the Flow. Discover Problems and Waste in Kanban post. The idea was quite simple: visualize the flow of a single user story or bug, and track their life cycle to Done:

You can spot such problems like delays and re-work very fast this way:

Now we’ve brought this idea to life. The Flow chart for every user story, bug and feature will be available shortly in TargetProcess v2.22.9.

The chart gives answers to a whole lot of hands-on questions:

• For how many days has this User Story been in this particular state?
• Were there any delays?
• Was there any re-work?
• Who was responsible for the User Story?
• When were Bugs and Tasks added and closed?
• How much time was spent each day?
• Were there any impediments?

Let’s look at some examples. The user story on the chart below has been  in Open state for 25 days (it means, in the Backlog). Then it jumped right into In Progress state. Two developers (Alla and Vladimir) started working on it (so it was pair programming). They’d been working for 3 days and then the story was moved into Re-open state. This is quite surprising, most likely they had to switch to something else (no good). Then they got back and spent 15 days working on this user story. That’s way too long. Most likely there were switches as well,  so this should be investigated.

Starting from Oct-18 the progress was very good: development went smooth, tester found several bugs and they were fixed in 2-3 days. Finally, the user story was released to production with no more delays.

You immediately get a high-level overview: delays and up/down state transitions. It is a clear sign of some problems, the systematic ones or not known so far, but we already have some background info to start an investigation).

Let’s check another example. It looks like the user story on the chart below was taken to development right as it was added. That’s true in fact, since it was a customer request to which we reacted immediately. It was implemented in a single day, and there was a small delay before tester took it to the  testing phase. We found quite many bugs and fixed them in 2 days, everything is fine. But then the completed user story was hanging in Release Branch state for 11 days, and that’s no good.

We’re planning to extend this Flow Chart and put more information there (comments, attachments, etc.) The goal is to provide the complete production timeline uncovering hidden malignant patterns and problems. You should be able to get a high-level overview in an instant and dig into as many details as possible.

Here is the snapshot of our real TargetProcess production Kanban Board taken today (November 18). You can see what we are working on right now, what is in progress, what is in testing and what will be released soon.

1. You don’t waste time on estimation

Estimation takes time. Even if you do planning poker and use story points, it still takes time. What do you do to improve estimation accuracy? You gather some data, analyze the data and discuss the results. You are spending time on all that. But are you sure that estimates really provide any value? Most likely not. It is a waste. You’d better spend this time doing something important for the product.

2. You shouldn’t explain to higher managers why it took soooo loooooooong

If you don’t have estimates, you can speak without fear and explain things clearly. You can enumerate problems and explain how they were resolved. You can show that you work really hard and take all the necessary actions to release this story as soon as possible with great quality.

If you have estimates, be ready to hear something like “you screwed up, maaan! You had to deliver this feature till the end of the month! What the f%$k is going on?” as an argument. The team put themselves on a weak side immediately and have to care about deadlines more, not about quality or a better solution. Is it something you really need?

3. You don’t give promises that are hard to keep

You are not relying on people’s optimism (which is built in). People are optimists (almost all of them) and inevitably give optimistic estimates. You can use  complex formulas or very simple rules to have better estimates, but is it really worth it? You can spend many hours defining correct formula for exactly this development team. People will not trust you, since you (rightfully) don’t believe in their estimates. Numbers will look “made up” and still you will have incorrect estimates in the end.

4. You don’t put additional pressure on development team

In some teams Estimate is equal to Deadline. That’s bad. What do you do to meet the estimate? Compromise quality? Tolerate average architectural solution? Abandon polishing? All that leads to poor solutions that end users will not like. It is better (almost always) to spend more time, than planned, and release something really usable and useful.

5. You focus on really important things

Imagine, you have a planning poker session and estimate all stories inside an epic. You sum up all the stories and define it as a 1000 pts epic. You just stick a “fear” label on the epic. People do fear large problems. A 1000pt problem is huge and psychologically it is harder to decide “let’s start it right now”. Product Owner will have a temptation to implement 100 smaller user stories instead of this huge epic. However, it may be a bad decision. This epic may be really important, the morst important thing for the product.

If you don’t estimate it and are not completely sure about its size, you have a better chance to actually start doing it.

Although he hadn’t heard of the TargetProcess contest beforehand, Robert just had to win the iPad for his autistic daughter. We never imagined that a simple contest on our Facebook page would bring such joy to someone — but we are delighted and touched that it did.

Robert J. Walker from Mindshare Technologies happened to be making some adjustments to their Kanban board, when a colleague of his suggested that he submit his board to the competition. ‘What competition?’ wondered Robert, but went ahead and submitted it anyway — and won. What of the iPad?

TargetProcess: How do you use the board?

Robert: At my company, we wanted to make use of a kanban board as a passive “information radiator,” something we could have up somewhere conspicuous, allowing anyone to walk by and see the status of the user stories and bugs we were working on. Our first pass on this was using a whiteboard and sticky notes. This fulfilled its purpose, but it meant that Jon Grover, our project manager, had to maintain the same items in both TargetProcess and on the whiteboard, which felt like a pretty significant waste of effort for someone who already had plenty to do. TargetProcess comes with a kanban page, of course, but its design is geared more for interactive work at a computer, and not so much as an information radiator.

Because I found the project interesting, I spent some free time writing what eventually became our kanban board. It periodically queries TargetProcess for the data, so Jon doesn’t have to worry about keeping a separate board up-to-date. Assignable items are shown as little numbered “tickets,” blue for user stories and red for bugs. Effort for those items is shown as a colored “tail” on the item; the longer the tail, the more effort that item has. The boards columns represent phases in the development process, and the effort in each phase is subtotaled at the top. This view is great for seeing the big picture across the entire development department, especially for seeing where bottlenecks are occurring.

It can also be filtered down to a particular iteration or team. In this view, there are fewer items on the screen, so the board allows you to expand the tickets to display the description of each item as well to make better use of the available space. This is great for our individual team stand-up meetings. You can also click on a ticket to view more details, jump directly to that item in TargetProcess, or change its state. We have a new development area with a touch screen in the meeting room, and with a few tweaks I was able to make the board work very nicely with a touch interface.

TargetProcess: How you learned about the contest?

Robert: If I remember right, I had made a couple of changes to the kanban board and was showing them to Jon. He said, “You really should enter this into TargetProcess’s contest.” I didn’t know about the contest, or even that TargetProcess was on Facebook! He filled me in, and he and others on our dev team urged me to enter.

TargetProcess: How do you feel about winning?

Robert: I have a 5-year-old daughter with autism. She’s very bright and my wife has worked very hard with her to help her improve, but she still struggles with communication and social problems. The iPad has been in the news a lot lately as being a remarkable tool in autism therapy. When I heard that the prize for this contest was an iPad, I knew I had to try and win the contest for my little girl. I’m so excited to be able to give this to her and am really looking forward to seeing the things that she can do with it.

One last thing:
Robert does have one request though, one which we are more than happy to help with – April 2 is the fourth annual World Autism Awareness Day. In commemoration, prominent buildings all over the world will be lit in blue light and many people will wear blue to raise awareness of autism and the struggles of those who have it. You can learn more about the event at their web site: http://www.lightitupblue.org.

Once again, thanks to Robert for sending this in, congratulations on winning, and we would love to hear how the iPad works out for you!

We are using Kanban for product development. It works great. As a Product Owner I can re-prioritize backlog anytime and put things into Planned state when I want to. It works. But. Every Product Owner is passionate about the product. He wants to improve so many things and do that RIGHT NOW. Prioritization is really hard and Planned state has a strong tendency for saturation and overflow. For example, our Planned state has limit of 20 items. In the worst case there were 50 items in it (most of them are bugs and small enhancements). Mea Culpa.

The obvious side effect is that most items are buried in Planned state for weeks.  A more important user story pops out and a small enhancement moves down. As a result, enhancements and bugs were never fixed/implemented. Planned column turned into Backlog column someday. That was a real problem. Yes, bugs were small and quite easy to fix, but more important problems constantly pushed them back.

We got tired of this and decided to try a new practice: The Ultimate Clean Up Day. The rules are quite simple:

1. From the start of the workday everybody who can do programming (including CEO :) take any bug/small story from Planned state and fix it. Repeat till the end of the day.
2. Someone suggested to draw a star on the whiteboard for each fixed bug and find out who is the best bug fixer.
3. Testers verify fixed bugs as soon as possible and cooperate with developers to ensure the fix. Repeat till the end of the day.

We were afraid that the clean up day will take longer than 1 day due to re-opened bugs and so on, but this did not happen. It was a very focused activity resulting in 23 fixed bugs in a single day. It was fun and development team decided to do this monthly. Next clean up day will be in the middle of December and I will re-fresh my .NET skills again :)

There are various ways to support agile team retrospectives. We’ve used all of them, so let me share our experience.

Cadence (usual retrospectives)

If you have iterative development like Scrum or XP, it is very convenient to run usual retrospectives meetings. For example, with 2-week iterations you have such meetings every other week, discuss issues, what worked, what not, brainstorm solutions and new things. We’ve tried mood boards, various formats for issues gathering and for action items tracking. We’ve tried a whole lot of things in 2 years. In general, it worked. But then we switched to Kanban and somehow retrospective meetings faded out…

Is there a better way to improve development process?

Stop-the-Line

We tried to apply stop-the-line practice. It states that as a mistake or malpractice is discovered all responsible people should immediately hold a meeting to resolve/prevent this specific problem. There were several stop-the-line events, but this practice did not survive. Why? There are two main reasons.

1. It is very disruptive. People are working on various unrelated tasks and are in the flow. Suddenly they should switch to a problem resolution brainstorming. Many people don’t like to do that.
2. It may take too long. Sometimes the problem is very hard and it takes much time to find a good solution. People impatiently drink coffee and want to get back to work.

So we dropped stop-the-line practice and replaced it with a pure pull system.

Pull / Issues Board

Issue Board is a very simple concept with 3 basic rules:

1. Every person in a development team can write a problem or a new idea he wants to discuss on a whiteboard (Issues Board).
2. There is a limit of 3 problems on the board.
3. When there are 3 problems on the board, we have a retrospective meeting right after a daily meeting.

We have been using this approach over the last several months and I like it most. It leaves off some problems both of the stop-the-line approach and the cadence approach. First, if there are no issues or ideas, there’s no need to have a meeting :) Second, no interruptions, since daily meeting is interruptive by itself, so it is just natural to have a retrospective meeting right away.

If you have other approaches to retrospectives, go ahead and share them!

Many interesting posts and discussions about Scrum and Kanban published last weeks. Someone even called this a war :) I don’t think it is a war, but some posts indeed may drive such feeling.

• Ken Schwaber bashed Kanban. “I was told that Kanban is frequently used when an organization cannot readily adopt Scrum. Many of Scrum most difficult aspects are then sidestepped. Managers are still in charge of telling people what to do. People can be interrupted at any time. People are still work in functional silos, preserving the jobs of functional managers. People are not allowed to work in containers, sharing skills and knowledge to bring complexity into solutions – instead they are worked on a pull (more sophisticated than push) production line.”
• Karl Scotland discusses Scrum and Kanban difference as Intentional vs. Implementational approaches. Interesting perspective in fact. Karl thinks that Kanban can be used with Scrum to reveal even more problems in development process.
• David J. Anderson posted the most rational article about Scrum and Kanban difference. I like it a lot. Some phrases are true gems: “Kanban is not a project management or software development lifecycle method. It is an approach to change management – a framework for catalyzing change in an organization.” and “Kanban uses a WIP limit as a change agent and Scrum uses commitments. This is a fundamental difference in approach.” David also posted some reflections later with interesting thoughts about anarchy and science.
• A year ago Tobias Mayer didn’t believe that it is possible to use Kanban at all. “I fundamentally disbelieve that there is any such thing as a “value stream” when you are working in a complex environment, in a creative process, building new products or generating new ideas.”

I think Ken is wrong. His arguments against Lean and Kanban quite ridiculous. Sure there is no intention to treat software development as a factory and apply lean manufacturing principles blindly. Of course there are people who will try (or already tried) that and fail. But vast majority of lean community in software development do understand the difference and working on concrete practices. Lean philosophy is great, but tools and practices can’t be taken from manufacturing directly.

Pull system is more complex than Push? C’mon!

David has wise arguments and perfect position. I think Kanban will be more popular than Scrum in long-term perspective, but it will take time. Visualization is a key thing to manage complexity, and software development is a complex system.

We live in the age of added value. It’s everywhere. Value-added services, value-added products, value-added goods etc. etc. etc.  Actually, so much value has been pumped up in our life, that it’s even strange that this value is not protruding from us like clothes from an overly packed vacation suitcase.

If we take a closer look at the back side of added value, a huge surprise is waiting there. The example I find most notorious is cell phones. What if I want a simplistic cell phone with NO Internet, no camera, even no voice mail, just live calls and SMS?  You’ll never ever find such a phone.

I bet that a phone manufacturer who stops the rush for more new features, would make a fortune in an instant selling the “new frugal” cell phones. In this case, the added value is content which comes from Internet, capabilities to exchange content. Why should someone want a phone without Internet?  My word, very soon we will see such phones on the market. The niche for them is already there. Here’s why:

More content and more various channels to produce and exchange content is now commonly presented as added-value. Hence, a communication device which happens to be a humble phone is supposed to deliver this value. But as we’re oversaturated with content, no buzz is a huge deal. The luxury of focusing on one thing at a time is something that only a few can afford. Besides, it’d be very frugal (frugal is actually the new buzzword :) to buy a phone for a reasonable price, and sell a used iPhone to some geek. Oh, pardon me. It’s all about iPads now :)

There’re plenty of such examples. Another one: the added value of having a car means lack of natural movement, the necessity to pay for gym etc. It actually brings along the whole array of more added value  goods and services that turn out to be not of added value but of less value, since you pay for what you could do naturally.

Take organic products. Now they’re added value. 100 years ago who could have thought that something natural adds value? Now it’s a rollback. What is simple and natural costs more, but has less value compared to the original added value concept for the matter, and the cycle goes on and on.

And we’re nailing it down to our favorite: project management tools. Versatility and too many features now have bounced back to the simplistic Kanban board.

It looks like it’s time to not only practice lean production, but to produce lean products. Gaining focus requires focused tools, one way or another.

Cumulative Flow diagram is a very good starting point for stop-the-line or retrospective meeting. Here is a real example from TargetProcess development.

You can see in the chart that we had a bottleneck in the beginning of December. It was caused by a quite complex user story. In theory a single user story should not affect cycle time significantly and should not create bottlenecks, but if you violate some rules it might be the case.

The user story was about replacing jQuery javascript framework with ExtJS framework. It took 1 month to implement by 1 developer. During this month we’ve made several releases and all went smooth. Then this user story was verified by testers and all existing acceptance tests were passed. So we’ve made a decision to merge this story to the main code line.

Unfortunately, after the merge quite many bugs were found in the build during smoke testing. These bugs took more than a week to fix, and during this time we were unable to release anything, since the merge was already done and the rollback was quite complex as well.

The lesson learned is to put more effort into testing of user stories that are more complex in nature. This particular story affected many places in the application and usual smoke testing was not enough. So we are going to introduce a new class of service (let’s call it  “technically complex story” so far :) which would mean more in-depth testing and verification before the merge.

In general, Cumulative Flow chart is a valuable tool to analyze historical data, but for emergency bottleneck identification Kanban Board is even better. If you have limits on Kanban Board, you see problems immediately.