Edge of Chaos | Agile Development Blog

Have you ever asked the question, ‘When are we planning to do bug analysis?’ in a Sprint retrospective? This is one of those questions that is thought about but not asked during the initial iterations of Agile projects. Also, unfortunately, this question is forgotten over several subsequent iterations and asked during the end of projects when it becomes too late. By ‘bug analysis’ I mean the analysis of a group of bugs to study factors such as distribution of defects on priority, severity or application module as well as other aspects such as reproducibility or dependencies.

Several years ago, I became cognizant of asking this question. It was never an ideal situation for us to deliver bug free production ready software at the end of iteration. I am sure, even these days delivering production ready software at the end of iteration is not very common across projects. It is a tough goal. If you have achieved this goal, I should say you belong to a high performance team. In all general circumstances, bugs do accumulate, get prioritized, and fixed. Is this an adequate approach to execute projects? Don’t we need to identify the right time to initiate bug analysis and have a consensus on the frequency of performing bug analysis?

The first time when I asked this question the answer was, ‘The product is not stable enough to perform bug analysis.’ That took me by surprise and triggered additional thoughts. I asked further, ‘Should we wait for the product to become stable to do bug analysis? Or should we perform bug analysis in order to find ways to make the product stable? Also, shouldn’t we use visual boards to display bug trends and analysis so that the team understand not only the progress of user stories but also trends on product quality?.’ That’s when our approach matured to the next level.

Essentially, it makes sense to have a sizeable number of bugs to perform bug analysis and it depends on your project. Also, how detailed you perform bug analysis depends on your project as well. Bug analysis can reveal variety of useful information such as distribution of bugs across modules, distribution in terms of bug priority or severity, distribution of bugs in terms of bug status etc., The earlier we analyze and understand these parameters, the better it is for us to improve product quality.

Thinking of bug analysis late in the game seldom yields efficient measures to improve product quality. Next time when you talk to your Agile team try to trigger this question. Am sure you will come across interesting responses.

All said and done, bug analysis is a practice that can yield results in Agile projects. Do you use a visual board that displays bug analysis results and trends in your work area? What has been your experience in performing bug analysis? How frequently you do it? Is it beneficial?

Raja Bavani is Technical Director of MindTree’s Product Engineering Services (PES) group and plays the role of Product Engineering Evangelist and Agile Coach. Check his Product Engineering blog.

We are using Kanban for product development. It works great. As a Product Owner I can re-prioritize backlog anytime and put things into Planned state when I want to. It works. But. Every Product Owner is passionate about the product. He wants to improve so many things and do that RIGHT NOW. Prioritization is really hard and Planned state has a strong tendency for saturation and overflow. For example, our Planned state has limit of 20 items. In the worst case there were 50 items in it (most of them are bugs and small enhancements). Mea Culpa.

The obvious side effect is that most items are buried in Planned state for weeks.  A more important user story pops out and a small enhancement moves down. As a result, enhancements and bugs were never fixed/implemented. Planned column turned into Backlog column someday. That was a real problem. Yes, bugs were small and quite easy to fix, but more important problems constantly pushed them back.

We got tired of this and decided to try a new practice: The Ultimate Clean Up Day. The rules are quite simple:

1. From the start of the workday everybody who can do programming (including CEO :) take any bug/small story from Planned state and fix it. Repeat till the end of the day.
2. Someone suggested to draw a star on the whiteboard for each fixed bug and find out who is the best bug fixer.
3. Testers verify fixed bugs as soon as possible and cooperate with developers to ensure the fix. Repeat till the end of the day.

We were afraid that the clean up day will take longer than 1 day due to re-opened bugs and so on, but this did not happen. It was a very focused activity resulting in 23 fixed bugs in a single day. It was fun and development team decided to do this monthly. Next clean up day will be in the middle of December and I will re-fresh my .NET skills again :)

I think unclear specifications is #1 source of bugs. If you deliver something that was not intended, there will be many complaints from customers. In customer’s opinion almost all these complaints are bugs. Also, without instant communication it is very easy to miss important details and release an un-polished user story with a number of small glitches.

Agile development is very rapid. In many cases the path from idea to user story in progress is extremely short, it may be even 1 day. With this hard pressure on,  it is very easy to make errors in user story description. Product Owner may miss some important details and as a result the story will do something wrong, something that was not expected.

This is the first question you must ask… is it a bug, or is it a change? I’ve seen a lot of bugs that have come up that were “We asked you for x, never thinking about y, so could you please change the system so that y is covered?” It’s a business scenario, so there’s no reason to expect a developer or tester to anticipate/test it. I know where we’ve struggled is in injecting items into the product backlog, we tend to classify it as a bug and so we end up with a lot of bugs, but not a lot of change in the back log. That’s exactly what I think we’re supposed to be avoiding in Scrum or any other agile methodology. As we see change, we should be taking it on as a feature and prioritizing it accordingly. Only when it’s something broken should it be called a bug.— Jim Leonardo

The best solution to this problem is communication.

Clarification meeting

The important thing in software development is to put everyone on the same page. Developer, Tester and Product Owner should run a small meeting about user story, thus forming a mini-team. If a user story has several developers and several testers, all should participate.

The goals of the meeting are:

• Ensure that everyone understands what should be implemented.
• Brainstorm acceptance tests/test cases and create a check list.
• Reveal and eliminate all discrepancies in story description/specification.

We have these meetings at TargetProcess and they are very efficient and helpful. Recently there have been some stories  with no meetings, and as a result the stories had more defects and a longer cycle time. Side effect of the meeting is that Product Owner receives less questions about functionality later, thus having less interruptions (which is good).

Stories should be ready before iteration start

It is a good idea to have all user stories described with good details before iteration start. Thus testers and developers will have time to review them and prepare questions to the iteration meeting. It may trigger a much better discussion and reveal some interesting problems that were unclear initially.

There is a danger to document more user stories than required, and it will be a form of a waste. Product Owner should maintain good balance here.

Are you familiar with zero defects mentality? It looks very good from the first sight. Zero defects… Let me think… Well, cool! I like it! I’d like to have zero defects in my projects.

So, what is zero defects mentality? Here is the quote from Lean Software Development an Agile Toolkit book:

“One of the fastest ways to kill motivation is what is called in the US Army a zero defects mentality. A zero defects mentality is an atmosphere that tolerates absolutely no mistakes; perfection is required down to the smallest detail. The army considers a zero defects mentality to be a serious leadership problem, because it kills the initiative necessary for success on a battlefield” — Mary & Tom Poppendieck

Obviously, zero defects mentality is not something that HOUSE M.D. likes ;) Moreover, I think Dr. House hates zero defects mentality. It causes several unpleasant effects:

• Not enough courage to refactor complex, messy, buggy, but important piece of code.
• Can’t make important decision, instead make less risky, but wrong decision.
• Do everything to avoid responsibility, that leads to coward and stupid behavior.

“Zero defects” may sound good. But in reality you still have errors after production. Even in predictable and (quite) easily testable industries (hardware, automobile, etc.) there are problems with 100,000 power adapters that should be replaced (or hard drive problems, or engine problems, I bet you can continue the list).

How can we expect zero defects in software development? It is harder to test, harder to define in details, harder to predict. Software development is a complex adaptive system, we can’t predict all effects. Bugs in production is a normal thing, and by “normal” I mean we can’t bring them to zero. We can (and should) minimize them using all possible ways, but The Last Bug is a mirage.

There are several obvious strategies that may help:

• Test Driven Development. Nice side effect of TDD is a unit tests suite. You have tests for new code, and you have unit tests for old code. More tests — less bugs.
• Continuous integration. Instant feedback helps to identify problems early and fix them early. It saves time (and money) and reduces bugs in production.
• Automated regression functional tests suite. Unit tests are good, but you need something else. Functional tests emulates user behavior and find user interface errors, integration errors, etc. Needles to say you should have continuous integration in place to succeed with automated functional tests.
• Root cause analysis. There are several ways to fix a bug. You may just hack the code and apply a patch (please, don’t do it at home!). You may think deeper and fix the bug nicely. Also you may look into the root of the problem and fix it. Yes, it may take time, but you will prevent future bugs from this domain.
• High development skills. Ironically, high skills do not always reduce bugs rate in production. “Stars” may be strong in some areas (business layer), but may not polish things. It may lead to many small bugs, especially on UI.

Is it a good thing to have “Zero Defects” goal in the sprint? The answer is “Are you kidding me? Zero bugs? It’s a miracle!”

Agile development books often skip bug management practice. However, there are several dangerous misunderstandings (anti-patterns) that are quite common. It is better to recognize the problem early and fight to win. Here are three common bug management anti-patterns in a development process.

Bug fixing phase inside iteration

Symptom: You have several days at the end of  iteration fully dedicated to  bug fixing.

If you have a bug fixing phase inside an iteration, you have mini-waterfall in each iteration, which is not a very good idea (in agile we blame waterfall for good reason, don’t we?). Story should be tested as soon as it is completed.

Bug fixing iteration

Symptom: You run several iterations with almost no or little bug fixing and then one (or even two!) iterations fully dedicated to bug fixing.

Bug fixing iterations kill your iterative development. It is even worse than mini-waterfall. Such iterations increase your work in progress dramatically. You have nothing done during several iterations before bug fixing iteration. You have nothing to ship. Moreover, bug fixing iterations reduce motivation, people don’t like to fix bugs for 2 weeks or for the whole month.

No “Zero open bugs” rule in user story’s Definition of Done

Symptom: You have open bugs in a completed iteration.

You may have decided that some bugs discovered during story development are unimportant and can be postponed to  future releases/iterations. This is a very dangerous practice that leads to bugs accumulation. Moreover, it may be a reason for Bug fixing iteration (see above). The best strategy is to have “Zero open bugs” rule in Definition of Done for user story.