Edge of Chaos | Agile Development Blog

Almost 2 years ago I published the Flow. Discover Problems and Waste in Kanban post. The idea was quite simple: visualize the flow of a single user story or bug, and track their life cycle to Done:

You can spot such problems like delays and re-work very fast this way:

Now we’ve brought this idea to life. The Flow chart for every user story, bug and feature will be available shortly in TargetProcess v2.22.9.

The chart gives answers to a whole lot of hands-on questions:

• For how many days has this User Story been in this particular state?
• Were there any delays?
• Was there any re-work?
• Who was responsible for the User Story?
• When were Bugs and Tasks added and closed?
• How much time was spent each day?
• Were there any impediments?

Let’s look at some examples. The user story on the chart below has been  in Open state for 25 days (it means, in the Backlog). Then it jumped right into In Progress state. Two developers (Alla and Vladimir) started working on it (so it was pair programming). They’d been working for 3 days and then the story was moved into Re-open state. This is quite surprising, most likely they had to switch to something else (no good). Then they got back and spent 15 days working on this user story. That’s way too long. Most likely there were switches as well,  so this should be investigated.

Starting from Oct-18 the progress was very good: development went smooth, tester found several bugs and they were fixed in 2-3 days. Finally, the user story was released to production with no more delays.

You immediately get a high-level overview: delays and up/down state transitions. It is a clear sign of some problems, the systematic ones or not known so far, but we already have some background info to start an investigation).

Let’s check another example. It looks like the user story on the chart below was taken to development right as it was added. That’s true in fact, since it was a customer request to which we reacted immediately. It was implemented in a single day, and there was a small delay before tester took it to the  testing phase. We found quite many bugs and fixed them in 2 days, everything is fine. But then the completed user story was hanging in Release Branch state for 11 days, and that’s no good.

We’re planning to extend this Flow Chart and put more information there (comments, attachments, etc.) The goal is to provide the complete production timeline uncovering hidden malignant patterns and problems. You should be able to get a high-level overview in an instant and dig into as many details as possible.

Humans receive 95% of information through visual perception. We spend countless hours staring into laptops reading, analyzing, interpreting and feeling information flows. I think visualization is something we lack in many disciplines and project management is not a lucky exception.

Interestingly, latest trends in agile project management do care about visualization. Kanban success heavily depends on great visualization. Everybody loves Task Boards and Burn Down charts in Scrum. As you can see, there is some progress, but I think it is not systematic, just a side effect of other activities. It would be fascinating to create an orthogonal movement in agile project management community that focuses on visualization, I call it Visual Project Management.

There is only one really important rule about visualization:

Visualization should reveal problems, states and trends

Sure, we can add more rules, but they’d be secondary. Imagine you can see a Backlog and diagnose a disease right away:

• too large
• contains too many old user stories and
• grows too quickly

Imagine you see a Board, and at the same instant you grasp all the bottlenecks:

• overloaded people
• problematic stories
• quality problems and
• overall development speed

I can’t provide quick visual solutions right away, but I think this approach will change the way  teams make decisions and improve their development process.

I wonder why we still have no good visualization tools for project management. I think the answer is that project managers are not strong in design, and designers are not strong in project management. Separation of responsibilities leads to basic, trivial decisions. PM should learn design and data visualization techniques to really invent new visualization. Designer should learn PM domain to invent new visualization. The ideal mix is a team where PM knows design and Designer knows project management.

Let’s review some examples to get a basic feeling of how visualization re-define things.

Why Kanban Board is a Great Step Forward

Information presentation really affects the ways we run projects. Here are two screenshots that contain exactly the same information. You can easily say which one is better.

This is a simple list of user stories taken from TargetProcess (I am working at TargetProcess, so forgive me biased screenshots). It contains quite many data about user stories like state, assigned people, effort, etc. However, this data is hidden. To make some decisions, you need to dig  and put some effort into it. It means you have a higher cognitive load. It means you’ll miss some details and sometimes make a wrong decision.

Next screenshot is a simple Kanban Board. While it is not the best example of Kanban Board, still it is good enough to reveal the difference. You see the same user stories on this screen. However, you can quickly identify  there’re too many user stories in In Progress state, but it is not critical, since there are no holes in development flow. You can somehow feel that most likely the team is doing pretty good. In some cases you can’t even quickly explain why you think so, you’ll need to analyze your own feelings to provide logical answers.

But the truth is that you have arrived to some conclusion with enough accuracy, really quickly. That is why visualization is so important in any discipline, including project management.

I believe we can have something much better than a simple Kanban Board. Kanban Board visualizes development flow, but misses other important things like people load, local problems, overall progress for all projects etc.

What happens if we have several teams and want to see their work on Kanban Board? Definitely, the board above will not show that. So far,  I haven’t seen good solutions to such problems. Software products are not good in presenting large amount of data in an interesting, meaningful manner. This should be changed.

Why Gantt Chart is Misused All The Time

Gantt Chart is a great tool as well. It is heavily used in traditional project management, but often with poor results.

Most of the project charts look the same and make the same mistakes: analytically thin, bureaucratic grid prison, not annotated, little quantitative data.
E. Tufte

This Gantt Chart is quite good and useful:

This Gantt Chart is bad and useless:

Stop. Think a little bit. Can you define the difference? Can you say why I’ve made such a conclusion?

First chart shows quite large project phases. Second chart shows very granular tasks. Gantt Chart is not a good tool to handle granularity. It works best to visualize long project phases, like releases or iterations. It just plain fails to visualize 500+ tasks in a project.

Gantt Chart has a large white space, it lacks information density. Thus it should be used carefully to visualize important information, not everything you have.

I think the frustration with Gantt charts arises more because of tool issues. People’s contexts of use may require information that is obscured by a Gantt chart. But Gantt charts are the dominant (sometimes only) visualization in many tools, and it’s difficult to impossible to extract and present the data in other forms.
Brian de Alwis

These were just two examples of visualization in project management. I believe we can be more creative to provide better tools and concepts, invent new ways to present project information and improve transparency. We definitely can do better. And we should.

I will post more about visual project management. This topic is really intriguing and promising to me.

They did it again. Companies are making the same mistake during agile adoption over and over again. We’ve made some of these mistakes as well. If by any chance you see something familiar in the list below, read on and find out why it is a mistake.

1. Start With a Tool

It is not always bad to start with a tool. If you want to dig a pit, you most likely want to find a shovel first. You can dig manually, without a tool, but it will take enormous amount of time.

Agile development is something different. A tool will not provide immediate effect and will not solve most of the problems by the mere fact of its existence. Moreover, you will put effort into tool adoption, shading more important goal — agile adoption.

We encounter this mistake quite often. People come to TargetProcess web site, register for a trial, install the tool and try to use it. They start to ask questions and it’s getting clear that they have no experience in agile development neither agile process established. Sometimes they don’t know what a Burn Down Chart is and how to use it. Sometimes they know nothing about iterative development. It happens. The only piece of advice we can give is to get rid of the tool and dig deeper into agile domain: become familiar with basic concepts and try some process with simplest tools like whiteboard and sticky notes. Then decide whether you need a more sophisticated tool.

2. Start With a Process

Starting with a tool is obviously a bad idea. Most companies start with a process. It is a less serious, but a more common mistake. So you read about Scrum, it looks easy initially. You apply all Scrum practices and after some sprints see that development somewhat improved, but not as drastically as expected. Excitement goes away and process degrades.

What are the reasons? Why has it failed? Most likely people didn’t get the core values of agile development. Process is the mechanics, while values are the core of any agile adoption. The first thing any company should do before trying any process is to focus on agile values such as: communication, collaboration, feedback, trust and passion. It is nearly impossible to apply a good development process if you compromise any of these values. As an agile champion you should enforce these changes. Sometimes these are company-wide changes, sometimes team-wide changes, but still it is absolutely required to apply them.

The fastest way to adopt agile is to start with people and culture. I can’t stress this enough. And it is quite obvious! But why so many companies (including ours) have made this mistake? It is blazingly hard. You will have to change culture, and that will be met with a resistance. You will have to change people and most likely you’ll have to fire some of them. It is a hard and complex process, there are not so many people in the world who know exactly how to apply it. Most of us have no such experience and skills, so we fear this change. We try to start with something easier, something we know. We try to start with a process, with a tool, or with hiring an external consultant (which is good, but not enough). We should fearlessly start with the hardest but the right thing.

Agile manifesto is very deep indeed. If you read it carefully and set the principles as a cornerstone of agile adoption, you will success eventually.

3. Development Practices Are Enough

Extreme Programming is a very cool agile software development process. I think it includes Scrum. Many people may disagree, but in general if you do XP fully, you don’t need Scrum. Interestingly, that developers tend to apply technical practices like TDD, Continuous Integration and even Pair programming, but pay less attention to such things as communication, integral team, having customer on-site and retrospectives.

Why is it like that? Developers are techies, and techies like technical stuff, while often disrespect social stuff. Many of them are introverts, they don’t like meetings, they don’t like to chat with people extensively. They do like to communicate about technical things and do that with passion. However, communication with customer is not about Clojure or fancy lambda expressions, it is about business. It’s business problem domain, and it’s not that interesting for them.

Again, agile is more about people and less about technology. Many practices are focused on people. Pair programming makes for more communications between developers. Customer on-site makes for more communications with end user. Retrospectives boost communication about team, processes, etc. It is absolutely required to apply practices that focus on communication, adoption and fast feedback. It is absolutely required to pay more attention to people, improve their skills, improve technical excellence and shift mindset.

4. Scrum is Enough

Is it possible to adopt agile without technical practices at all? No, it is not. However, it is a less severe mistake than the previous one. If you apply Scrum right, you will inevitably decide to try some technical practices at a retrospective meeting.

The true mistake will be to rely on Scrum solely as a process that will solve all the problems. It can do that, but only if you are open-minded and willing to try various things: from pair programming to BDD. Best coaches believe that Scrum should be adopted in tight pack with XP technical practices. This is a good advice to follow.

5. CSM Knows Everything

Certified Scrum Master is not a demigod. Yes, he has some basic knowledge about Scrum, but in many cases that’s just it. He may have no hands-on experience neither strong theoretical background. He should not act as a PM and prescribe what to do and what not to do. Team should decide. The only real things SM should care about are teams impediments and meta-process. By meta-process I mean a set of rules/procedures that allow team to reflect and improve existing development process.

Common manifestation of this mistake is phrases like “I insist we should change iteration duration from 1 month to 2 weeks” or “I strongly believe we should keep doing code reviews”. It might be that he is right, but it does not matter, the language speaks for itself. There should be no “I” in SM phrases, there should be “We”. Otherwise agile adoption will fail. If SM doesn’t believe in team, the team as such will not gel and will degrade eventually.

Read Part II for last 5 mistakes.

New Year time is the time when people give resolutions and are particularly enthusiastic about keeping up to their resolve.

One of TargetProcess resolves for this year (another resolve is our roadmap for 2010) is coming up with more blog posts. We promise to keep up feeding  practical and insightful content!

Since everything around is all about newness at this time, let’s think a little bit of how we approach new tools that we use in our work.  Say, you want an agile project management tool, like TargetProcess, and you’re certain that you know very well how to manage agile projects, so all you need is just a tool that would replicate your model of management.

There you go: you sign up for a trial, you start looking at a tool, you take time to see what and how works in there, you discover that this tool is very customizable, but in some cases the tool does not do things the way you do them.  That’s the point where you can submit a feature request and give up, or that’s the point where you can thoughtfully look at the way you do things and see if probably the tool guides you and allows you do agile project management in some better, more lean, more lightweight way?

Imagine that you have an axe but you’re totally unaware that hammers exist in this world. You’re used to fixing nails to their place with the rear part of your axe, and you have no idea that there’s another tool that allows you to do the same task with the same goal — that is, to put the nails where they should be — but in an easier and more straightforward way.  So, when someone hands you a hammer — you look at it and think — how do I use this thing to fix the nails? This tool doesn’t chop wood at all, it has no blade, no powerful rear part — it’s just a small piece of  iron (remember, you’re a man with no idea about hammers, this might be difficult to imagine :) ) Then someone shows you how to hammer nails. You see that hammer is definitely much better for this task.  A hammer head is well-balanced against its handle and it allows to put nails in place with more control and precision as opposed to a heavy axe head.  Then you understand that this new tool does things in some new way that you haven’t known about before. And you like that new way — you grab the hammer!

Sometimes, however, hammers and axes are mixed into one monster like this one:

axe&hammer

You might choose to go and worship the monster praising its versatility. It’s really a fine line between following guidance of a tool that suggests more practical way of accomplishing your tasks and worshiping functionality of a monster, sacrificing your working process and taking time to learn the tool just for the sake of the tool itself. You do learn something new, but is it worth while to nurture a monster, when you can take hammer for hammering,  axe — for chopping wood, and see if you probably need to add a saw, pliers and drill to your set of tools?

The message is: whenever you need a tool to do whatever you need to do, watch out for monsters eating up your work and effort just to learn how they work.  At the same time be open to looking at things from new, unknown angles and don’t stick to old schemes. Who knows,  maybe the tool you’re trying out right now will help you build your ideal working process.

I am really glad that finally we released v.2.14. Check the press release: TargetProcess Announced v.2.14 Release With Full Waterfall Support.

As you probably know we at TargetProcess develop agile project management software. We have a free community edition that is becoming popular and have about 1300 downloads already. People who request the free edition or trial provide comments. Sometimes these comments really disappoint me. Especially if I see comment like “I hope this community release help me dive into scrum. Thanks.”

What’s wrong with it? Well, I don’t know any software tool that can help understand the agile process better. If a company starts SCRUM implementation or any other agile process implementation we do not recommend to start with a tool. Hire agile coaches, read books, try simplest tools like whiteboards and sticky notes, visit conferences, try best practices, and learn! Do not invest into any software tool till you feel the need.

The team may be distributed, or large, or needs real-time progress reports desperately, or anything else. But the team should try to operate with simplest tools first. And focus on live communication. It is the only way to “feel” agile spirit. It is the only way to dive into agile.

Do not rely on a tool. It may help to resolve some problems. It may even help a lot if you have a distributed team. But it may dictate or recommend some best practices, project structure, process structure and the way how you develop  software. That is something to avoid in the beginning of agile adoption. The team should invent their own best practices,  project structure,  process. With this in hands the team can decide to use a tool to resolve real discovered problems.

Agile PM tool is not a silver bullet after all. Do not rely on it blindly. It should adopt to your development process, not vise versa.

Peter Stevens has interesting posts about different agile tools. In this article we will find out whether whiteboard can beat software tools.

Let’s assume that we have a large and shining nail. What is the best tool for the nail? Hopefully, the answer is obvious for the most of us. Now, let’s assume that we have a development team and a “shining”, promising, cool new agile development process. Most likely the hammer will not help (well, perhaps, but certainly not recommended, as a surprise weapon against irresponsible team members who ignore daily meetings).

To tackle this problem, it is essential to have at your disposal a tool that enables requirements gathering, iteration planning, progress tracking and reporting. You can’t rely on memory for requirements gathering, you can’t rely on the universal perception for iteration planning and definitely you can’t rely on telepathy for progress tracking and reporting. You need a tool that will do the job with minimum effort and minimum side effects.

There are two approaches: simplest tools (I mean index cards, whiteboards, etc… no hammer or nails) and software tools. Some people think that index cards and whiteboards are the only possible tools for agile development and all the other tools jeopardize communication in a project team. Some people laugh just seeing a photo of a burndown chart sent by email as a weekly progress report. Obviously I shall say that there is no silver bullet, environments are different and grass is green. Let’s try to find out if the grass is truly greener on the other side by subjecting tangible tools and software toolkits to a comparative analysis.

If you ever tried Extreme Programming or Scrum, you would be familiar with the simplest tools commonly used for the fine-grained short-term project plans creation and progress tracking: Index Cards and Big Visible Charts. These tools have one major advantage over all others’ — they are tangible. You can take an Index Card and move it into another place. You can stick Tasks on the wall thus creating fine-grained plan. Then you can take a marker and draw a new segment on a burn down chart. Sweet! Whiteboards, markers, post-it notes, index cards, board magnets, paper and scissors — many people love to use them. It is entertaining and you have more reasons to get your butt out of the chair.

However, tangible tools do have some significant limitations. For one, no matter how big Visible Charts are, they are still visible to project teams only! Executives should visit the room to see the plan or check on the progress. When was the last time your CTO dropped by? Time tracking and remaining time calculation is manual and someone should update the charts. And for remote teams this approach will simply not work.

Let’s try to compare simplest tools with the web-based tools for the collocated team. Some areas are definitely more important than others. Obviously, communication is more important than fancy plan update or automatic time tracking — therefore we’ll use weights to assign relative values. There are three weights (1-3) and four scores:

1 – Poor

2 – Average

3 – Good

4 – Great

The formula is quite simple: Category score = Weight * Score.

Total score is just a sum of all categories’ scores. In the end we expect to have some numbers that we will use in our analysis.

We have 56 for tangible tools, 52 for web-based software tools and as little as 43 for spreadsheets. Are you still using spreadsheets? Although the above scores are somewhat subjective, it is still clear there is a better way than spreadsheets or other tools not designed for Agile! The totals show that you are better off taking a trip to Staples and getting post-it notes, index cards and markers. If you decided to choose software route, visiting www.targetprocess.com and getting the best agile project management software is certainly another option.

Tangible tools are more preferable for agile processes — use them if you can! But remember, that it remains the case for collocated teams only! It is impossible to use usual tangible tools for remote teams. Remote teams hardly can share whiteboards and task boards. They need more formal processes and more formal tools. Agile offshore development is there and definitely is better than traditional waterfall process. More and more distributed teams use agile development processes successfully and that is the fact of life. What should you use in this case? Obviously, web based software is a great tool for sharing knowledge, project state and other project information. It coordinates remote teams nicely.

In fact, there are some guidelines that you may think about when reviewing the comparison table.

You should prefer White Boards, Cards and Markers if:

• You are trying Extreme Programming or Scrum for the first time (that’s important).
• Team is collocated.
• You tried simplest tools, achieved great results and don’t feel a need to change anything.

You should prefer web-based software tool for project management in agile projects if:

• You have a distributed team (offshore development).
• You have large collocated team (20+ people).
• Your executives need status reports and without them would not accept the agile process (“OK, if I can see real-time project status somewhere, you may try XP/Scrum/Lean”).
• You tried simplest tools and for whatever reason they did not work in your environment. (In that case, you should try to define exact reasons for the failure. Perhaps, there are communication problems or something else that can be fixed).

We may combine results is a small matrix.

Another important question is how can we alleviate potential problems that may arise with web-based tools for project management in agile environment? Some thoughts:

• Focus on communication and collaboration. Do not think that email is OK and enough. It is just not true. Use Skype, use web cams, whenever it is possible — travel and meet in person. Try to break the borders in all possible ways.
• Plan iterations as usual — using cards! Enter data into the system at the completion of the planning game.
• Integrate information radiators with software. For example, you may bind Ambient Orb to project status. It will be green if everything is OK and red if iteration is in trouble. You may print out iteration plan on A3 paper and stick it to the wall.

I think agile software vendors (and TargetProcess as well) should invent new ways to support communication using software and tangible devices. I truly believe it will happen in the very near future.

We are excited to announce the availability of the Free Community Edition of TargetProcess!.

You may request and download On-Site version with 5-users licenses included.

The amazing thing is that the Community Edition has no restrictions at all! It is a full featured version (yes, you have ALL features included: integrated bug tracking, test cases management, subversion integration, time sheets, Web Services API, etc). Moreover, Community Edition has no expiration date – it is simply Free…forever!

We hope that TargetProcess Community Edition will help you to move your Agile development forward.

I’ve manage one small project (3 developers – 6 months planned). Project was canceled due to some reasons (project progress was great, so the reasons are not in development :-). I briefly describe here what we’ve tried and what was useful for us. I think this information will be interesting. This is a real example after all. The project was on Perl.

XP-style Planning. Small 2 weeks iterations. Small iteration helps to track project progress and create better estimates. Planning is quite easy and can be done in Excel without sophisticated tools like MS Project. We are going to use XP-style planning in future for high-risk projects (with new technologies, new team, or whatever make project risk high). Sure, web based tool for XP-planning can help.

Customer Feedback. Manage a project without client representative is a hell on the earth. You are working like in a vacuum. No response, no feedback and no attention. You can’t establish rapport with a client, and there is a chance that system will not satisfy client expectations. And sure it won’t. Project Manager must make every effort to get a person from the client side for the constant feedback.

Info for Top Managers. All information for top managers should be in a short and clear form. Do not send 20 pages requirements docs or lengthy emails. Top managers do not have enough time to read such docs. It is better to extract most important features and ideas at a single page and attach full document to be on a safe side.

Phone Calls. Phone calls are much better than emails or IMs. Sure, phone is impolite and it is hard to clear express your thoughts, but, anyway, phone is better. Many issues can be resolved faster and easier.

Test-Driven Development. We wrote unit tests for all business objects before actual code, and it worked great. However, we did not have tests for controllers. Controllers are very close to GUI and they changed often (during first month). Maybe we should write unit tests for controllers as well, maybe not. This is unclear. Anyway, TDD is good and we will use it as a mandatory practice.

Acceptance Tests. About 50 tests were created based on SAMIE. We modified tests quite often and spent near half an hour hour per day to keep ‘em working. These tests helped a lot when we performed major refactoring in controllers and GUI layers. But it seems that SAMIE is not the best tool for acceptance testing. It is easy to use, but sometimes unstable and Win32 restricted. But I did not find a better one so far.

Refactoring. It helps to reduce code duplication and keeps system architecture clear. We had doing refactoring constantly, and sometimes we spent whole day on major refactoring. As a result, new functionality implementation appeared easier than we expected. We will use refactoring as a mandatory practice in future.

Walking Skeleton. This is a first, fast and small release, which contains single system feature ‘in depth’. For example, show projects list, add project into the database, check user permissions on project. Walking Skeleton for the system took near 2 weeks. We managed to investigate many architectural decisions and choose the best. Small and fast release is a great thing.

Pair Programming. We created the most important system modules in pair. The code quality and overall system architecture appeared significantly better when pair programming used. Constant code review, communication and experience sharing are really great things. It is not easy to program 7 or more hours in a row, and you feel exhausted in the end of the day. But work satisfaction is great. We will use pair programming for teaching, experience sharing and critical modules implementation in future.

Code Metrics. LOCs dynamic, Unit Tests # dynamic, Acceptance Tests dynamic. Code metrics helped to track project progress and predict team velocity and project size.

Project Site. Helps to organize important project related information (news, documents, actual project state, links) in a single place. However, Project site update should be more simpler. For example, we can use a basic CMS for project site maintenance in future.

O/R Mapping. Tangram was used as an O/R mapping tool. It took about 2-3 days to resolve some tricky issues, but in general O/R mapping speeds up development and supports database abstraction. Maybe OODBMS is a better solution, but if you stick to RDBMS, O/R mapping will help. However, for small systems or web sites O/R mapping is not required.

System Prototypes. Helps to resolve many issues on the early stage. In general, the prototype should be created if project budget allows to do that. But I do not recommend to create a comprehensive prototype, since it just wastes time. Prototype should contain major system modules and maybe some secondary.

Honesty. Just be honest with your teammates. They will trust you, and trust is one of the major factors for gelled teams.

Documentation in Source Code. Almost all system architecture related documentation was in source code. We used a tool to build linked html pages from POD documentation. As a result, documentation maintenance was easier.