Edge of Chaos | Agile Development Blog

There’s so much room for observations and analogies in the evolution of production trends. Analogies are not merely a candy for the brain. They bring along a deeper understanding of phenomena and ultimately are one of the greatest aides to align (or misalign) with mainstream.

If we look back, to the 18-19th century - mass production was a dream. The philosophy was: produce more.  Lavish architecture designs, garments, gardens - everything created by people was about going more massive and taking much space. Standards of innovation have been changing over the centuries  - what’s been innovative and massive, has been becoming obsolete. Minimalism is the new innovative. Is it because humans subconsciously feel they’ve taken too much terrain and sky on this planet for industrial experiments and now are trying to compensate for that by being minimalistic in everything? Or simply finding ways to fit in?

Hardware/software as well as visual designs and interfaces are meekly following the same trend. This just shows how subtly the “new innovative” standards are taking over. We remember huge PCs. Now we’ve got all kinds of minimalistic devices the names of which start with “i“. We remember waterfall. CMMI standards  with their tons of rigid rules, regulations, documented processes. Now we’re  going “lean” and “agile” - the same minimalistic tendency.

People have managed to stuff the overproduced artifacts not only all over the planet but all over themselves. Fatness is the problem. Again, what a shift in standards -  as late as in the beginning or even middle of the 20th century it was considered trendy to be fat. Well, not obese, but hearty fat. Now, we’re all about lean. Off with plump beauties. Straighten up now, lean is the philosophy of minimalism in production.

P.S. I truly believe that all the fat folks are hiding the “lean” insignia deeply inside them. It just takes some effort to peel off the layers

Recently I’ve come across a post on harmfulness of analogies with martial arts. Indeed, there’s a handful of posts comparing software development, or agile adoption, or product development  with  martial arts  - Aikido, Karate, Judo etc.

If you make a direct analogy of software product development and mastering some martial art, it would not be exactly accurate. I guess  people revert to the analogies as they try to project their copies as romantic martial arts disciples into their usual lives as software developers, managers etc.  Perhaps, they’re making the image of their lives more mission-filled this way since there’s not too much space for showing primeval qualities of warrior in software development. But the need to exercise this attitude is still there, as it turns out.

We don’t have beasts to fight, bloody battles and mortal combats. This wrap-up for courage, strong will, persistence in achieving goals and readiness to fight has remained in the past largely. With no wrap-up, people forget that there’s lots of space to exercise these qualities in their usual life.

Let’s go back to analogies. Roger Federer is an unbeatable specimen of mastered elegant performance to me. Look at the way he plays. No waste. He knows, what to do, he knows when to do what. It’s a perfect flow,  the model for effective lean production with no waste and -  the model for perfect warrior  in our modern life. Elegant, no blood on his hands, but he fights, has pitfalls on the way,  stands up, recovers, marches on and wins gracefully.

But he’s just a guy in red T-Shirt. As many software developers

Another point is that it’s much harder to fight, win and achieve goals when there’s no immediate physical danger involved as in tennis.  Soo much room for elegant warriors, isn’t it ?

The point I’m trying to make is - let people compare their lives and their jobs to whatever they want. If it inspires and motivates them, makes them feel good about their lives and their work - and makes them feel like warriors, achievers, believers, fighters, winners.

P.S.  February 23 is a perfect day for such a blog post Wish you well, guys!

Cumulative Flow diagram is a very good starting point for stop-the-line or retrospective meeting. Here is a real example from TargetProcess development.

You can see in the chart that we had a bottleneck in the beginning of December. It was caused by a quite complex user story. In theory a single user story should not affect cycle time significantly and should not create bottlenecks, but if you violate some rules it might be the case.

The user story was about replacing jQuery javascript framework with ExtJS framework. It took 1 month to implement by 1 developer. During this month we’ve made several releases and all went smooth. Then this user story was verified by testers and all existing acceptance tests were passed. So we’ve made a decision to merge this story to the main code line.

Unfortunately, after the merge quite many bugs were found in the build during smoke testing. These bugs took more than a week to fix, and during this time we were unable to release anything, since the merge was already done and the rollback was quite complex as well.

The lesson learned is to put more effort into testing of user stories that are more complex in nature. This particular story affected many places in the application and usual smoke testing was not enough. So we are going to introduce a new class of service (let’s call it  “technically complex story” so far which would mean more in-depth testing and verification before the merge.

In general, Cumulative Flow chart is a valuable tool to analyze historical data, but for emergency bottleneck identification Kanban Board is even better. If you have limits on Kanban Board, you see problems immediately.

I’m digging into Lean Manufacturing and it’s so interesting to learn manufacturing history and its trends. There are so many parallels between manufacturing and software development. Of course they are not direct, but a curious mind can draw them quite easily. Let’s take Cellular Manufacturing as an example.

Cellular Manufacturing is a workspace organization technique. The main principle is to organize production of a single product in a small dedicated unit (cell) that consists of several people and several workstations. The benefits are surprisingly huge:

“The result is very fast throughput. Communication is easy since every operator is close to the others. This improves quality and coordination. Proximity and a common mission enhance teamwork.”

It is a significant improvement over the functional space configuration, when you have large queues and long transition time.

Now, if we think about software development, it is becoming clear why distributed teams have problems and why functional departments will not survive in the near future.

Cellular manufacturing applied to software development directly leads to cross-functional teams that work on a single project. Cell is a single team that has its own inventory and cross-trained people. Expanding it further, it means that developers should be able to do testers’ job, and vice versa.

From lean perspective, managers job is to setup cells configuration efficiently. It is not required to track individual work, but to track cell work instead. It brings process optimization to a higher level and powers system thinking.

It is really amazing how much we can extract from lean manufacturing and adopt this knowledge to software development…

I got this insight on lean and agile techniques in military context when reading  “Ideas Made to Stick” book.  The workflow of the military was described there as an example of how important it is to make messages as concise as possible to accomplish tasks.

The evolution of the military strategy with dates and sources is not a subject of this blog
With no extra details, here’s the core of the point I’m trying to make:

First the Army’s approach was to make sure that every single action is planned down to smallest details. But, as they found out, “enemy bears no plans”. An unexpected move could destroy the whole set-up so  “all king’s horses and all king’s men” could not make the Army function again. Effectively, I mean.

So, they reverted to something very similar to agile technique of creating user stories - Commander’s Intent. Commander’s Intent is “the commander’s stated vision which defines the purpose of an operation, the end state with respect to the relationship among the force, the enemy and the terrain; it must enable subordinates to quickly grasp the successful end state and their part in achieving it”.

Do you see the resemblance? It’s a replica of lean production principles, only in military, not in civic, context. And it’s exactly like the agile principle of engaging people and encouraging their creativity to achieve one common goal.

I will not go into quoting sources on the web on Commanders’ Intent and military doctrines to find more analogies and similarities in military setup and lean/agile methodology.  I just outlined the idea.

There’s a misconception related to this topic: Some people think that agile works all OK for lazy folks. It’s all about freedom, about looseness, no obligations. Not at all. Agile is less conventional, it does not care about being in the office at 9 am sharp, or about wearing a tight business suit. But agile projects do have their Commanders’ Intent and they require genuine responsibility and engagement from the team - the soldiers.

Now, as  the Army follows agile principles, would you need any better proof on the effectiveness of agile and lean methodology?

You Kanbanized your development process and setup a perfect flow. You see the flow and feel its power. You even measure cycle time and want to reduce it. But it is not as easy as expected. Some problems in your development process are hard to find and are revealed on team level. Here I’ll describe an interesting technique that may help you.

The idea is quite simple. Take a single user story and visualize its states transition from start to finish, thus you see the whole flow of a single story. Draw all available states on Y-axis and days on X-axis. As a result, you will have a chart like that (click to enlarge).

It is a real user story from TargetProcess project. It was quite complex and its cycle time was 19 days. Let me provide some comments.

User story has been in Planned state during just 1 day, then developers took it and moved to In Progress state. So far so good. 4 days of development and user story is moved to Coded state, but it sits there for 2 days. Why? Why testers did not take it immediately? Clearly, we found 2 days of delay and the reason of the delay should be investigated, since it increases story cycle time.

Then testers took the user story and tested it for 2 days. Then the  story was moved to In Progress state again and developers worked on it for 2 more days (including Saturday). Why? It is a half of the initial development time. It is a clear sign of a waste, but we need more details. It appeared, that testers found 11 bugs. Half of the bugs were caused by unclear user story specification, other are subject to investigate. We should apply root-cause analysis here to find the real problems.

OK. Then testers have verified the user story for 2 days and moved it to Ready to Merge state. It was merged quite fast (in 1 day), but still there may be a waste to eliminate. Finally, the user story have lived in Merged state for 5 days and was released Jan-29.

In total (and as a minimum!) we discovered about 9 days of waste. It means we can reduce cycle time significantly by eliminating this wait time (waste). If you check several user stories using this technique, I believe you will find comparable results for simple and complex stories.

Here is another example for a quite simple user story. Still we clearly see wait times in our flow! We can reduce cycle time to 2 days instead of 6 by just eliminating waits in the flow. But it is not easy

This chart is a really powerful tool. It is easy to create and you could use it to find waste in the development flow.

Kanban looks so simple. In theory. Map your flow, set limits, draw some lines and stick stories on the wall. There you go. Is that it? Obviously, no. One of the hardest things is to change your behavior, shift from “push mindset” to “pull mindset”. And that may be tricky.

You have Kanban board and there are no free slots in “In Progress” column. It means new stories should not be started — you’ve reached the limit. Developer comes and says “I know that there are no free slots, but I want to start this new small user story, since all the people are already working on the items and my help is not required”. You might think “Hmm, indeed we are moving forward with a good pace, this small story should not take much time…” and you say “OK, John, take it. I don’t think this would be a problem to exceed the limit right now.”

Two hours later Pete comes to you with a similar question: “I just completed my user story, testers will take it soon and I want to start something new.” You say “Well, we just took one story above the limit…”. Pete replies “Please, boss, I don’t want to sit and do nothing, I want to add some value, I am so much into work today!” Can you say no? I bet in most cases you say something like “OK, let’s do it!” to not discourage your teammate.

What happens then? Testers find some bugs in just implemented user stories, but there are no free developer to fix them. John and Pete are working on new stories. They may switch to bugs, but they’d lose focus and you have 2 stories in progress that can’t be passed to other developers easily (it will take time to make the transition and it looks like waste, isn’t it?). As a result, you deliver some stories later. Yes, later!

I understand that it is very hard initially to say something like “No, you’d better go and read something interesting about new technology. We can’t start new story right now, because if testers find bugs in some running stories, we should fix them ASAP, so we need a free developer”. Moreover, you should train your team to not ask such questions. WIP limit is a rule that should be followed. You may change it, but not arbitrarily. This change should have something behind (better productivity, more developers, etc). And this change should be discussed in retrospective meeting or during a Kaizen event.

I should say it is really, really hard to accept the fact, that sometimes developers and testers should do nothing (I mean do not start new stories when they are free). But it’s absolutely required to do so, otherwise pull system will not work.

Everything has its life-cycle. Even stars. Even The Universe. Everything.

It is quite sad for intelligent creatures, but it is just a fact of life. I want to draw some quite obvious parallels to reveal the real danger of Technical Debt for a software product.

Humans

Let’s take a human. When we are young, we are overflowed with energy. Do you know how a 7 year-old boy spends his day? I have a son, so I’ll tell you. He wakes up at 7 am (sometimes before 6 am!) and tries to play some quite silent games (it is too early to make much noise). He can restrain himself till about 8 am, then he starts to jump, does somersets,  some boxing etc… By 8.30 we have a real “tornado” in our house. It may take him an hour to exhaust and have a breakfast. Such periods of activity can be easily repeated 2-3 times a day. And he literally turns off around 9 pm - sleep is when he recharges his batteries.

An old man is quite the opposite. He saves every movement, everything is hard and slow. He makes many mistakes and health maybe getting worth. Learning new things is also getting harder at times.  Yes, wisdom and tremendous life experience is there,  but very often there is no energy to move forward.

Software products

Software products have exactly the same behavior. Young product is energetic. It has an excited development team behind. It moves forward quickly and feels warm sun on the top. It learns lightning fast and grows on new knowledge. Everything is smooth and shiny. Time to market is everything and tradeoffs are so common! Let’s fix it fast and mark with //TODO: refactor this comment. Let’s hack this framework here to release another cool new feature this week. Let’s re-invent the wheel and write some custom javascript code. Gosh! We don’t have time to learn this new cool javascript framework! These are quite common patterns for young and energetic products.

But what’s next? Years later progress stops. Product becomes old with all the bad and good side-effects. Suddenly it is much harder to add new features. Ages of development make the product more complex. Millions of technical debts rotten its body like cancer . There is no clear architecture anymore and there are so many patches that one small change can produce a totally unpredictable impact and bring along new bugs in unexpected areas. Yes, development team has wisdom. But often no courage and energy to revive the product.

Technical Debt Is a Cancer

How long does this cycle last? It really depends. We all visit doctors. Most of us want to live longer and healthier. Early cancer detection gives good chances to fight and win the disease. We should do exactly the same with software. Here are typical symptoms:

• Velocity has dropped significantly  over the last several iterations/releases
• A bug fix triggers one or several new bugs too often
• Nobody knows ideas behind the original software architecture
• Team spends more time fixing bugs than developing/improving features
• Automatic tests are red 80% of the time (if it is 100% — the product is most likely in coma)

If you see at least 2 such symptoms, you’ve just discovered a product cancer — Technical Debt. Technical debt is a true killer when you have deadline (time to market). If you have the symptoms, you should fight the disease right now. You may think that it is OK to wait  several months, add some more “cool and highly requested” features and then get back to the real problems. It is a wrong decision, believe me. I used to make it and I used to fail with it. It bogs you down. You lose focus and make stupid mistakes. It leads to fear. And fear is a bad ally. You go from one extreme to another to  only increase entropy, nothing else.

If you miss the deadline, all the possible actions will not help. Cancer will win. And then you will have just two options: re-write the product completely from  scratch or start a new product.

If you see these symptoms, you should stop and think about the attitude of your development team. If you’ve survived over several years, priorities should be changed. Reset your development team and use chemotherapy.

• Focus on quality. Fix the roots of the problems.
• Teach the original architecture to all, pair program, communicate.
• Introduce “No new code without tests” rule.
• Fight fear. Let the knowledge spread. Knowledge eliminates fear.
• Put the most experienced people on fundamental problems solving.

You have to fight the cancer to bring energy back, to bring courage back, to live and produce a great software product. Otherwise it will be as dead as Lotus Notes.

Inspired by Kanban at Lonely Planet. I think it is a good idea to share software development processes from different companies. Maybe even it would be great to create a repository of the processes somewhere on a popular agile web site/blog.

Context

• Product development (quite large web application)
• Two development teams: Core (5 developers, 1 scrum master, 3 testers) and Integration (3 developers - one of them plays scrum master role, 1 tester)
• Quite mature product: 4 years
• Technology: C#, ASP.NET, ExtJS, NHibernate

Process

• Development team has all the power to try new practices, keep working practices and remove bad practices. There’s a little push from scrum master side.
• We do not use iterations.
• We do use releases, but often it is just a set of all the features implemented last month.
• We do not have iteration planning meeting, but discuss every user story JIT before implementation on a meeting with PO + Testers + Developers.
• We do not estimate stories and bugs, just a very brief estimate like (”oh, this will be a large story, let’s split it”).
• We split stories, but sometimes not as aggressively as required.
• We release new builds every week (sometimes 2 builds per week). The goal is to release every day, but we do not have a good automated functional tests coverage so far.
• Smoke testing of the build takes 3 hours (3 testers).
• We have a limit of 3 user stories or bugs in progress.
• We have retrospective meetings every 2 weeks.
• We sometimes have technical user stories (for example, remove jQuery from application), but that is rare.
• We use Kanban Board in TargetProcess heavily and have a large screen in development room.
• We have a flat hierarchy in the company. There are only two layers and no boundaries between the layers. No formalism at all. Also we do not have positions like Senior Developers, Junior Developers, etc. All people are equal (however, there are informal leaders).
• Average Cycle time for user story is 13 days and 5 days for bug.
• We track spent/remaining time on tasks and stories.

Development practices

• We develop each user story or bug in a separate branch. Master is always ready for release (after smoke testing). We use Git (switched from Subversion).
• Pair programming on all user stories and complex bug fixes. Sometimes people work alone on smaller/simpler problems.
• Developers switch pairs every day, even if user story is not completed yet. The goal is to a have fresh look, knowledge spreading and real collective code ownership.
• Before story implementation, pairs discuss possible solutions. It is quite formal.
• TDD. All the new code is covered by unit tests. Situation is worse on JavaScript side, we just started TDD adoption for UI.
• We use Cruise Control and have quite many different setups. Smoke builds, release builds, etc. Functional tests run in parallel and it takes 1 hour to run them all.
• Selenium. Automated functional tests are based on Selenium. We’ve created a framework in C# especially for our application to simplify tests creation. Tests coverage is not good so far, but we are working on it (there are 2 automation testers in our team).

ToDo

• We want to add performance tests into automated builds.
• We want to have 80% functional tests coverage to be able to release new builds every day
• We are incorporating User Experience into development process. This is our main goal for the next year. It includes Customers UX Community creation, interactive prototypes for all user stories, usability testing.

In previous post I mentioned 5 wrong reasons to apply Kanban. Tobias Mayer asked to blog about 5 right reasons Obviously there should be right reasons, otherwise Kanban adoption would fail everywhere. So there they go:

#1. Ability to release anytime

In Scrum or XP you can’t release in the middle of an iteration. In Kanban you may release anytime. When user story is ready, you may release it. Definitely it is a challenge to setup development process this way. It is required to have “branch-by-feature” source control policy, merge often, integrate and test often. But it gives you an ability to release often. That is something worth to fight for.

As a PO I like this ability very much. An important user story implemented? OK, let’s release it tomorrow in v.2.15.2. Our customers may benefit from this release as soon as possible. Lead time is faster — customers are happier.

One thing to mention — you need to have a good automated test suite, otherwise it will be problematic to make small releases with a good quality.

#2. Ability to change priorities on the fly

In Scrum you can’t add stories on the fly into sprint (usually). At least it is not an easy process and development team often resists to replacing a story from Sprint backlog. The story has been discussed, estimated etc. A new story may be discussed in a hurry, some details will be missed and as a result significant re-work will be required. So in general iteration or sprint should be frozen.

In Kanban if you have an urgent request to implement or a really important user story, you can just put it on top of the queue. It will be taken as soon as a free slot is available. It is simply a Product Owner’s dream

#3. No need in iterations

Why you need iterations? Initially iterations help to reveal real problems in the development process quickly. Further on they establish a project rhythm and rituals. However at some point in project I think you no longer need them.

My vision is that you need to iterate first, then flow. Our backlog is fuzzy now, plans change often. We learn how to do agile in general and iterations are not helpful there anymore. Without iterations there is no need in iteration planning and iteration demo meetings. They are waste. Instead we have more smaller just-in-time meetings with people in a feature-team before starting the development for each user story.  It works perfect.

Some people are missing the rhythm, but I think it is more to habit. Kanban establishes more complex rhythm and it may take time for development team to recognize it. Still stable rhythm may be the only reason to keep iterations in the late phase of product development, in my opinion.

#4. No need in estimates

Why you need estimates? In iterative development you need them to predict how many stories you may take in the next iteration. You may even predict a release date based on estimated backlog and iteration velocity. The other reason is that PO wants to know how big the user story is. If it is big enough, PO  may consider moving it to the next release. If it is small, PO may decide to put it into the next iteration.

Obviously, iterative development is hardly possible without estimates. But if you drop iterations, it is not a problem anymore. Can Product Owner live without estimates? Well, yes. In our company we don’t estimate user stories. Why not? Because I as a PO don’t need them and we don’t use iterations. All I may ask is a very rough estimate like  normal, large, very large.

In our situation (I stress that, in our situation), estimates are waste. We don’t spend time on estimates. We have a prioritized backlog and just take the most important user story and implement it. It may not work in some conditions, but we have quite a mature product and hundreds requests from customers. There is no clean field development, so normally we don’t have any defined release date. We release when it is ready (I understand that many companies can’t do that, but we have this favor).

#5. Perfect flow visualization

Kanban Board provides a very clear view on current work in progress. It visualizes flow and enables fast planning and tracking. It is really a great tool, we love it.