Edge of Chaos | Agile Development Blog

Many interesting posts and discussions about Scrum and Kanban published last weeks. Someone even called this a war I don’t think it is a war, but some posts indeed may drive such feeling.

• Ken Schwaber bashed Kanban. “I was told that Kanban is frequently used when an organization cannot readily adopt Scrum. Many of Scrum most difficult aspects are then sidestepped. Managers are still in charge of telling people what to do. People can be interrupted at any time. People are still work in functional silos, preserving the jobs of functional managers. People are not allowed to work in containers, sharing skills and knowledge to bring complexity into solutions – instead they are worked on a pull (more sophisticated than push) production line.”
• Karl Scotland discusses Scrum and Kanban difference as Intentional vs. Implementational approaches. Interesting perspective in fact. Karl thinks that Kanban can be used with Scrum to reveal even more problems in development process.
• David J. Anderson posted the most rational article about Scrum and Kanban difference. I like it a lot. Some phrases are true gems: “Kanban is not a project management or software development lifecycle method. It is an approach to change management - a framework for catalyzing change in an organization.” and “Kanban uses a WIP limit as a change agent and Scrum uses commitments. This is a fundamental difference in approach.” David also posted some reflections later with interesting thoughts about anarchy and science.
• A year ago Tobias Mayer didn’t believe that it is possible to use Kanban at all. “I fundamentally disbelieve that there is any such thing as a “value stream” when you are working in a complex environment, in a creative process, building new products or generating new ideas.”

I think Ken is wrong. His arguments against Lean and Kanban quite ridiculous. Sure there is no intention to treat software development as a factory and apply lean manufacturing principles blindly. Of course there are people who will try (or already tried) that and fail. But vast majority of lean community in software development do understand the difference and working on concrete practices. Lean philosophy is great, but tools and practices can’t be taken from manufacturing directly.

Pull system is more complex than Push? C’mon!

David has wise arguments and perfect position. I think Kanban will be more popular than Scrum in long-term perspective, but it will take time. Visualization is a key thing to manage complexity, and software development is a complex system.

We live in the age of added value. It’s everywhere. Value-added services, value-added products, value-added goods etc. etc. etc.  Actually, so much value has been pumped up in our life, that it’s even strange that this value is not protruding from us like clothes from an overly packed vacation suitcase.

If we take a closer look at the back side of added value, a huge surprise is waiting there. The example I find most notorious is cell phones. What if I want a simplistic cell phone with NO Internet, no camera, even no voice mail, just live calls and SMS?  You’ll never ever find such a phone.

I bet that a phone manufacturer who stops the rush for more new features, would make a fortune in an instant selling the “new frugal” cell phones. In this case, the added value is content which comes from Internet, capabilities to exchange content. Why should someone want a phone with Internet?  My word, very soon we will see such phones on the market. The niche for them is already there. Here’s why:

More content and more various channels to produce and exchange content is now commonly presented as added-value. Hence, a communication device which happens to be a humble phone is supposed to deliver this value. But as we’re oversaturated with content, no buzz is a huge deal. The luxury of focusing on one thing at a time is something that only a few can afford. Besides, it’d be very frugal (frugal is actually the new buzzword to buy a phone for a reasonable price, and sell used iPhone to some geek. Oh, pardon me. It’s all about iPads now

There’re plenty of such examples. Another one: the added value of having a car means lack of natural movement, the necessity to pay for gym etc. It actually brings along the whole array of more added value  goods and services that turn out to be not of added value but of less value, since you pay for what you could do naturally.

Take organic products. Now they’re added value. 100 years ago who could have thought that something natural adds value? Now it’s a rollback. What is simple and natural costs more, but has less value compared to the original added- value concept for the matter, and the cycle goes on and on.

And we’re nailing it down to our favorite: project management tools. Versatility and too many features now have bounced back to the simplistic Kanban board.

It looks like it’s time to not only practice lean production, but to produce lean products. Gaining focus requires focused tools, one way or another.

When reading this Kill Your To-Do List blog post, I thought that managing personal to-do list can be similar to product backlog management. Not in the part that you should totally kill your product backlog, but in the “one thing at a time” part. This is by no means a new thought.  E.g.  Mary Poppendieck was heard saying that product backlog should be eliminated.

So, as in to-do lists, there’s no point in nurturing and moving product backlog items back and forth.  In this case, a lot depends on what is your definition and understanding of product backlog. I’d say backlog is an environment in which a product grows and exists but not a set of components that should be implemented for sure.  Looking further into, what kind of environment is it? Backlog might be a repository of all the slightest shades of ideas that have a very vague chance to be implemented. A chaotic heap. This heap can hardly be called a “backlog” but rather a by-product of brainstorming. OR a product backlog might represent a careful selection of user stories that will be implemented for sure.

Both of these options, as often is the case, have the optimal state somewhere in between. In our production workflow, we’re now using several buffers - the first layer is raw Requests and Ideas (coming from our HelpDesk, or from the PO, or from the team). The second layer is Product Backlog with User Stories (Requests and Ideas are now groomed and converted to User Stories by  Product Owner who decides that some time these will be implemented for sure - based on how many people requested this feature, based on current product development strategy etc.). The third layer is Planned state for User Stories in Kanban board.

Here’s a quick graphical representation of this product backlog upward funnel:

That’s how User Stories lifecycle looks on  Kanban board, from Planned state on:

Backlog is not seen on Kanban board.  When done with their current user stories, developers  pull new user stories from Planned state.

Previously, when we worked with iterations before switching to Kanban, there was no buffer between inception and implementation - so time-boxed releases and iterations have been planned directly from the backlog. With Kanban, the buffering  is done by moving stories to Planned state and prioritizing them.

I’d say that planning with iterations provides less flexibility than planning with Kanban. As you drop user stories to time-boxed iterations, you  commit to implementing all of them within a given period of time. Kanban is way more flexible since user stories can be pulled one-by-one from Planned state and implemented with no time restrictions. I can’t resist citing an analogy here: it’s the same as moving with the smallest possible steps to posture yourself before hitting the ball in tennis. With large steps, you do not have flexibility. With small steps - you’re very agile and flexible to position yourself for the perfect shot.

So, the buffered Planned state in Kanban is like this breaking down into small steps, instead of taking one giant leap and committing to the whole pack of user stories in iteration.

That’s the way it goes for us. You’re better off moving by small steps, taking it one-by-one (this brings us back to the inspiration blog post reference in the beginning than with giant leaps.

“You can’t apply Scrum without an external expert”
“You can’t apply Scrum without a Certified Scrum Master”
“You can’t apply Scrum without XYZ”

You can replace Scrum with any other buzzword. Is it really necessary to have an agile coach on board to join agile camp? Is it really required to send someone to CSM courses and delegate Scrum adoption to this brave knight in a shiny armor? I believe it is not the only way to go, and there are 2 reasons that support my vision:

1. I did not attend any courses, conferences and other events, but I did learn agile and became an agile expert.
2. We implemented agile process in our company without any external help.

To me it looked (and still looks) natural to improve development process internally. It was an ad-hoc agile adoption  in our company. There was no defined structure, but on-going learning. People have been reading books about agile development which I recommended. I’ve been making some presentations about agile history, main processes and so on. People have been reading articles and discussing new ideas. It worked out, finally. Even with such an unstructured approach you are able to set up an agile company. But are there better ways?

Agile Coaches

What’s wrong with external coach/trainer/consultant? Nothing, actually. You pay someone to teach you. You pay someone to inject agile culture into your company. You pay someone to get it faster.

The problem is that this process is so slooow. It is inevitably slow, because all the mindset changes are slow, and you can’t speed them up greatly. My estimate is about 1 year as a minimum (no proof, sorry, just personal experience). You can’t absorb all the agile spirit in a shorter period of time (maybe some geniuses can, but they don’t need coaches anyway). I do believe that a good agile coach can speed up agile adoption, but not as greatly as often advertised.

OK, so changes are slow. It is absolutely required to manage the change over a long period of time. You can’t setup something, run it for a couple of iterations and leave. There is a high chance that development team will degrade and slip back to old-and-oh-so-known practices quite fast. It leads to several possible conclusions:

1. If you hire agile coach, it is better to have about 1 year contract.
2. If you hire agile coach for just 3-6 months, setup internal learning/change process as fast as possible.

The main idea of agile adoption is a paradigm shift. People should change their habits, rituals, working patterns and activities. It is SO hard! Some people just can’t accept it and leave the company. The goal of agile adoption is to change mindset of as many people as possible and let rock-hard minority go.

So why I personally don’t like the idea to hire someone who will teach us agile? It immediately puts our intelligence to question. Are we really not able to learn ourselves? Can’t we read some books, discuss them and try, for example, Scrum in a single dev. team? If we want to hire someone, to me it looks like we’ve already given up and can’t do anything cool without external help. It looks like we got exhausted and now need a power recharge. It is similar to an external CEO hired in a desperate attempt to save company.

Don’t get me wrong. Agile coach can help greatly. But I don’t buy the paramount idea that development team can’t go agile without external help.

Problems and Solutions

I like challenges and like to solve problems without external help. What I like is a full team involvement and participation. It is so cool when people discuss problems and generate solutions. If I ever hire an external consultant, it will be a person who will teach how to solve problems. Problem solving is the MOST IMPORTANT skill (can I stress it more?) for any good developer/tester/designer/you-name-it.

Here I see a major flaw of Scrum as a methodology (so far?). It does provide a framework, but it says nothing about problem solving. OK, you should have a retrospective meeting every other week. You should identify problems, generate ideas and write action items. You should execute action items and retrospect in 2 weeks. Sounds good. But HOW to identify problems? HOW to generate ideas? Scrum says nothing about that, but I believe this should be the core of any agile process.

That is why I pay more and more attention to lean and other problem solving techniques like 5 Whys, TRIZ, etc.

Knowledge From Within

There are various ways to ignite the team and solve problems internally, but you can’t solve problems effectively without knowledge. Study Groups could be the most promising way to educate people. To be honest, we did not try it. As I mentioned, we did not have any structural approach so far, but intuitively came to something similar to Study Groups.

Study Group is a small group of people (4-6) that have cadence sessions e.g. weekly. There’s no distinct leader, they rotate with every session. People should be really interested in the domain. For example, if you have Study Group to learn TDD, all the members of the group should be passionate about TDD learning and actively participate in discussions.

Why I think Study Group should work? First, there’s no Boss inside. People can freely discuss various problems without pretending that they know everything. Second, there is a clear focus. You should not form a group with a goal to solve all the development problems, instead it is better to form groups to study UX, study TDD, study Scrum, study Lean. Once again, Study Group is not a problem solving technique, but it helps to educate people. And it is absolutely clear that educated people will generate better solutions faster.

Here is the nice summary about Study Group. And here is the list of other learning methods that every agile team should pay attention to.

There’s so much room for observations and analogies in the evolution of production trends. Analogies are not merely a candy for the brain. They bring along a deeper understanding of phenomena and ultimately are one of the greatest aides to align (or misalign) with mainstream.

If we look back, to the 18-19th century - mass production was a dream. The philosophy was: produce more.  Lavish architecture designs, garments, gardens - everything created by people was about going more massive and taking much space. Standards of innovation have been changing over the centuries  - what’s been innovative and massive, has been becoming obsolete. Minimalism is the new innovative. Is it because humans subconsciously feel they’ve taken too much terrain and sky on this planet for industrial experiments and now are trying to compensate for that by being minimalistic in everything? Or simply finding ways to fit in?

Hardware/software as well as visual designs and interfaces are meekly following the same trend. This just shows how subtly the “new innovative” standards are taking over. We remember huge PCs. Now we’ve got all kinds of minimalistic devices the names of which start with “i“. We remember waterfall. CMMI standards  with their tons of rigid rules, regulations, documented processes. Now we’re  going “lean” and “agile” - the same minimalistic tendency.

People have managed to stuff the overproduced artifacts not only all over the planet but all over themselves. Fatness is the problem. Again, what a shift in standards -  as late as in the beginning or even middle of the 20th century it was considered trendy to be fat. Well, not obese, but hearty fat. Now, we’re all about lean. Off with plump beauties. Straighten up now, lean is the philosophy of minimalism in production.

P.S. I truly believe that all the fat folks are hiding the “lean” insignia deeply inside them. It just takes some effort to peel off the layers

Recently I’ve come across a post on harmfulness of analogies with martial arts. Indeed, there’s a handful of posts comparing software development, or agile adoption, or product development  with  martial arts  - Aikido, Karate, Judo etc.

If you make a direct analogy of software product development and mastering some martial art, it would not be exactly accurate. I guess  people revert to the analogies as they try to project their copies as romantic martial arts disciples into their usual lives as software developers, managers etc.  Perhaps, they’re making the image of their lives more mission-filled this way since there’s not too much space for showing primeval qualities of warrior in software development. But the need to exercise this attitude is still there, as it turns out.

We don’t have beasts to fight, bloody battles and mortal combats. This wrap-up for courage, strong will, persistence in achieving goals and readiness to fight has remained in the past largely. With no wrap-up, people forget that there’s lots of space to exercise these qualities in their usual life.

Let’s go back to analogies. Roger Federer is an unbeatable specimen of mastered elegant performance to me. Look at the way he plays. No waste. He knows, what to do, he knows when to do what. It’s a perfect flow,  the model for effective lean production with no waste and -  the model for perfect warrior  in our modern life. Elegant, no blood on his hands, but he fights, has pitfalls on the way,  stands up, recovers, marches on and wins gracefully.

But he’s just a guy in red T-Shirt. As many software developers

Another point is that it’s much harder to fight, win and achieve goals when there’s no immediate physical danger involved as in tennis.  Soo much room for elegant warriors, isn’t it ?

The point I’m trying to make is - let people compare their lives and their jobs to whatever they want. If it inspires and motivates them, makes them feel good about their lives and their work - and makes them feel like warriors, achievers, believers, fighters, winners.

P.S.  February 23 is a perfect day for such a blog post Wish you well, guys!

Cumulative Flow diagram is a very good starting point for stop-the-line or retrospective meeting. Here is a real example from TargetProcess development.

You can see in the chart that we had a bottleneck in the beginning of December. It was caused by a quite complex user story. In theory a single user story should not affect cycle time significantly and should not create bottlenecks, but if you violate some rules it might be the case.

The user story was about replacing jQuery javascript framework with ExtJS framework. It took 1 month to implement by 1 developer. During this month we’ve made several releases and all went smooth. Then this user story was verified by testers and all existing acceptance tests were passed. So we’ve made a decision to merge this story to the main code line.

Unfortunately, after the merge quite many bugs were found in the build during smoke testing. These bugs took more than a week to fix, and during this time we were unable to release anything, since the merge was already done and the rollback was quite complex as well.

The lesson learned is to put more effort into testing of user stories that are more complex in nature. This particular story affected many places in the application and usual smoke testing was not enough. So we are going to introduce a new class of service (let’s call it  “technically complex story” so far which would mean more in-depth testing and verification before the merge.

In general, Cumulative Flow chart is a valuable tool to analyze historical data, but for emergency bottleneck identification Kanban Board is even better. If you have limits on Kanban Board, you see problems immediately.

I’m digging into Lean Manufacturing and it’s so interesting to learn manufacturing history and its trends. There are so many parallels between manufacturing and software development. Of course they are not direct, but a curious mind can draw them quite easily. Let’s take Cellular Manufacturing as an example.

Cellular Manufacturing is a workspace organization technique. The main principle is to organize production of a single product in a small dedicated unit (cell) that consists of several people and several workstations. The benefits are surprisingly huge:

“The result is very fast throughput. Communication is easy since every operator is close to the others. This improves quality and coordination. Proximity and a common mission enhance teamwork.”

It is a significant improvement over the functional space configuration, when you have large queues and long transition time.

Now, if we think about software development, it is becoming clear why distributed teams have problems and why functional departments will not survive in the near future.

Cellular manufacturing applied to software development directly leads to cross-functional teams that work on a single project. Cell is a single team that has its own inventory and cross-trained people. Expanding it further, it means that developers should be able to do testers’ job, and vice versa.

From lean perspective, managers job is to setup cells configuration efficiently. It is not required to track individual work, but to track cell work instead. It brings process optimization to a higher level and powers system thinking.

It is really amazing how much we can extract from lean manufacturing and adopt this knowledge to software development…

I got this insight on lean and agile techniques in military context when reading  “Ideas Made to Stick” book.  The workflow of the military was described there as an example of how important it is to make messages as concise as possible to accomplish tasks.

The evolution of the military strategy with dates and sources is not a subject of this blog
With no extra details, here’s the core of the point I’m trying to make:

First the Army’s approach was to make sure that every single action is planned down to smallest details. But, as they found out, “enemy bears no plans”. An unexpected move could destroy the whole set-up so  “all king’s horses and all king’s men” could not make the Army function again. Effectively, I mean.

So, they reverted to something very similar to agile technique of creating user stories - Commander’s Intent. Commander’s Intent is “the commander’s stated vision which defines the purpose of an operation, the end state with respect to the relationship among the force, the enemy and the terrain; it must enable subordinates to quickly grasp the successful end state and their part in achieving it”.

Do you see the resemblance? It’s a replica of lean production principles, only in military, not in civic, context. And it’s exactly like the agile principle of engaging people and encouraging their creativity to achieve one common goal.

I will not go into quoting sources on the web on Commanders’ Intent and military doctrines to find more analogies and similarities in military setup and lean/agile methodology.  I just outlined the idea.

There’s a misconception related to this topic: Some people think that agile works all OK for lazy folks. It’s all about freedom, about looseness, no obligations. Not at all. Agile is less conventional, it does not care about being in the office at 9 am sharp, or about wearing a tight business suit. But agile projects do have their Commanders’ Intent and they require genuine responsibility and engagement from the team - the soldiers.

Now, as  the Army follows agile principles, would you need any better proof on the effectiveness of agile and lean methodology?

You Kanbanized your development process and setup a perfect flow. You see the flow and feel its power. You even measure cycle time and want to reduce it. But it is not as easy as expected. Some problems in your development process are hard to find and are revealed on team level. Here I’ll describe an interesting technique that may help you.

The idea is quite simple. Take a single user story and visualize its states transition from start to finish, thus you see the whole flow of a single story. Draw all available states on Y-axis and days on X-axis. As a result, you will have a chart like that (click to enlarge).

It is a real user story from TargetProcess project. It was quite complex and its cycle time was 19 days. Let me provide some comments.

User story has been in Planned state during just 1 day, then developers took it and moved to In Progress state. So far so good. 4 days of development and user story is moved to Coded state, but it sits there for 2 days. Why? Why testers did not take it immediately? Clearly, we found 2 days of delay and the reason of the delay should be investigated, since it increases story cycle time.

Then testers took the user story and tested it for 2 days. Then the  story was moved to In Progress state again and developers worked on it for 2 more days (including Saturday). Why? It is a half of the initial development time. It is a clear sign of a waste, but we need more details. It appeared, that testers found 11 bugs. Half of the bugs were caused by unclear user story specification, other are subject to investigate. We should apply root-cause analysis here to find the real problems.

OK. Then testers have verified the user story for 2 days and moved it to Ready to Merge state. It was merged quite fast (in 1 day), but still there may be a waste to eliminate. Finally, the user story have lived in Merged state for 5 days and was released Jan-29.

In total (and as a minimum!) we discovered about 9 days of waste. It means we can reduce cycle time significantly by eliminating this wait time (waste). If you check several user stories using this technique, I believe you will find comparable results for simple and complex stories.

Here is another example for a quite simple user story. Still we clearly see wait times in our flow! We can reduce cycle time to 2 days instead of 6 by just eliminating waits in the flow. But it is not easy

This chart is a really powerful tool. It is easy to create and you could use it to find waste in the development flow.