Edge of Chaos | Agile Development Blog

Kai Gilb started a series of posts that challenge Agile Software Development. Part 2 is about creativity and “how well” focus in software development. While I agree with the problem, I strongly disagree with the solution. Kai recommends to write requirements (the format he uses doesn’t allow me to call it “user stories”) in a form that does not contain solution:

Developers Are Not Product Owners

Then developers should creatively find the ways to implement this requirement, to invent the solution and to solve the “how well” puzzle. We’ve got a huge problem here indeed. Most developers can’t invent solutions at this high level. They can create beautiful architecture and build the quality in. They can write tests upfront and use powerful tools to speed up development. They can do many things, but often they can’t invent a “how well” solution that will make customers happy. Why?

1. Most developers know nothing about User Experience (and don’t care).
2. Most developers can’t create usable design.
3. Many developers barely know a problem domain and in any case don’t want to dig into much details here.

In short, most developers are engineers (so far?), so what Kai proposes will not help. It is just a theoretical solution to “how well” problem. It’s the same like trying to grow 4 hands and 2 heads to double your productivity (well, sacrificing beauty maybe…)

In Scrum Product Owner is responsible for writing User Stories. It means he should invent solutions and care about “how well” thing.  I don’t see a significant problem here. Of course, it’s great if developers are interested in requirements handling and solutions investigation process, but if they are not, Product Owner can do that alone or with special people who care. Who they can be?

Those people are UX specialists. They know how to solve “how well” problems effectively. They know many things about design patterns and all new trends in interfaces. They like to explore problem domain and talk to customers. If you have such people among developers — you are SO lucky! But often this is not the case.

We, at TargetProcess, try to instill UX culture into development process. We started in August, now it’s February and I can’t say we’ve had much success. Sure there are significant improvements, but to be honest most developers are not so interested in UX staff, they like BDD, DDD, C# and SOA more. I expect it will be a long process that will take years. And our company is quite small. I can’t even imagine an effort for such a radical shift in a huge company.

Broad Requirements Are Not Testable

I wonder how you can write BDD scenarios for broad requirements as Kai suggests. How the hell can you test anything, if you don’t have any clue on how it should work? So broad requirements can’t replace user stories obviously. They can live in backlog, but developers should deal with concrete requirements that are testable right away.

I can agree that you can split backlog in 2 parts. The first small part contains detailed stories that can be put to development asap. The second large part contains broad requirements without solutions. That might be a good idea. But we should clearly separate UX phase and Development phase. During UX phase we should solve “how well/how cool” problems, and during development phase we should solve technical problems only (with some corrections in “how well” for sure if required).

P.S. Kai, I hate gray text on gray background in your blog. It is not user friendly.

I read Agile development grows UP article written by Mark Lines. It has quite a strange taste… Is it an attempt to advertise new methodology? I hope it is, otherwise Mark is completely missing the point and agile development trends.

I want to review the article. Just can’t resist, sorry.

Many of us don’t have the benefit of co-location, where the entire team works in close proximity (ideally in the same room), with the user representative present at all times to answer questions

Sure we all know that a co-located team is better. It is not a surprise that consultants advise to co-locate team members. If you need to win the race, it is better to drive BMW M3 than Ford Focus. There are so many discussions around distributed agile this year. There are so many ideas and experience reports on how to implement agile remotely. Agile works in remote teams.

Pair programming i.e. 2 programmers sharing 1 keyboard is an expenditure most management will not endorse

It’s a very strange argument. Many managers will not support continuous integration, iterative development and BDD. It does not mean that these practices are bad. Context is everything. In some companies and on some projects pair programming will increase productivity. While on other projects there will be no benefits or even degradation. How many agile coaches insist on pair programming and say “You are not agile without PP”? Not many. It is wrong to spread this argument to the whole agile community.

Delivering a system without moderately detailed requirements (beyond User Stories) is not acceptable for testing, writing training materials, and production support purposes.

C’mon! User stories can be VERY detailed. For example, you may use BDD-style user stories format and write many cases that will describe functionality in some format, that can be directly converted to unit tests! It is an incredible thing, to be honest.

Fixing architecture as you go (i.e. refactoring) without some initial architectural design is not appropriate for large-scale enterprise systems

Most people in agile community believe that it is required to spend some time on architecture. Again, it is so context dependent. If you create a simple application, one day may be enough to nail down all important decisions. If you create a complex system, it may take several months. Common sense is a critical factor here. There is always a danger to over-architect things and working software is the best proof of concept.

Also something called “emergent architecture” is being developed. I like the idea and maybe it is a right direction.

Most projects are not completely independent and cannot be built in isolation. Dependencies and integrations with other systems are required and require some co-ordination.

Agile adoption is spreading rapidly and indeed there are no common/standard approaches to solve this problem. But I believe it will be resolved during agile enterprise adoption (and we already stepping into this phase).

Organizations usually have some governance practices in place (such as PMOs, ISO, CMMI) that necessitate a level of bureaucracy and documentation

Does it all mean that PMO, ISO and CMMI are good and we should give up and follow the rules? Agile community tries to fight bureaucracy and I personally love that. Documented process means NOTHING for success of software development project. In 90% of cases it holds you in the spider’s net and blocks creativity. There is NO software development without creativity. Wake up, Mark.

We live with these standards, but we should fight them.

Deploying software to users every 1 or 2 weeks is usually not practical in most organizations. Just migrating software through development, test, system test, user acceptance, production environments is a major undertaking. Having many projects trying to do this in parallel on a weekly basis simply isn’t feasible.

This argument is so common… and “not practical in most organizations” is a truly masterpiece. Any statistic data? Did Mark try to deploy anything outside the enterprise-scale Java mastodon applications world? Many SaaS services do painless weekly updates. Some services do painless updates on a Daily basis! It is unbelievable, but I like that. Customers like that!

Mark promotes OpenUP methodology (based on Unified Process). Only advantages in the post, no disadvantages or context at all. Folks, I think it is a so long-awaited silver bullet! Finally it is there! Let’s bury Scrum. Let’s bury XP. Let’s burn Kanban and all the other processes that spark agile movement. God bless the OpenUP silver bullet. Amen.

Cumulative Flow diagram is a very good starting point for stop-the-line or retrospective meeting. Here is a real example from TargetProcess development.

You can see in the chart that we had a bottleneck in the beginning of December. It was caused by a quite complex user story. In theory a single user story should not affect cycle time significantly and should not create bottlenecks, but if you violate some rules it might be the case.

The user story was about replacing jQuery javascript framework with ExtJS framework. It took 1 month to implement by 1 developer. During this month we’ve made several releases and all went smooth. Then this user story was verified by testers and all existing acceptance tests were passed. So we’ve made a decision to merge this story to the main code line.

Unfortunately, after the merge quite many bugs were found in the build during smoke testing. These bugs took more than a week to fix, and during this time we were unable to release anything, since the merge was already done and the rollback was quite complex as well.

The lesson learned is to put more effort into testing of user stories that are more complex in nature. This particular story affected many places in the application and usual smoke testing was not enough. So we are going to introduce a new class of service (let’s call it  “technically complex story” so far which would mean more in-depth testing and verification before the merge.

In general, Cumulative Flow chart is a valuable tool to analyze historical data, but for emergency bottleneck identification Kanban Board is even better. If you have limits on Kanban Board, you see problems immediately.

I’m digging into Lean Manufacturing and it’s so interesting to learn manufacturing history and its trends. There are so many parallels between manufacturing and software development. Of course they are not direct, but a curious mind can draw them quite easily. Let’s take Cellular Manufacturing as an example.

Cellular Manufacturing is a workspace organization technique. The main principle is to organize production of a single product in a small dedicated unit (cell) that consists of several people and several workstations. The benefits are surprisingly huge:

“The result is very fast throughput. Communication is easy since every operator is close to the others. This improves quality and coordination. Proximity and a common mission enhance teamwork.”

It is a significant improvement over the functional space configuration, when you have large queues and long transition time.

Now, if we think about software development, it is becoming clear why distributed teams have problems and why functional departments will not survive in the near future.

Cellular manufacturing applied to software development directly leads to cross-functional teams that work on a single project. Cell is a single team that has its own inventory and cross-trained people. Expanding it further, it means that developers should be able to do testers’ job, and vice versa.

From lean perspective, managers job is to setup cells configuration efficiently. It is not required to track individual work, but to track cell work instead. It brings process optimization to a higher level and powers system thinking.

It is really amazing how much we can extract from lean manufacturing and adopt this knowledge to software development…

Books by Edward Tufte are a piece of art. I’ve been savoring them to myself for a while, and now I decided to share some sketches and criticism inspired by Tufte’s high art visual designs.

Commonly, designers represent visual information by scarce means of 2D realm: screen and paper. Our universe is 3D (if not 5D, 6D or whatever more dimensions), but people got used to squeezing  images into 2D flatland.  Even rock paintings of pre-historic humans have their touch of 2-D abstraction and symbolism.

Our universe is not just 3D. It’s dynamic 3D. Paper is static (paper planes are exceptions). That’s another limitation of 2D.

Limitations are great. They motivate designers to find solutions. The more limitations - the harder it is to find a solution. Good designers love difficult tasks, since they view them as great opportunities to put their brains to use. Bad designers do not want to use their brains - they want to use templates.

The image below is a template solution for a weather map. View from above.  Let alone template thinking, the representation of this template is poor.

The appalling hint of white shade is a helpless attempt to compensate inadequate color selection for numbers.  What do you think of blue numbers on blue background? You hate that, to say the least of it. What’s the message of these pseudo-3D grey circles? Are they some grey moons? Or cavities  in the designer’s brain?

Now let’s take a look at the Euronews channel weather map.  One may think that this map represents the effects of global warming and Australia is completely hidden under water now. Also, what do those bold numbers show? Probably the depth of the ocean in this area. In meters. Or in miles? But the area is still lit by sunshine, which instills some hope.

As a contrast, here’s a weather map from a Japanese daily, beautiful in its simplicity.  This is the same Japan as on the first weather map above, only from the ocean perspective.  This map provides 0°C и 10°C  isotherms.  You see fine clouds on this map. The map shows sun movement.  OMG, it shows stratosphere! And it’s nothing more than just a weather map from a daily newspaper  - but created by a good visual designer.

Of course, Japan is well-suited for such a nice graphical representation. But you gotta have guts to catch and use this ocean perspective, instead of helplessly surrendering to boilerplate view-from-above weather maps imposed by paper sheet or screen limitations.

You Kanbanized your development process and setup a perfect flow. You see the flow and feel its power. You even measure cycle time and want to reduce it. But it is not as easy as expected. Some problems in your development process are hard to find and are revealed on team level. Here I’ll describe an interesting technique that may help you.

The idea is quite simple. Take a single user story and visualize its states transition from start to finish, thus you see the whole flow of a single story. Draw all available states on Y-axis and days on X-axis. As a result, you will have a chart like that (click to enlarge).

It is a real user story from TargetProcess project. It was quite complex and its cycle time was 19 days. Let me provide some comments.

User story has been in Planned state during just 1 day, then developers took it and moved to In Progress state. So far so good. 4 days of development and user story is moved to Coded state, but it sits there for 2 days. Why? Why testers did not take it immediately? Clearly, we found 2 days of delay and the reason of the delay should be investigated, since it increases story cycle time.

Then testers took the user story and tested it for 2 days. Then the  story was moved to In Progress state again and developers worked on it for 2 more days (including Saturday). Why? It is a half of the initial development time. It is a clear sign of a waste, but we need more details. It appeared, that testers found 11 bugs. Half of the bugs were caused by unclear user story specification, other are subject to investigate. We should apply root-cause analysis here to find the real problems.

OK. Then testers have verified the user story for 2 days and moved it to Ready to Merge state. It was merged quite fast (in 1 day), but still there may be a waste to eliminate. Finally, the user story have lived in Merged state for 5 days and was released Jan-29.

In total (and as a minimum!) we discovered about 9 days of waste. It means we can reduce cycle time significantly by eliminating this wait time (waste). If you check several user stories using this technique, I believe you will find comparable results for simple and complex stories.

Here is another example for a quite simple user story. Still we clearly see wait times in our flow! We can reduce cycle time to 2 days instead of 6 by just eliminating waits in the flow. But it is not easy

This chart is a really powerful tool. It is easy to create and you could use it to find waste in the development flow.

Kanban looks so simple. In theory. Map your flow, set limits, draw some lines and stick stories on the wall. There you go. Is that it? Obviously, no. One of the hardest things is to change your behavior, shift from “push mindset” to “pull mindset”. And that may be tricky.

You have Kanban board and there are no free slots in “In Progress” column. It means new stories should not be started — you’ve reached the limit. Developer comes and says “I know that there are no free slots, but I want to start this new small user story, since all the people are already working on the items and my help is not required”. You might think “Hmm, indeed we are moving forward with a good pace, this small story should not take much time…” and you say “OK, John, take it. I don’t think this would be a problem to exceed the limit right now.”

Two hours later Pete comes to you with a similar question: “I just completed my user story, testers will take it soon and I want to start something new.” You say “Well, we just took one story above the limit…”. Pete replies “Please, boss, I don’t want to sit and do nothing, I want to add some value, I am so much into work today!” Can you say no? I bet in most cases you say something like “OK, let’s do it!” to not discourage your teammate.

What happens then? Testers find some bugs in just implemented user stories, but there are no free developer to fix them. John and Pete are working on new stories. They may switch to bugs, but they’d lose focus and you have 2 stories in progress that can’t be passed to other developers easily (it will take time to make the transition and it looks like waste, isn’t it?). As a result, you deliver some stories later. Yes, later!

I understand that it is very hard initially to say something like “No, you’d better go and read something interesting about new technology. We can’t start new story right now, because if testers find bugs in some running stories, we should fix them ASAP, so we need a free developer”. Moreover, you should train your team to not ask such questions. WIP limit is a rule that should be followed. You may change it, but not arbitrarily. This change should have something behind (better productivity, more developers, etc). And this change should be discussed in retrospective meeting or during a Kaizen event.

I should say it is really, really hard to accept the fact, that sometimes developers and testers should do nothing (I mean do not start new stories when they are free). But it’s absolutely required to do so, otherwise pull system will not work.

Code base of a large project is getting worse over time. I hope there are lucky exceptions, but in general it is true for most projects. The reasons are quite obvious:

• More and more features. It leads to increased complexity.
• Shortcuts and hacks to support “We need this fancy search till August. Period!” features
• Developers rotation. New developers don’t know all the fundamental decisions and ideas behind the architecture. Knowledge gets lost with transition inevitably.
• Development team growth. More people - less communication. Less communication - bad decisions. It leads to code duplication, hacks to make something work without deep understanding of the underlying conditions etc.

Suddenly you can’t add new features easily, you can’t make significant changes easily, you have tons of technical debts and development team is close to bankruptcy. You want to change that and have just two options: refactoring or rewriting everything from scratch.

Refactoring and Punctuated Equilibrium

Punctuated Equilibrium is a theory in biology, but it is applied to other complex adaptive systems like societies as well. It states that the system has almost no changes in significant period of time and then changes very rapidly. Then it acquires the equilibrium state again. Sounds familiar? Sure, refactoring is almost the same.

Any change in the system introduces chaos in the short term. The goal of refactoring is to eliminate chaos, but often you increase it initially. When you are working on changes you make system less stable, but as the change is completed,  the system can be more ordered. That is not always true for software development. If you use branches,  this makes changes safer. Still significant changes increase the risk of new bugs.

The real danger of refactoring are local optimums. With a full rewrite you may have a better architecture than with refactoring. Solution? If you have a vision of final architecture, use it and try to make a path from current architecture to the new architecture. In general simplicity should emerge from refactoring.

Rewrite and Chaos

When you rewrite from scratch, you add such a large portion of chaos that it is hard to predict the final result. You have a new singularity that will explode to the new product universe. But are you certain that it will be better than the previous universe? How many the ’same bugs’ will you fix in the new product version?

However, rewrite may look faster. I mean you may release a new version faster with rewrite, but most likely with more bugs and less stable.

I think we may expect something like that:

The green line shows how chaos changes with refactoring. After each refactoring there is a small increase of chaos, but then the system becomes stable and chaos decreases. You see that the final release is quite late, but keep in mind that there have been many releases before, so customers benefit earlier.

Black line is how chaos changes with a full rewrite. We have the old system during rewrite, so chaos is constant. After the public release chaos increases significantly. Quite many new (and old) bugs and quirks are expected, so stabilization period is longer. But the release itself is faster. The reason is that there is no burden behind. For example, there is no need to support all the places when you do a change, while in refactoring it is required  to keep system working and stable all the time, and it demands additional effort.

Adaptation vs. Revolution

There is another angle to this problem. Refactoring is adaptation, while full rewrite is revolution. Again, revolution is a chaotic beast. You may slowly adapt the product for new external conditions or make one revolutionary rewrite.

So, Rewrite or Refactor?

I do think there is no universal correct answer to this question. If time to market is paramount, if you feel that you’d lose business if a new version will not be published in 6 months, you may try a full rewrite. But beware side effects! Quality may drop significantly and long stabilization period may hurt existing customers.

In most cases refactoring is preferable. Slow pace, high quality, constant improvements, happy customers. I like it more. But rewrite is so magnetic…

Everything has its life-cycle. Even stars. Even The Universe. Everything.

It is quite sad for intelligent creatures, but it is just a fact of life. I want to draw some quite obvious parallels to reveal the real danger of Technical Debt for a software product.

Humans

Let’s take a human. When we are young, we are overflowed with energy. Do you know how a 7 year-old boy spends his day? I have a son, so I’ll tell you. He wakes up at 7 am (sometimes before 6 am!) and tries to play some quite silent games (it is too early to make much noise). He can restrain himself till about 8 am, then he starts to jump, does somersets,  some boxing etc… By 8.30 we have a real “tornado” in our house. It may take him an hour to exhaust and have a breakfast. Such periods of activity can be easily repeated 2-3 times a day. And he literally turns off around 9 pm - sleep is when he recharges his batteries.

An old man is quite the opposite. He saves every movement, everything is hard and slow. He makes many mistakes and health maybe getting worth. Learning new things is also getting harder at times.  Yes, wisdom and tremendous life experience is there,  but very often there is no energy to move forward.

Software products

Software products have exactly the same behavior. Young product is energetic. It has an excited development team behind. It moves forward quickly and feels warm sun on the top. It learns lightning fast and grows on new knowledge. Everything is smooth and shiny. Time to market is everything and tradeoffs are so common! Let’s fix it fast and mark with //TODO: refactor this comment. Let’s hack this framework here to release another cool new feature this week. Let’s re-invent the wheel and write some custom javascript code. Gosh! We don’t have time to learn this new cool javascript framework! These are quite common patterns for young and energetic products.

But what’s next? Years later progress stops. Product becomes old with all the bad and good side-effects. Suddenly it is much harder to add new features. Ages of development make the product more complex. Millions of technical debts rotten its body like cancer . There is no clear architecture anymore and there are so many patches that one small change can produce a totally unpredictable impact and bring along new bugs in unexpected areas. Yes, development team has wisdom. But often no courage and energy to revive the product.

Technical Debt Is a Cancer

How long does this cycle last? It really depends. We all visit doctors. Most of us want to live longer and healthier. Early cancer detection gives good chances to fight and win the disease. We should do exactly the same with software. Here are typical symptoms:

• Velocity has dropped significantly  over the last several iterations/releases
• A bug fix triggers one or several new bugs too often
• Nobody knows ideas behind the original software architecture
• Team spends more time fixing bugs than developing/improving features
• Automatic tests are red 80% of the time (if it is 100% — the product is most likely in coma)

If you see at least 2 such symptoms, you’ve just discovered a product cancer — Technical Debt. Technical debt is a true killer when you have deadline (time to market). If you have the symptoms, you should fight the disease right now. You may think that it is OK to wait  several months, add some more “cool and highly requested” features and then get back to the real problems. It is a wrong decision, believe me. I used to make it and I used to fail with it. It bogs you down. You lose focus and make stupid mistakes. It leads to fear. And fear is a bad ally. You go from one extreme to another to  only increase entropy, nothing else.

If you miss the deadline, all the possible actions will not help. Cancer will win. And then you will have just two options: re-write the product completely from  scratch or start a new product.

If you see these symptoms, you should stop and think about the attitude of your development team. If you’ve survived over several years, priorities should be changed. Reset your development team and use chemotherapy.

• Focus on quality. Fix the roots of the problems.
• Teach the original architecture to all, pair program, communicate.
• Introduce “No new code without tests” rule.
• Fight fear. Let the knowledge spread. Knowledge eliminates fear.
• Put the most experienced people on fundamental problems solving.

You have to fight the cancer to bring energy back, to bring courage back, to live and produce a great software product. Otherwise it will be as dead as Lotus Notes.

Once upon a time there lived a very brave and adventurous young man by the name Arthur. He lived in a large kingdom with knights and castles and an extremely cunning king who had a beautiful daughter - Caroline. She was so beautiful, that every morning birds would fly through the open window with fresh roses in their beaks. Birds would drop rose petals near her bed and fill the air with pleasant tweets, every five minutes exactly. Then they would fly away to never come back…they would simply die outside, because they couldn’t live without princess’ beauty anymore.

The king was greedy so he held annual competitions during which the contestants had to pay 10 sovereigns each to participate. The winner would marry Caroline, but missions were so hard, that there were no winners ever!

New competition was announced and brave Arthur decided to win the highest prize! He saw the beautiful princess only once before and fell desperately in love with her.

On a sunny day all the contestants gathered together on a central square. The king said: “Brave men! There will be 3 main missions… and a very easy 4th mission to the winner. Today I’ll tell you about the first mission only. You have to build a house in 10 days. It should have 2 rooms, doors, windows and a roof. That’s all you have to do!!”

Mission #1

Is it possible to build a house in 10 days? Arthur was disappointed. He had hoped to kill a dragon, catch a witch or do something else that a brave knight is supposed to do. But he was smart (and brave). He spent all the day doing nothing but thinking. Other contestants laughed at him: “Look at Arthur! He gave up miserably! Arthur, go home, you are such a loser!” But as you know, our Arthur was smart so he decided to build the simplest house possible. And he did it: the ground became a floor; house had only 2 tiny rooms and the light would enter through the 2 windows near the front doors; the roof was there to simply protect from rain and snow - no more no less! Most importantly, he finished it in 8 days! Many, many other brave men tried but failed. Some started to build a large and solid house and ran out of time. Others started building too quickly and their houses would collapse half way through. However, brave Arthur and a dozen other guys completed the first mission successfully.

“I am proud of you! - said the cunning king. “Now here is the second mission for you all. This time, you should build a basement that will be twice as large as the house itself. Also you should build a fireplace. You all have a total of 12 days to get the job done.”

Mission #2

And again Arthur spent the whole day thinking. “I don’t even have a floor in the house… How can I build a basement? The house will simply break down… If I spend time on the floor, I’ll have no time to do the rest.” Then he found a workaround. He decided to dig deeper than usual. “Deep basement should be safe enough and floor is not required in this case! Fireplace will take 2 days to create, and I can definitely make this basement in 9 days!”

Arthur worked like crazy and completed the mission in 11 days. He slept for the whole 12th day – the last day of the mission. At the end, only 5 other brave men have completed this stage of competition.

“You did a fantastic job!” – the king said. - “Now listen to the last mission. You should build an upper level in your house. An upper level may have only one room, but you have only 5 days to do it.”

Mission #3

All the contestants were quite excited. It looked like an easy task. But Arthur was sad. “If I build another level” - he thought - “my house will crash…” This time he started to work immediately. He brought several large logs into the basement and made props. It took him a total of 2 days. Arthur was not sure that it’d be enough, but he had no more time and decided to accept the risk. Luckily, he had a flat roof and therefore it turned out to be relatively easy to create another level right on top of it. Arthur was tired, but extremely happy. After all, he was the only one who made it through the last mission!

“My boy!” - King happily hugged Arthur. - “You did it! I can’t believe, but you did it! You are just one step away from the Caroline’s hand. Your last mission can’t be simpler - You should live in the house you built during the first 2 weeks. This is it. And then you will have my blessings and the hand of my beloved daughter, Caroline!”

“Wow, so easy!” - said Arthur. “Yes, it is” - answered the king with a smile on his face.

Mission #4

The next day brave Arthur was killed by the crashed ceiling. It was a windy and rainy day…