Edge of Chaos | Agile Development Blog

Kai Gilb started a series of posts that challenge Agile Software Development. Part 2 is about creativity and “how well” focus in software development. While I agree with the problem, I strongly disagree with the solution. Kai recommends to write requirements (the format he uses doesn’t allow me to call it “user stories”) in a form that does not contain solution:

Developers Are Not Product Owners

Then developers should creatively find the ways to implement this requirement, to invent the solution and to solve the “how well” puzzle. We’ve got a huge problem here indeed. Most developers can’t invent solutions at this high level. They can create beautiful architecture and build the quality in. They can write tests upfront and use powerful tools to speed up development. They can do many things, but often they can’t invent a “how well” solution that will make customers happy. Why?

1. Most developers know nothing about User Experience (and don’t care).
2. Most developers can’t create usable design.
3. Many developers barely know a problem domain and in any case don’t want to dig into much details here.

In short, most developers are engineers (so far?), so what Kai proposes will not help. It is just a theoretical solution to “how well” problem. It’s the same like trying to grow 4 hands and 2 heads to double your productivity (well, sacrificing beauty maybe…)

In Scrum Product Owner is responsible for writing User Stories. It means he should invent solutions and care about “how well” thing.  I don’t see a significant problem here. Of course, it’s great if developers are interested in requirements handling and solutions investigation process, but if they are not, Product Owner can do that alone or with special people who care. Who they can be?

Those people are UX specialists. They know how to solve “how well” problems effectively. They know many things about design patterns and all new trends in interfaces. They like to explore problem domain and talk to customers. If you have such people among developers — you are SO lucky! But often this is not the case.

We, at TargetProcess, try to instill UX culture into development process. We started in August, now it’s February and I can’t say we’ve had much success. Sure there are significant improvements, but to be honest most developers are not so interested in UX staff, they like BDD, DDD, C# and SOA more. I expect it will be a long process that will take years. And our company is quite small. I can’t even imagine an effort for such a radical shift in a huge company.

Broad Requirements Are Not Testable

I wonder how you can write BDD scenarios for broad requirements as Kai suggests. How the hell can you test anything, if you don’t have any clue on how it should work? So broad requirements can’t replace user stories obviously. They can live in backlog, but developers should deal with concrete requirements that are testable right away.

I can agree that you can split backlog in 2 parts. The first small part contains detailed stories that can be put to development asap. The second large part contains broad requirements without solutions. That might be a good idea. But we should clearly separate UX phase and Development phase. During UX phase we should solve “how well/how cool” problems, and during development phase we should solve technical problems only (with some corrections in “how well” for sure if required).

P.S. Kai, I hate gray text on gray background in your blog. It is not user friendly.

I read Agile development grows UP article written by Mark Lines. It has quite a strange taste… Is it an attempt to advertise new methodology? I hope it is, otherwise Mark is completely missing the point and agile development trends.

I want to review the article. Just can’t resist, sorry.

Many of us don’t have the benefit of co-location, where the entire team works in close proximity (ideally in the same room), with the user representative present at all times to answer questions

Sure we all know that a co-located team is better. It is not a surprise that consultants advise to co-locate team members. If you need to win the race, it is better to drive BMW M3 than Ford Focus. There are so many discussions around distributed agile this year. There are so many ideas and experience reports on how to implement agile remotely. Agile works in remote teams.

Pair programming i.e. 2 programmers sharing 1 keyboard is an expenditure most management will not endorse

It’s a very strange argument. Many managers will not support continuous integration, iterative development and BDD. It does not mean that these practices are bad. Context is everything. In some companies and on some projects pair programming will increase productivity. While on other projects there will be no benefits or even degradation. How many agile coaches insist on pair programming and say “You are not agile without PP”? Not many. It is wrong to spread this argument to the whole agile community.

Delivering a system without moderately detailed requirements (beyond User Stories) is not acceptable for testing, writing training materials, and production support purposes.

C’mon! User stories can be VERY detailed. For example, you may use BDD-style user stories format and write many cases that will describe functionality in some format, that can be directly converted to unit tests! It is an incredible thing, to be honest.

Fixing architecture as you go (i.e. refactoring) without some initial architectural design is not appropriate for large-scale enterprise systems

Most people in agile community believe that it is required to spend some time on architecture. Again, it is so context dependent. If you create a simple application, one day may be enough to nail down all important decisions. If you create a complex system, it may take several months. Common sense is a critical factor here. There is always a danger to over-architect things and working software is the best proof of concept.

Also something called “emergent architecture” is being developed. I like the idea and maybe it is a right direction.

Most projects are not completely independent and cannot be built in isolation. Dependencies and integrations with other systems are required and require some co-ordination.

Agile adoption is spreading rapidly and indeed there are no common/standard approaches to solve this problem. But I believe it will be resolved during agile enterprise adoption (and we already stepping into this phase).

Organizations usually have some governance practices in place (such as PMOs, ISO, CMMI) that necessitate a level of bureaucracy and documentation

Does it all mean that PMO, ISO and CMMI are good and we should give up and follow the rules? Agile community tries to fight bureaucracy and I personally love that. Documented process means NOTHING for success of software development project. In 90% of cases it holds you in the spider’s net and blocks creativity. There is NO software development without creativity. Wake up, Mark.

We live with these standards, but we should fight them.

Deploying software to users every 1 or 2 weeks is usually not practical in most organizations. Just migrating software through development, test, system test, user acceptance, production environments is a major undertaking. Having many projects trying to do this in parallel on a weekly basis simply isn’t feasible.

This argument is so common… and “not practical in most organizations” is a truly masterpiece. Any statistic data? Did Mark try to deploy anything outside the enterprise-scale Java mastodon applications world? Many SaaS services do painless weekly updates. Some services do painless updates on a Daily basis! It is unbelievable, but I like that. Customers like that!

Mark promotes OpenUP methodology (based on Unified Process). Only advantages in the post, no disadvantages or context at all. Folks, I think it is a so long-awaited silver bullet! Finally it is there! Let’s bury Scrum. Let’s bury XP. Let’s burn Kanban and all the other processes that spark agile movement. God bless the OpenUP silver bullet. Amen.

I’m digging into Lean Manufacturing and it’s so interesting to learn manufacturing history and its trends. There are so many parallels between manufacturing and software development. Of course they are not direct, but a curious mind can draw them quite easily. Let’s take Cellular Manufacturing as an example.

Cellular Manufacturing is a workspace organization technique. The main principle is to organize production of a single product in a small dedicated unit (cell) that consists of several people and several workstations. The benefits are surprisingly huge:

“The result is very fast throughput. Communication is easy since every operator is close to the others. This improves quality and coordination. Proximity and a common mission enhance teamwork.”

It is a significant improvement over the functional space configuration, when you have large queues and long transition time.

Now, if we think about software development, it is becoming clear why distributed teams have problems and why functional departments will not survive in the near future.

Cellular manufacturing applied to software development directly leads to cross-functional teams that work on a single project. Cell is a single team that has its own inventory and cross-trained people. Expanding it further, it means that developers should be able to do testers’ job, and vice versa.

From lean perspective, managers job is to setup cells configuration efficiently. It is not required to track individual work, but to track cell work instead. It brings process optimization to a higher level and powers system thinking.

It is really amazing how much we can extract from lean manufacturing and adopt this knowledge to software development…

I got this insight on lean and agile techniques in military context when reading  “Ideas Made to Stick” book.  The workflow of the military was described there as an example of how important it is to make messages as concise as possible to accomplish tasks.

The evolution of the military strategy with dates and sources is not a subject of this blog
With no extra details, here’s the core of the point I’m trying to make:

First the Army’s approach was to make sure that every single action is planned down to smallest details. But, as they found out, “enemy bears no plans”. An unexpected move could destroy the whole set-up so  “all king’s horses and all king’s men” could not make the Army function again. Effectively, I mean.

So, they reverted to something very similar to agile technique of creating user stories - Commander’s Intent. Commander’s Intent is “the commander’s stated vision which defines the purpose of an operation, the end state with respect to the relationship among the force, the enemy and the terrain; it must enable subordinates to quickly grasp the successful end state and their part in achieving it”.

Do you see the resemblance? It’s a replica of lean production principles, only in military, not in civic, context. And it’s exactly like the agile principle of engaging people and encouraging their creativity to achieve one common goal.

I will not go into quoting sources on the web on Commanders’ Intent and military doctrines to find more analogies and similarities in military setup and lean/agile methodology.  I just outlined the idea.

There’s a misconception related to this topic: Some people think that agile works all OK for lazy folks. It’s all about freedom, about looseness, no obligations. Not at all. Agile is less conventional, it does not care about being in the office at 9 am sharp, or about wearing a tight business suit. But agile projects do have their Commanders’ Intent and they require genuine responsibility and engagement from the team - the soldiers.

Now, as  the Army follows agile principles, would you need any better proof on the effectiveness of agile and lean methodology?

Kanban looks so simple. In theory. Map your flow, set limits, draw some lines and stick stories on the wall. There you go. Is that it? Obviously, no. One of the hardest things is to change your behavior, shift from “push mindset” to “pull mindset”. And that may be tricky.

You have Kanban board and there are no free slots in “In Progress” column. It means new stories should not be started — you’ve reached the limit. Developer comes and says “I know that there are no free slots, but I want to start this new small user story, since all the people are already working on the items and my help is not required”. You might think “Hmm, indeed we are moving forward with a good pace, this small story should not take much time…” and you say “OK, John, take it. I don’t think this would be a problem to exceed the limit right now.”

Two hours later Pete comes to you with a similar question: “I just completed my user story, testers will take it soon and I want to start something new.” You say “Well, we just took one story above the limit…”. Pete replies “Please, boss, I don’t want to sit and do nothing, I want to add some value, I am so much into work today!” Can you say no? I bet in most cases you say something like “OK, let’s do it!” to not discourage your teammate.

What happens then? Testers find some bugs in just implemented user stories, but there are no free developer to fix them. John and Pete are working on new stories. They may switch to bugs, but they’d lose focus and you have 2 stories in progress that can’t be passed to other developers easily (it will take time to make the transition and it looks like waste, isn’t it?). As a result, you deliver some stories later. Yes, later!

I understand that it is very hard initially to say something like “No, you’d better go and read something interesting about new technology. We can’t start new story right now, because if testers find bugs in some running stories, we should fix them ASAP, so we need a free developer”. Moreover, you should train your team to not ask such questions. WIP limit is a rule that should be followed. You may change it, but not arbitrarily. This change should have something behind (better productivity, more developers, etc). And this change should be discussed in retrospective meeting or during a Kaizen event.

I should say it is really, really hard to accept the fact, that sometimes developers and testers should do nothing (I mean do not start new stories when they are free). But it’s absolutely required to do so, otherwise pull system will not work.

We’re all tied together by things we do. Projects we work on, conferences we go to as a whole team, even bugs we fix together, problems we solve together. Insights we share together.

Some people call it team spirit. Some people call it good collaboration. In any case, this is the light of live communication and talk between people.  My deepest belief after all is that no matter how sophisticated telecom stuff gets our days, there’s nothing more valuable - and ROI-generating as well - as the real talk.

I was perplexed the other day as I saw the phrase  - “we’re trying to maximize our face time with them” - this was said about some guys from an off-site location visiting the main office. My first thought was - so the rest of the time they work  is their a** time? As obviously software guys spend most of their time sitting on their derrier?? Though of course it’s worth a praise that after all they’re still trying to maximize the face time..

As hype is the statement that with globalization you could get the best software developers out of anywhere in the world, as eternal is the truth that to produce people need to REALLY collaborate. This goes for software, for any production.  So it’s not about picking up people as vegetables on the market - even if it’s a huge market. People are more than vegetables. They need live communication to feel alive, to collaborate and to be productive (for those who want to count figures, you can sit down now and compare the monetary value of pure programming skills  - with no communication skills at all).

I’m not saying that we should all go local in our quest for best products and profits. But what I really see is that people are squeezing their way to follow some of agile communication practices using telecom. As a proof, look at hot discussions at LinkedIn agile groups, and Jean Tabaka’s observations on the reasons of agile adoption failure.

Blasphemy to the religion of remote/distributed teams - but the picture is slowly starting to get clear for me:  it really takes good skills to balance the equilibrium of infrastructure and management to activate collaborative agile work in remote teams. Meaning 5 here, 7 there, 6 elsewhere - as one team.

Is it possible to trust without seeing each other? I doubt there’re teams who are absolutely OK with remote customers/product owners etc. I even suspect that waterfall can be the best solution for remote teams-customers that haven’t developed enough trust to each other. If anyone has real-life stories to prove that I’m wrong, speak up.

Everything has its life-cycle. Even stars. Even The Universe. Everything.

It is quite sad for intelligent creatures, but it is just a fact of life. I want to draw some quite obvious parallels to reveal the real danger of Technical Debt for a software product.

Humans

Let’s take a human. When we are young, we are overflowed with energy. Do you know how a 7 year-old boy spends his day? I have a son, so I’ll tell you. He wakes up at 7 am (sometimes before 6 am!) and tries to play some quite silent games (it is too early to make much noise). He can restrain himself till about 8 am, then he starts to jump, does somersets,  some boxing etc… By 8.30 we have a real “tornado” in our house. It may take him an hour to exhaust and have a breakfast. Such periods of activity can be easily repeated 2-3 times a day. And he literally turns off around 9 pm - sleep is when he recharges his batteries.

An old man is quite the opposite. He saves every movement, everything is hard and slow. He makes many mistakes and health maybe getting worth. Learning new things is also getting harder at times.  Yes, wisdom and tremendous life experience is there,  but very often there is no energy to move forward.

Software products

Software products have exactly the same behavior. Young product is energetic. It has an excited development team behind. It moves forward quickly and feels warm sun on the top. It learns lightning fast and grows on new knowledge. Everything is smooth and shiny. Time to market is everything and tradeoffs are so common! Let’s fix it fast and mark with //TODO: refactor this comment. Let’s hack this framework here to release another cool new feature this week. Let’s re-invent the wheel and write some custom javascript code. Gosh! We don’t have time to learn this new cool javascript framework! These are quite common patterns for young and energetic products.

But what’s next? Years later progress stops. Product becomes old with all the bad and good side-effects. Suddenly it is much harder to add new features. Ages of development make the product more complex. Millions of technical debts rotten its body like cancer . There is no clear architecture anymore and there are so many patches that one small change can produce a totally unpredictable impact and bring along new bugs in unexpected areas. Yes, development team has wisdom. But often no courage and energy to revive the product.

Technical Debt Is a Cancer

How long does this cycle last? It really depends. We all visit doctors. Most of us want to live longer and healthier. Early cancer detection gives good chances to fight and win the disease. We should do exactly the same with software. Here are typical symptoms:

• Velocity has dropped significantly  over the last several iterations/releases
• A bug fix triggers one or several new bugs too often
• Nobody knows ideas behind the original software architecture
• Team spends more time fixing bugs than developing/improving features
• Automatic tests are red 80% of the time (if it is 100% — the product is most likely in coma)

If you see at least 2 such symptoms, you’ve just discovered a product cancer — Technical Debt. Technical debt is a true killer when you have deadline (time to market). If you have the symptoms, you should fight the disease right now. You may think that it is OK to wait  several months, add some more “cool and highly requested” features and then get back to the real problems. It is a wrong decision, believe me. I used to make it and I used to fail with it. It bogs you down. You lose focus and make stupid mistakes. It leads to fear. And fear is a bad ally. You go from one extreme to another to  only increase entropy, nothing else.

If you miss the deadline, all the possible actions will not help. Cancer will win. And then you will have just two options: re-write the product completely from  scratch or start a new product.

If you see these symptoms, you should stop and think about the attitude of your development team. If you’ve survived over several years, priorities should be changed. Reset your development team and use chemotherapy.

• Focus on quality. Fix the roots of the problems.
• Teach the original architecture to all, pair program, communicate.
• Introduce “No new code without tests” rule.
• Fight fear. Let the knowledge spread. Knowledge eliminates fear.
• Put the most experienced people on fundamental problems solving.

You have to fight the cancer to bring energy back, to bring courage back, to live and produce a great software product. Otherwise it will be as dead as Lotus Notes.

Once upon a time there lived a very brave and adventurous young man by the name Arthur. He lived in a large kingdom with knights and castles and an extremely cunning king who had a beautiful daughter - Caroline. She was so beautiful, that every morning birds would fly through the open window with fresh roses in their beaks. Birds would drop rose petals near her bed and fill the air with pleasant tweets, every five minutes exactly. Then they would fly away to never come back…they would simply die outside, because they couldn’t live without princess’ beauty anymore.

The king was greedy so he held annual competitions during which the contestants had to pay 10 sovereigns each to participate. The winner would marry Caroline, but missions were so hard, that there were no winners ever!

New competition was announced and brave Arthur decided to win the highest prize! He saw the beautiful princess only once before and fell desperately in love with her.

On a sunny day all the contestants gathered together on a central square. The king said: “Brave men! There will be 3 main missions… and a very easy 4th mission to the winner. Today I’ll tell you about the first mission only. You have to build a house in 10 days. It should have 2 rooms, doors, windows and a roof. That’s all you have to do!!”

Mission #1

Is it possible to build a house in 10 days? Arthur was disappointed. He had hoped to kill a dragon, catch a witch or do something else that a brave knight is supposed to do. But he was smart (and brave). He spent all the day doing nothing but thinking. Other contestants laughed at him: “Look at Arthur! He gave up miserably! Arthur, go home, you are such a loser!” But as you know, our Arthur was smart so he decided to build the simplest house possible. And he did it: the ground became a floor; house had only 2 tiny rooms and the light would enter through the 2 windows near the front doors; the roof was there to simply protect from rain and snow - no more no less! Most importantly, he finished it in 8 days! Many, many other brave men tried but failed. Some started to build a large and solid house and ran out of time. Others started building too quickly and their houses would collapse half way through. However, brave Arthur and a dozen other guys completed the first mission successfully.

“I am proud of you! - said the cunning king. “Now here is the second mission for you all. This time, you should build a basement that will be twice as large as the house itself. Also you should build a fireplace. You all have a total of 12 days to get the job done.”

Mission #2

And again Arthur spent the whole day thinking. “I don’t even have a floor in the house… How can I build a basement? The house will simply break down… If I spend time on the floor, I’ll have no time to do the rest.” Then he found a workaround. He decided to dig deeper than usual. “Deep basement should be safe enough and floor is not required in this case! Fireplace will take 2 days to create, and I can definitely make this basement in 9 days!”

Arthur worked like crazy and completed the mission in 11 days. He slept for the whole 12th day – the last day of the mission. At the end, only 5 other brave men have completed this stage of competition.

“You did a fantastic job!” – the king said. - “Now listen to the last mission. You should build an upper level in your house. An upper level may have only one room, but you have only 5 days to do it.”

Mission #3

All the contestants were quite excited. It looked like an easy task. But Arthur was sad. “If I build another level” - he thought - “my house will crash…” This time he started to work immediately. He brought several large logs into the basement and made props. It took him a total of 2 days. Arthur was not sure that it’d be enough, but he had no more time and decided to accept the risk. Luckily, he had a flat roof and therefore it turned out to be relatively easy to create another level right on top of it. Arthur was tired, but extremely happy. After all, he was the only one who made it through the last mission!

“My boy!” - King happily hugged Arthur. - “You did it! I can’t believe, but you did it! You are just one step away from the Caroline’s hand. Your last mission can’t be simpler - You should live in the house you built during the first 2 weeks. This is it. And then you will have my blessings and the hand of my beloved daughter, Caroline!”

“Wow, so easy!” - said Arthur. “Yes, it is” - answered the king with a smile on his face.

Mission #4

The next day brave Arthur was killed by the crashed ceiling. It was a windy and rainy day…

It is always interesting to look behind the scene. In previous post I  described our development process, now I will show how we use Kanban Board in TargetProcess. Maybe it will give you some new ideas to try.

The most interesting part of every Kanban Board is its flow. So here is it:

Planned: contains all stories and bugs that are ready for development. It means they have full details/specification, so developers may start working on any item from this list anytime. Items are prioritized in this columns, first item should be taken first. Only product owner can put stories and bugs into Planned state from Backlog.

WIP Bugs: this is quite a special column. When testers find bug in just coded user stories, these bugs are drag-and-dropped in WIP bugs column. It means developers should take them ASAP to make this story Done-Done ASAP. Sometimes we put important bugs from production here as well. This state was introduced just several weeks ago. Works fine.

In Progress: shows stories and bugs that are in development now. When development pair takes a new story, the story moves here.

Coded: shows that story is implemented by developers. It is a queue for testers in fact.

Testing: shows stories that are in testing now.

Ready to Merge: we implement each story into a separate branch. If user story is in this state, it means it is implemented and tested, so may be merged into master. Every day someone merges all the items from this column.

Merged: this column contains all the stories and bugs that will be included into the next public build/release. We normally do weekly releases.

Done: shows several items included into the latest build.

We have a large TV in the office that displays Kanban Board all the time. Daily meetings are near the TV, so everybody can see it and understand what we are talking about.

I’ve started using BDD to create user stories specifications several months ago. It works great. Very good format. Developers understand it without problems, Testers can write acceptance tests quickly and it is easy to follow for Product Owner.

Here are some examples of real user stories specs in BDD for TargetProcess product:

As a Scrum Master I want to see Release BD Chart drawn by weeks

As a Scrum Master
I want to see Release BD Chart drawn by weeks
When Iterations practice is disabled
So that I can benefit from BD chart

Given any development process
When I turn off Iterations practice in Admin -> Process -> Edit
And navigate to Release BD chart
Then iteration velocity replaced by weekly velocity
And Chart end date is the same as Release end date
And BD chart drawn by weeks instead of iterations
And Chart Start Date is the same as Release Start Date

As a Scrum Master I want to see Lead/Cycle time progress

As a Scrum Master
I want to see Lead/Cycle time progress
So that I know whether we are improving our development process or not

Scenario #1
Given Reports section in project and Bug Tracking practice is disabled
When I navigate to Lead and Cycle Time Report
Then I see Lead Time chart
And chart contains 1 line for stories

Scenario #2
Given Reports section in project and Bug Tracking practice is disabled
When I navigate to Lead and Cycle Time Report
Then I see Cycle Time
And chart contains 1 line for stories

Scenario #3
Given Reports section in project and Bug Tracking practice is ENABLED
When I navigate to Lead and Cycle Time Report
Then I see Lead Time chart
And chart contains 2 lines (for stories and bugs)

Scenario #4
Given Reports section in project and Bug Tracking practice is ENABLED
When I navigate to Lead and Cycle Time Report
Then I see Cycle Time
And chart contains 2 lines (for stories and bugs)

Additional Info
X-axis - Months (display 12 last CALENDAR months)
Y-axis - Lead time or Cycle Time (in days)

To display value for given months, we take stories completed this month (determined by End Date).
For example, in June we’ve completed 5 stories, in July 10.
Then lead/cycle time will be calculated for June by sum(5 stories lead time)/5 and for July sum(10 stories lead time)/10.

Charts should have labels for each month.

Story line color: #507cb6
Bug line color: #cc060d