Edge of Chaos | Agile Development Blog

Kai Gilb started a series of posts that challenge Agile Software Development. Part 2 is about creativity and “how well” focus in software development. While I agree with the problem, I strongly disagree with the solution. Kai recommends to write requirements (the format he uses doesn’t allow me to call it “user stories”) in a form that does not contain solution:

Developers Are Not Product Owners

Then developers should creatively find the ways to implement this requirement, to invent the solution and to solve the “how well” puzzle. We’ve got a huge problem here indeed. Most developers can’t invent solutions at this high level. They can create beautiful architecture and build the quality in. They can write tests upfront and use powerful tools to speed up development. They can do many things, but often they can’t invent a “how well” solution that will make customers happy. Why?

1. Most developers know nothing about User Experience (and don’t care).
2. Most developers can’t create usable design.
3. Many developers barely know a problem domain and in any case don’t want to dig into much details here.

In short, most developers are engineers (so far?), so what Kai proposes will not help. It is just a theoretical solution to “how well” problem. It’s the same like trying to grow 4 hands and 2 heads to double your productivity (well, sacrificing beauty maybe…)

In Scrum Product Owner is responsible for writing User Stories. It means he should invent solutions and care about “how well” thing.  I don’t see a significant problem here. Of course, it’s great if developers are interested in requirements handling and solutions investigation process, but if they are not, Product Owner can do that alone or with special people who care. Who they can be?

Those people are UX specialists. They know how to solve “how well” problems effectively. They know many things about design patterns and all new trends in interfaces. They like to explore problem domain and talk to customers. If you have such people among developers — you are SO lucky! But often this is not the case.

We, at TargetProcess, try to instill UX culture into development process. We started in August, now it’s February and I can’t say we’ve had much success. Sure there are significant improvements, but to be honest most developers are not so interested in UX staff, they like BDD, DDD, C# and SOA more. I expect it will be a long process that will take years. And our company is quite small. I can’t even imagine an effort for such a radical shift in a huge company.

Broad Requirements Are Not Testable

I wonder how you can write BDD scenarios for broad requirements as Kai suggests. How the hell can you test anything, if you don’t have any clue on how it should work? So broad requirements can’t replace user stories obviously. They can live in backlog, but developers should deal with concrete requirements that are testable right away.

I can agree that you can split backlog in 2 parts. The first small part contains detailed stories that can be put to development asap. The second large part contains broad requirements without solutions. That might be a good idea. But we should clearly separate UX phase and Development phase. During UX phase we should solve “how well/how cool” problems, and during development phase we should solve technical problems only (with some corrections in “how well” for sure if required).

P.S. Kai, I hate gray text on gray background in your blog. It is not user friendly.

In previous post I mentioned 5 wrong reasons to apply Kanban. Tobias Mayer asked to blog about 5 right reasons Obviously there should be right reasons, otherwise Kanban adoption would fail everywhere. So there they go:

#1. Ability to release anytime

In Scrum or XP you can’t release in the middle of an iteration. In Kanban you may release anytime. When user story is ready, you may release it. Definitely it is a challenge to setup development process this way. It is required to have “branch-by-feature” source control policy, merge often, integrate and test often. But it gives you an ability to release often. That is something worth to fight for.

As a PO I like this ability very much. An important user story implemented? OK, let’s release it tomorrow in v.2.15.2. Our customers may benefit from this release as soon as possible. Lead time is faster — customers are happier.

One thing to mention — you need to have a good automated test suite, otherwise it will be problematic to make small releases with a good quality.

#2. Ability to change priorities on the fly

In Scrum you can’t add stories on the fly into sprint (usually). At least it is not an easy process and development team often resists to replacing a story from Sprint backlog. The story has been discussed, estimated etc. A new story may be discussed in a hurry, some details will be missed and as a result significant re-work will be required. So in general iteration or sprint should be frozen.

In Kanban if you have an urgent request to implement or a really important user story, you can just put it on top of the queue. It will be taken as soon as a free slot is available. It is simply a Product Owner’s dream

#3. No need in iterations

Why you need iterations? Initially iterations help to reveal real problems in the development process quickly. Further on they establish a project rhythm and rituals. However at some point in project I think you no longer need them.

My vision is that you need to iterate first, then flow. Our backlog is fuzzy now, plans change often. We learn how to do agile in general and iterations are not helpful there anymore. Without iterations there is no need in iteration planning and iteration demo meetings. They are waste. Instead we have more smaller just-in-time meetings with people in a feature-team before starting the development for each user story.  It works perfect.

Some people are missing the rhythm, but I think it is more to habit. Kanban establishes more complex rhythm and it may take time for development team to recognize it. Still stable rhythm may be the only reason to keep iterations in the late phase of product development, in my opinion.

#4. No need in estimates

Why you need estimates? In iterative development you need them to predict how many stories you may take in the next iteration. You may even predict a release date based on estimated backlog and iteration velocity. The other reason is that PO wants to know how big the user story is. If it is big enough, PO  may consider moving it to the next release. If it is small, PO may decide to put it into the next iteration.

Obviously, iterative development is hardly possible without estimates. But if you drop iterations, it is not a problem anymore. Can Product Owner live without estimates? Well, yes. In our company we don’t estimate user stories. Why not? Because I as a PO don’t need them and we don’t use iterations. All I may ask is a very rough estimate like  normal, large, very large.

In our situation (I stress that, in our situation), estimates are waste. We don’t spend time on estimates. We have a prioritized backlog and just take the most important user story and implement it. It may not work in some conditions, but we have quite a mature product and hundreds requests from customers. There is no clean field development, so normally we don’t have any defined release date. We release when it is ready (I understand that many companies can’t do that, but we have this favor).

#5. Perfect flow visualization

Kanban Board provides a very clear view on current work in progress. It visualizes flow and enables fast planning and tracking. It is really a great tool, we love it.

Many complex systems are based on simple rules. A set of several simple rules leads to complex, intelligent behavior. While a set of complex rules often leads to a dumb and primitive behavior. There are many examples.

Ants Colony

How ants search for food? They do not have cell phones, cars and mini-markets near the nest. They should have something simpler to communicate.

Here is how ants work:

1. Travel randomly  in search for food.
2. Take a piece of food and head straight back to the nest. On the way back to the nest lay down an odor trail.
3. Notify nestmates of the discovered food encouraging them to leave the nest. These newly recruited ants will follow the odor trail directly to the food source. In their turn, each ant will reinforce the odor trail until the food is gone.

Sounds simple? Take a look at this very nice ants colony model. Drop some food and enjoy the action.

Birds Flocks

Birds flocks are beautiful. You may think that the movement gets orchestrated by one savvy bird. But this is not the case. A bird flock is guided by three simple principles (every decent bird knows them):

1. Separation: steer to avoid stumbling upon local flockmates.
2. Alignment: steer towards the average heading of local flockmates.
3. Cohesion: steer to move towards the average position of local flockmates.

Simple? Yes, it is. Look at the picture on the right. It’s just amazing!

Game of Life

Game of Life was invented in 1970 by John Conway. It is a cellular automaton and simulates the birth, death, etc., of organisms based on certain rules:

1. Each cell with one or no neighbors dies, as if of loneliness.
2. Each cell with four or more neighbors dies, as if of overpopulation.
3. Each cell with two or three neighbors survives.
4. Each empty cell with three neighbors becomes populated.

Simple rules. But these rules lead to fantastic diversity of the forms. Different types of the forms have been discovered e.g.  still objects, oscillators, gliders, spaceships, etc. It is impossible to predict the state of a system after several thousands steps. Cellular automaton has properties of a complex system such as emergency and butterfly effect. Small changes in the stable structure (for example, adding one more living cell) may cause death of the whole cells population.

Moreover, it is possible to build a computer based on Life! It is possible to implement logic based on stable structures and execute simple calculations. The Game of Life is a Turing machine! How can someone suggest that such a simple system based on four rules has so much power?

Take a break and have fun with Life game simulation.

Simple Rules and Software Development

Is there any relation between simple rules and software development? Sure, there is. Software development process should be simple. Process complexity leads to mechanistic and dumb behavior.

1. Information exchange and collaboration vs. standard status reporting meetings.
2. Learning vs. stagnation.
3. Emergent behavior and creativity vs. Following rules, standards, instructions.

Agile methodologies are simple. Scrum is a very simple thing. It has just several rules, roles and artifacts. Well, it is a lot harder to really implement Scrum. It is hard because you need to break stereotypes and habits. Many people are resistant and don’t want to try new things. However, Scrum works. It works because of its simplicity, it lives in accordance with complex systems.

Scrum stimulates learning, feedback, communication and cooperation. Emergency is possible in Scrum.

Here is one sample of unnecessary complexity — too many hierarchy levels:

A potential problem, unlikely in small and medium organisations, is deep organisational structures. According to Peters and Waterman (1982)18, both Toyota and Roman Catholic Church have only five layers of management in comparison to Ford’s seventeen.

Do you by any chance happen to have a software development process description in a huge 100+ pages document? Are you still in business?

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

• Is there an alternative to Scrum of Scrums? Will Read proposes Mesh network as a Scrum of Scrums replacement. Looks interesting as an information exchange flows in an organization.
• After The Crisis: A Parody of 15 Corporate Logos. Apple logo is very funny
• Jeff Atwood thinks that web application should not follow desktop application design. “A web app that apes the conventions of a desktop application is attempting to cross the uncanny valley of user interface design. This is a bad idea for all the same reasons; the tiny flaws and imperfections of the simulation will be grossly magnified for users.”
• Herb Sutter found two fallacies in Jeff’s post and think that this is a natural evolution. “Today, everyone writing rich Web 2.0 applications is doing their own thing, borrowing as best they can from Macs and Windows and others — but the results are all over the map, and will continue to be until there actually is such a thing as a UI standard for rich-GUI web applications.”

I tend to agree with Herb. I don’t feel bad when using web based app that looks familiar. And I definitely agree that in the future we will have several common frameworks for rich UI applications development with quite similar paradigms and interface concepts.

• New ASP.NET Charting Control. Looks good, but I prefer flash charts.
• Trashing Scrum or Reflecting Reality? David J. Anderson thinks that Scrum in it’s current state is almost opposite to agile development “My observation from the outside is that the Scrum community reflects the antithesis of our agile values. It is run from the top. The message is strictly controlled. Dissent is not permitted. It resembles a cult of personality and appears to be the very definition of command’n'control in its execution”.
• Is Scrum Failing Us? Alan Shalloway does not have such extreme feeling, but believe that there are many misunderstanding in Scrum “I want to mention some misunderstandings I have seen many people in the industry have and how we, as practitioners, can get beyond them. I have also included a few things many Scrum trainers believe that I don’t agree with”.

In the previous parts I’ve described basic properties of complex adaptive systems and it’sbeen shown that software development is a CAS. In my next few posts I am going to describe CAS properties in depth and apply them to software development. In this part we will look into the Edge of Chaos.

Many researchers tend to think that CAS may work efficiently at the edge of chaos. A system with a very high order can’t solve complex creative tasks, as well as a pure chaotic system. It seems the system should balance between chaos and order to survive and adapt.

The “edge of chaos” term was emerged during self-reproductive cellular automata researches in 1990. It appeared that at a particular noise level the system can reproduce its state, while at a low noise levels states are random. This edge value was mathematically analyzed and it was shown that in such a state the system has a maximum of information.

The “Edge of Chaos” term  spread and applied to complex adaptive systems. For example, we may suggest that evolution drives systems to the edge of chaos, and it is one of the reasons of life origin. Life is a very interesting and complex thing in itself. Most likely it could not appear neither in order state and in chaos state. Life exists if there is a stable base, but with possibilities of changes.

Edge of Chaos and Software Development

How can we apply the edge of chaos concept to software development? Obviously, edge of chaos is a state where the development team works with maximum efficiency. Complex processes and explicit rules impede creativity. The team becomes too ordered to think :). On the other hand, no processes and no rules lead to chaos. The development team can’t complete anything useful, and that is something we call hack & fix process.

As a software development folks, we are particularly interested in two questions:

1. How can we drive the team to the edge of chaos?
2. How we will ensure that the team is at the edge of chaos?

The human who knows the answers to these questions most certainly will be rich and famous. However I am not sure whether you can find such a man.

It is obvious that edge of chaos = hyper productivity. This term often used in Scrum. However it is quite hard to define hyper productivity.

Well, currently we may discover hyper productive teams and identify their properties. But if we try to apply CAS to the edge of chaos, we may define the following properties that may lead a development team to hyper productivity:

1. Feedback
2. Information exchange
3. Cooperation
4. Self-organization
5. High Adaptability

Agile software development processes have some of these properties explicitly, and some implicitly. For example, feedback, information exchange and adaptability are roots of agile development. Self-organization is something most are expecting to have in an agile process, but sometimes it is skipped. I think it is a huge fault if, for example, Scrum Master does not include self-organization into development process. All these properties are important and must have to build a highly efficient team. If you have 4 of them, but missed the last one, you do not have hyper productive team.

Clinton Keith identified four important conditions for hyper productive teams:

Let’s try to bind these conditions to CAS properties defined above. Independence is one of the self-organization pre-requisites. Leadership on my opinion is not a must, but likely is an effect of self-organization. The development team got a leader as a result of natural selection Core competency is a result of a feedback, learning and adaptation. The last condition obviously is an information exchange and cooperation.

We may easily define conditions, that block hyper productivity:

• No feedback. That is one of the reasons for waterfall process failure. In a waterfall process feedback is delayed and sometimes even does not exist. Team can’t learn and adapt. Efficiency is low.
• Poor information exchange. Large teams, distributed teams, huge functional divisions, interpersonal problems - all that impedes information exchange.
• No cooperation. Political games inside a company or inside a development team, dumb corporate rules, wrong motivation schemes, communication problems due to personal likes and dislike - just some factors that make cooperation almost impossible.
• Impossibility of self-organization and weak adaptability. Complex corporate rules, hard hierarchy, command and control management, no autonomy make self-organization impossible.

We’ll cover these problems in more details later. Even in such a brief overview it’s getting clear what the main responsibilities of CTO, CIO, PM, Scrum Master, etc. should be. They should remove obstacles listed above (and more) that block a trip of the development team to the edge of chaos.

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

“Are we agile?”, “How agile are we?”, “Are we more agile than they are?” Honestly, I am getting tired of these questions. Why do you care? Will it make you happier when you are able to tattoo “100% agile” on your body? Is it a goal to be “agile”? Definitely not! All agile tests are just a garbage. All efforts on agile process certifications/assessments are useless.

There are so many factors influencing software development process that make impossible any certification. Your company is special, you have special people in the development team, you have special conditions, rules, and other external factors. CMMI, PMBOK, and other heavy approaches do not help to build legendary team. They only may help to build an average team and eliminate some quite obvious mistakes in the development process.

The right question to ask is: “How can we be more productive as a team?”. It is your project. It is your team. You have goals to improve team productivity (”Done-Done” stories in a period of time), create outstanding software and make customers happy. Agile Tests that answer question “Are we agile?” shift the focus to the wrong direction.

Look, if your team has to pass an Agile “test”, they will focus on passing it. That is a plain dumb goal and a waste of time. Let’s say a manager reads about a famous agile test and sets the goal to pass it. Development team, as a complex adaptive system, adapts to the rules and environment. It will change development process and apply practices to pass the test as effectively as possible. However the team’s productivity may suffer. Why? Simply because the test is too general and can’t be applied to any team. Remember, your team is special.

Let’s take Scrum and famous Nokia test. The first question is “Describe your iterations” and the worst answer to the question is “We do not use iterations”. Well, it is a Scrum test and Scrum insists on having iterations (sprints). But what if your team works more effectively without iterations? What if your team implements “by-feature” using Kanban? What if you have a large project and long iterations are a relief? What if your team holds retrospective meeting and decides to use iteration-less development? Hey, you will fail your “agility” test! Will you be less agile? Why do you care? You will be more productive, that is what you need.

Another example. Third question in Nokia test is “Describe your requirements at the time an iteration starts”. The best answer according to Nokia test is “We have good user stories tied to agile specifications as needed”. What the hell agile specification is? I don’t know such a term. Is it “just enough” documentation? Is it executable specification? Who knows… Maybe your team is fine with just user stories and produces great results. But you will receive less points on the test.

The only good “test” I’ve ever seen is from exceptional Alistair’s book. It contains seven properties of successful Agile development projects. Even in this very broad test some properties may not improve your productivity. So I can’t imagine how it is possible to create general test that will work for all teams/projects/environments/etc. It is as achievable as perpetual motion machine…

Anyway, my point is to focus on real goals and throw away fake goals to pass an agile test, be “100% agile” and “agilier” than the team in a next cubicle.

Here is the very good quote:

From my readings of the literature on Japanese management practice, the focus appears to be on the “hows” and “whats” rather than why something should be done. Although there is some mention of “why”, this appears to be almost incidental. This may be a result of the cohesive nature of Japanese society, the lack of industrial conflict that was endemic in the UK and many other Western countries, or a product of Japanese management thinking.

I do agree. We often fall into a set of practices, tools and “how-to” solutions. We often take Scrum, or XP, or any other process and apply it in hope to solve all the problems. We don’t so often ask “why” indeed. Why we need to change our software development process? It is very important to develop a “need for change” message for your team, your product, your company.

People often ask questions like “What is the difference between Scrum/XP/Lean/Agile/You name it“? I don’t see much difference between XP and Scrum on that matter. If you already have XP, most likely don’t need to switch. Maybe adopt some Scrum specific practices for a better scalability like Scrum-of-Scrums. Almost all the other practices like daily meetings, iterations, roles separation, retrospectives, etc. exist in XP already.

In fact I am not sure if such separation brings benefits. It is better to spot what’s going wrong in your development process during retrospective meetings and apply practices from any agile process (or create own solutions) to your specific problems. XP, Scrum or Lean give you a framework that helps to be adaptive and creative, whereas traditional processes give you a set of rules hindering any creative behavior.

Your team and your project IS special. Think and communicate to sharpen your development process.

I do understand why people are so enthusiastic about Scrum or XP. These processes provide tools and practices out of the box. You may just start doing it and see what happens. It is not the best way, since quite often agile adoption fails. People don’t get the new way, they implement it without proper support and under the pressure. They maybe don’t have a deep understanding how it should work. So it is expected that agile adoption might not be smooth for all.

Agile Development Future

I wish all agile methodologies evolve into one elegant, smart, simple framework. For me it is a natural trend and it should be something like Theory of Everything in physics. This framework should explain why agile processes work. It should based on natural laws of Complex Adaptive Systems. It should provide general guidance on agile adoption and a repository of possible practices that might be used to build your own development process. The framework should be simple and unobtrusive, since complex rules will not work for software development. The framework should empower creativity, communication, learning, feedback and self-organization. The framework should support nonlinearity, emergence and edge of chaos concepts.

Software development is young. Currently we are having transition from waterfall to agile. It is mainly based on practice rather than theory. Practice has proven that agile works better. However, there’s almost no research explaining why. There are some blog posts and short explanations with reference to Complexity Theory. That is a right direction I think, but there’s no depth behind these posts, they are quite superficial.

I expect we will have deeper understanding of the agile roots and why it works. This knowledge will simplify transition and help us to find better practices, patterns and solutions. I expect there will be a unified agile software development framework in the future.

So far Scrum and Lean are closest processes to this framework. Lean has a strong philosophy and nice background. It is general enough. Scrum is simpler and more understandable for mere mortals. XP has a set of best practices and heavier in general. All of them have strong and weak sides. But still I feel there’s a deeper level and all modern processes are just a subset of some hypothetical (at the moment) unified agile software development framework.