Edge of Chaos | Agile Development Blog

• Is there an alternative to Scrum of Scrums? Will Read proposes Mesh network as a Scrum of Scrums replacement. Looks interesting as an information exchange flows in an organization.
• After The Crisis: A Parody of 15 Corporate Logos. Apple logo is very funny :)
• Jeff Atwood thinks that web application should not follow desktop application design. “A web app that apes the conventions of a desktop application is attempting to cross the uncanny valley of user interface design. This is a bad idea for all the same reasons; the tiny flaws and imperfections of the simulation will be grossly magnified for users.”
• Herb Sutter found two fallacies in Jeff’s post and think that this is a natural evolution. “Today, everyone writing rich Web 2.0 applications is doing their own thing, borrowing as best they can from Macs and Windows and others — but the results are all over the map, and will continue to be until there actually is such a thing as a UI standard for rich-GUI web applications.”

I tend to agree with Herb. I don’t feel bad when using web based app that looks familiar. And I definitely agree that in the future we will have several common frameworks for rich UI applications development with quite similar paradigms and interface concepts.

• New ASP.NET Charting Control. Looks good, but I prefer flash charts.
• Trashing Scrum or Reflecting Reality? David J. Anderson thinks that Scrum in it’s current state is almost opposite to agile development “My observation from the outside is that the Scrum community reflects the antithesis of our agile values. It is run from the top. The message is strictly controlled. Dissent is not permitted. It resembles a cult of personality and appears to be the very definition of command’n'control in its execution”.
• Is Scrum Failing Us? Alan Shalloway does not have such extreme feeling, but believe that there are many misunderstanding in Scrum “I want to mention some misunderstandings I have seen many people in the industry have and how we, as practitioners, can get beyond them. I have also included a few things many Scrum trainers believe that I don’t agree with”.

In the previous parts I’ve described basic properties of complex adaptive systems and it’sbeen shown that software development is a CAS. In my next few posts I am going to describe CAS properties in depth and apply them to software development. In this part we will look into the Edge of Chaos.

Many researchers tend to think that CAS may work efficiently at the edge of chaos. A system with a very high order can’t solve complex creative tasks, as well as a pure chaotic system. It seems the system should balance between chaos and order to survive and adapt.

The “edge of chaos” term was emerged during self-reproductive cellular automata researches in 1990. It appeared that at a particular noise level the system can reproduce its state, while at a low noise levels states are random. This edge value was mathematically analyzed and it was shown that in such a state the system has a maximum of information.

The “Edge of Chaos” term  spread and applied to complex adaptive systems. For example, we may suggest that evolution drives systems to the edge of chaos, and it is one of the reasons of life origin. Life is a very interesting and complex thing in itself. Most likely it could not appear neither in order state and in chaos state. Life exists if there is a stable base, but with possibilities of changes.

Edge of Chaos and Software Development

How can we apply the edge of chaos concept to software development? Obviously, edge of chaos is a state where the development team works with maximum efficiency. Complex processes and explicit rules impede creativity. The team becomes too ordered to think :). On the other hand, no processes and no rules lead to chaos. The development team can’t complete anything useful, and that is something we call hack & fix process.

As a software development folks, we are particularly interested in two questions:

1. How can we drive the team to the edge of chaos?
2. How we will ensure that the team is at the edge of chaos?

The human who knows the answers to these questions most certainly will be rich and famous. However I am not sure whether you can find such a man.

It is obvious that edge of chaos = hyper productivity. This term often used in Scrum. However it is quite hard to define hyper productivity.

Well, currently we may discover hyper productive teams and identify their properties. But if we try to apply CAS to the edge of chaos, we may define the following properties that may lead a development team to hyper productivity:

1. Feedback
2. Information exchange
3. Cooperation
4. Self-organization
5. High Adaptability

Agile software development processes have some of these properties explicitly, and some implicitly. For example, feedback, information exchange and adaptability are roots of agile development. Self-organization is something most are expecting to have in an agile process, but sometimes it is skipped. I think it is a huge fault if, for example, Scrum Master does not include self-organization into development process. All these properties are important and must have to build a highly efficient team. If you have 4 of them, but missed the last one, you do not have hyper productive team.

Clinton Keith identified four important conditions for hyper productive teams:

Let’s try to bind these conditions to CAS properties defined above. Independence is one of the self-organization pre-requisites. Leadership on my opinion is not a must, but likely is an effect of self-organization. The development team got a leader as a result of natural selection :) Core competency is a result of a feedback, learning and adaptation. The last condition obviously is an information exchange and cooperation.

We may easily define conditions, that block hyper productivity:

• No feedback. That is one of the reasons for waterfall process failure. In a waterfall process feedback is delayed and sometimes even does not exist. Team can’t learn and adapt. Efficiency is low.
• Poor information exchange. Large teams, distributed teams, huge functional divisions, interpersonal problems – all that impedes information exchange.
• No cooperation. Political games inside a company or inside a development team, dumb corporate rules, wrong motivation schemes, communication problems due to personal likes and dislike – just some factors that make cooperation almost impossible.
• Impossibility of self-organization and weak adaptability. Complex corporate rules, hard hierarchy, command and control management, no autonomy make self-organization impossible.

We’ll cover these problems in more details later. Even in such a brief overview it’s getting clear what the main responsibilities of CTO, CIO, PM, Scrum Master, etc. should be. They should remove obstacles listed above (and more) that block a trip of the development team to the edge of chaos.

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

“Are we agile?”, “How agile are we?”, “Are we more agile than they are?” Honestly, I am getting tired of these questions. Why do you care? Will it make you happier when you are able to tattoo “100% agile” on your body? Is it a goal to be “agile”? Definitely not! All agile tests are just a garbage. All efforts on agile process certifications/assessments are useless.

There are so many factors influencing software development process that make impossible any certification. Your company is special, you have special people in the development team, you have special conditions, rules, and other external factors. CMMI, PMBOK, and other heavy approaches do not help to build legendary team. They only may help to build an average team and eliminate some quite obvious mistakes in the development process.

The right question to ask is: “How can we be more productive as a team?”. It is your project. It is your team. You have goals to improve team productivity (“Done-Done” stories in a period of time), create outstanding software and make customers happy. Agile Tests that answer question “Are we agile?” shift the focus to the wrong direction.

Look, if your team has to pass an Agile “test”, they will focus on passing it. That is a plain dumb goal and a waste of time. Let’s say a manager reads about a famous agile test and sets the goal to pass it. Development team, as a complex adaptive system, adapts to the rules and environment. It will change development process and apply practices to pass the test as effectively as possible. However the team’s productivity may suffer. Why? Simply because the test is too general and can’t be applied to any team. Remember, your team is special.

Let’s take Scrum and famous Nokia test. The first question is “Describe your iterations” and the worst answer to the question is “We do not use iterations”. Well, it is a Scrum test and Scrum insists on having iterations (sprints). But what if your team works more effectively without iterations? What if your team implements “by-feature” using Kanban? What if you have a large project and long iterations are a relief? What if your team holds retrospective meeting and decides to use iteration-less development? Hey, you will fail your “agility” test! Will you be less agile? Why do you care? You will be more productive, that is what you need.

Another example. Third question in Nokia test is “Describe your requirements at the time an iteration starts”. The best answer according to Nokia test is “We have good user stories tied to agile specifications as needed”. What the hell agile specification is? I don’t know such a term. Is it “just enough” documentation? Is it executable specification? Who knows… Maybe your team is fine with just user stories and produces great results. But you will receive less points on the test.

The only good “test” I’ve ever seen is from exceptional Alistair’s book. It contains seven properties of successful Agile development projects. Even in this very broad test some properties may not improve your productivity. So I can’t imagine how it is possible to create general test that will work for all teams/projects/environments/etc. It is as achievable as perpetual motion machine…

Anyway, my point is to focus on real goals and throw away fake goals to pass an agile test, be “100% agile” and “agilier” than the team in a next cubicle.

Here is the very good quote:

From my readings of the literature on Japanese management practice, the focus appears to be on the “hows” and “whats” rather than why something should be done. Although there is some mention of “why”, this appears to be almost incidental. This may be a result of the cohesive nature of Japanese society, the lack of industrial conflict that was endemic in the UK and many other Western countries, or a product of Japanese management thinking.

I do agree. We often fall into a set of practices, tools and “how-to” solutions. We often take Scrum, or XP, or any other process and apply it in hope to solve all the problems. We don’t so often ask “why” indeed. Why we need to change our software development process? It is very important to develop a “need for change” message for your team, your product, your company.

People often ask questions like “What is the difference between Scrum/XP/Lean/Agile/You name it“? I don’t see much difference between XP and Scrum on that matter. If you already have XP, most likely don’t need to switch. Maybe adopt some Scrum specific practices for a better scalability like Scrum-of-Scrums. Almost all the other practices like daily meetings, iterations, roles separation, retrospectives, etc. exist in XP already.

In fact I am not sure if such separation brings benefits. It is better to spot what’s going wrong in your development process during retrospective meetings and apply practices from any agile process (or create own solutions) to your specific problems. XP, Scrum or Lean give you a framework that helps to be adaptive and creative, whereas traditional processes give you a set of rules hindering any creative behavior.

Your team and your project IS special. Think and communicate to sharpen your development process.

I do understand why people are so enthusiastic about Scrum or XP. These processes provide tools and practices out of the box. You may just start doing it and see what happens. It is not the best way, since quite often agile adoption fails. People don’t get the new way, they implement it without proper support and under the pressure. They maybe don’t have a deep understanding how it should work. So it is expected that agile adoption might not be smooth for all.

Agile Development Future

I wish all agile methodologies evolve into one elegant, smart, simple framework. For me it is a natural trend and it should be something like Theory of Everything in physics. This framework should explain why agile processes work. It should based on natural laws of Complex Adaptive Systems. It should provide general guidance on agile adoption and a repository of possible practices that might be used to build your own development process. The framework should be simple and unobtrusive, since complex rules will not work for software development. The framework should empower creativity, communication, learning, feedback and self-organization. The framework should support nonlinearity, emergence and edge of chaos concepts.

Software development is young. Currently we are having transition from waterfall to agile. It is mainly based on practice rather than theory. Practice has proven that agile works better. However, there’s almost no research explaining why. There are some blog posts and short explanations with reference to Complexity Theory. That is a right direction I think, but there’s no depth behind these posts, they are quite superficial.

I expect we will have deeper understanding of the agile roots and why it works. This knowledge will simplify transition and help us to find better practices, patterns and solutions. I expect there will be a unified agile software development framework in the future.

So far Scrum and Lean are closest processes to this framework. Lean has a strong philosophy and nice background. It is general enough. Scrum is simpler and more understandable for mere mortals. XP has a set of best practices and heavier in general. All of them have strong and weak sides. But still I feel there’s a deeper level and all modern processes are just a subset of some hypothetical (at the moment) unified agile software development framework.