Edge of Chaos | Agile Development Blog

Code base of a large project is getting worse over time. I hope there are lucky exceptions, but in general it is true for most projects. The reasons are quite obvious:

• More and more features. It leads to increased complexity.
• Shortcuts and hacks to support “We need this fancy search till August. Period!” features
• Developers rotation. New developers don’t know all the fundamental decisions and ideas behind the architecture. Knowledge gets lost with transition inevitably.
• Development team growth. More people - less communication. Less communication - bad decisions. It leads to code duplication, hacks to make something work without deep understanding of the underlying conditions etc.

Suddenly you can’t add new features easily, you can’t make significant changes easily, you have tons of technical debts and development team is close to bankruptcy. You want to change that and have just two options: refactoring or rewriting everything from scratch.

Refactoring and Punctuated Equilibrium

Punctuated Equilibrium is a theory in biology, but it is applied to other complex adaptive systems like societies as well. It states that the system has almost no changes in significant period of time and then changes very rapidly. Then it acquires the equilibrium state again. Sounds familiar? Sure, refactoring is almost the same.

Any change in the system introduces chaos in the short term. The goal of refactoring is to eliminate chaos, but often you increase it initially. When you are working on changes you make system less stable, but as the change is completed,  the system can be more ordered. That is not always true for software development. If you use branches,  this makes changes safer. Still significant changes increase the risk of new bugs.

The real danger of refactoring are local optimums. With a full rewrite you may have a better architecture than with refactoring. Solution? If you have a vision of final architecture, use it and try to make a path from current architecture to the new architecture. In general simplicity should emerge from refactoring.

Rewrite and Chaos

When you rewrite from scratch, you add such a large portion of chaos that it is hard to predict the final result. You have a new singularity that will explode to the new product universe. But are you certain that it will be better than the previous universe? How many the ’same bugs’ will you fix in the new product version?

However, rewrite may look faster. I mean you may release a new version faster with rewrite, but most likely with more bugs and less stable.

I think we may expect something like that:

The green line shows how chaos changes with refactoring. After each refactoring there is a small increase of chaos, but then the system becomes stable and chaos decreases. You see that the final release is quite late, but keep in mind that there have been many releases before, so customers benefit earlier.

Black line is how chaos changes with a full rewrite. We have the old system during rewrite, so chaos is constant. After the public release chaos increases significantly. Quite many new (and old) bugs and quirks are expected, so stabilization period is longer. But the release itself is faster. The reason is that there is no burden behind. For example, there is no need to support all the places when you do a change, while in refactoring it is required  to keep system working and stable all the time, and it demands additional effort.

Adaptation vs. Revolution

There is another angle to this problem. Refactoring is adaptation, while full rewrite is revolution. Again, revolution is a chaotic beast. You may slowly adapt the product for new external conditions or make one revolutionary rewrite.

So, Rewrite or Refactor?

I do think there is no universal correct answer to this question. If time to market is paramount, if you feel that you’d lose business if a new version will not be published in 6 months, you may try a full rewrite. But beware side effects! Quality may drop significantly and long stabilization period may hurt existing customers.

In most cases refactoring is preferable. Slow pace, high quality, constant improvements, happy customers. I like it more. But rewrite is so magnetic…

Everything has its life-cycle. Even stars. Even The Universe. Everything.

It is quite sad for intelligent creatures, but it is just a fact of life. I want to draw some quite obvious parallels to reveal the real danger of Technical Debt for a software product.

Humans

Let’s take a human. When we are young, we are overflowed with energy. Do you know how a 7 year-old boy spends his day? I have a son, so I’ll tell you. He wakes up at 7 am (sometimes before 6 am!) and tries to play some quite silent games (it is too early to make much noise). He can restrain himself till about 8 am, then he starts to jump, does somersets,  some boxing etc… By 8.30 we have a real “tornado” in our house. It may take him an hour to exhaust and have a breakfast. Such periods of activity can be easily repeated 2-3 times a day. And he literally turns off around 9 pm - sleep is when he recharges his batteries.

An old man is quite the opposite. He saves every movement, everything is hard and slow. He makes many mistakes and health maybe getting worth. Learning new things is also getting harder at times.  Yes, wisdom and tremendous life experience is there,  but very often there is no energy to move forward.

Software products

Software products have exactly the same behavior. Young product is energetic. It has an excited development team behind. It moves forward quickly and feels warm sun on the top. It learns lightning fast and grows on new knowledge. Everything is smooth and shiny. Time to market is everything and tradeoffs are so common! Let’s fix it fast and mark with //TODO: refactor this comment. Let’s hack this framework here to release another cool new feature this week. Let’s re-invent the wheel and write some custom javascript code. Gosh! We don’t have time to learn this new cool javascript framework! These are quite common patterns for young and energetic products.

But what’s next? Years later progress stops. Product becomes old with all the bad and good side-effects. Suddenly it is much harder to add new features. Ages of development make the product more complex. Millions of technical debts rotten its body like cancer . There is no clear architecture anymore and there are so many patches that one small change can produce a totally unpredictable impact and bring along new bugs in unexpected areas. Yes, development team has wisdom. But often no courage and energy to revive the product.

Technical Debt Is a Cancer

How long does this cycle last? It really depends. We all visit doctors. Most of us want to live longer and healthier. Early cancer detection gives good chances to fight and win the disease. We should do exactly the same with software. Here are typical symptoms:

• Velocity has dropped significantly  over the last several iterations/releases
• A bug fix triggers one or several new bugs too often
• Nobody knows ideas behind the original software architecture
• Team spends more time fixing bugs than developing/improving features
• Automatic tests are red 80% of the time (if it is 100% — the product is most likely in coma)

If you see at least 2 such symptoms, you’ve just discovered a product cancer — Technical Debt. Technical debt is a true killer when you have deadline (time to market). If you have the symptoms, you should fight the disease right now. You may think that it is OK to wait  several months, add some more “cool and highly requested” features and then get back to the real problems. It is a wrong decision, believe me. I used to make it and I used to fail with it. It bogs you down. You lose focus and make stupid mistakes. It leads to fear. And fear is a bad ally. You go from one extreme to another to  only increase entropy, nothing else.

If you miss the deadline, all the possible actions will not help. Cancer will win. And then you will have just two options: re-write the product completely from  scratch or start a new product.

If you see these symptoms, you should stop and think about the attitude of your development team. If you’ve survived over several years, priorities should be changed. Reset your development team and use chemotherapy.

• Focus on quality. Fix the roots of the problems.
• Teach the original architecture to all, pair program, communicate.
• Introduce “No new code without tests” rule.
• Fight fear. Let the knowledge spread. Knowledge eliminates fear.
• Put the most experienced people on fundamental problems solving.

You have to fight the cancer to bring energy back, to bring courage back, to live and produce a great software product. Otherwise it will be as dead as Lotus Notes.

Many complex systems are based on simple rules. A set of several simple rules leads to complex, intelligent behavior. While a set of complex rules often leads to a dumb and primitive behavior. There are many examples.

Ants Colony

How ants search for food? They do not have cell phones, cars and mini-markets near the nest. They should have something simpler to communicate.

Here is how ants work:

1. Travel randomly  in search for food.
2. Take a piece of food and head straight back to the nest. On the way back to the nest lay down an odor trail.
3. Notify nestmates of the discovered food encouraging them to leave the nest. These newly recruited ants will follow the odor trail directly to the food source. In their turn, each ant will reinforce the odor trail until the food is gone.

Sounds simple? Take a look at this very nice ants colony model. Drop some food and enjoy the action.

Birds Flocks

Birds flocks are beautiful. You may think that the movement gets orchestrated by one savvy bird. But this is not the case. A bird flock is guided by three simple principles (every decent bird knows them):

1. Separation: steer to avoid stumbling upon local flockmates.
2. Alignment: steer towards the average heading of local flockmates.
3. Cohesion: steer to move towards the average position of local flockmates.

Simple? Yes, it is. Look at the picture on the right. It’s just amazing!

Game of Life

Game of Life was invented in 1970 by John Conway. It is a cellular automaton and simulates the birth, death, etc., of organisms based on certain rules:

1. Each cell with one or no neighbors dies, as if of loneliness.
2. Each cell with four or more neighbors dies, as if of overpopulation.
3. Each cell with two or three neighbors survives.
4. Each empty cell with three neighbors becomes populated.

Simple rules. But these rules lead to fantastic diversity of the forms. Different types of the forms have been discovered e.g.  still objects, oscillators, gliders, spaceships, etc. It is impossible to predict the state of a system after several thousands steps. Cellular automaton has properties of a complex system such as emergency and butterfly effect. Small changes in the stable structure (for example, adding one more living cell) may cause death of the whole cells population.

Moreover, it is possible to build a computer based on Life! It is possible to implement logic based on stable structures and execute simple calculations. The Game of Life is a Turing machine! How can someone suggest that such a simple system based on four rules has so much power?

Take a break and have fun with Life game simulation.

Simple Rules and Software Development

Is there any relation between simple rules and software development? Sure, there is. Software development process should be simple. Process complexity leads to mechanistic and dumb behavior.

1. Information exchange and collaboration vs. standard status reporting meetings.
2. Learning vs. stagnation.
3. Emergent behavior and creativity vs. Following rules, standards, instructions.

Agile methodologies are simple. Scrum is a very simple thing. It has just several rules, roles and artifacts. Well, it is a lot harder to really implement Scrum. It is hard because you need to break stereotypes and habits. Many people are resistant and don’t want to try new things. However, Scrum works. It works because of its simplicity, it lives in accordance with complex systems.

Scrum stimulates learning, feedback, communication and cooperation. Emergency is possible in Scrum.

Here is one sample of unnecessary complexity — too many hierarchy levels:

A potential problem, unlikely in small and medium organisations, is deep organisational structures. According to Peters and Waterman (1982)18, both Toyota and Roman Catholic Church have only five layers of management in comparison to Ford’s seventeen.

Do you by any chance happen to have a software development process description in a huge 100+ pages document? Are you still in business?

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

In the previous parts I’ve described basic properties of complex adaptive systems and it’sbeen shown that software development is a CAS. In my next few posts I am going to describe CAS properties in depth and apply them to software development. In this part we will look into the Edge of Chaos.

Many researchers tend to think that CAS may work efficiently at the edge of chaos. A system with a very high order can’t solve complex creative tasks, as well as a pure chaotic system. It seems the system should balance between chaos and order to survive and adapt.

The “edge of chaos” term was emerged during self-reproductive cellular automata researches in 1990. It appeared that at a particular noise level the system can reproduce its state, while at a low noise levels states are random. This edge value was mathematically analyzed and it was shown that in such a state the system has a maximum of information.

The “Edge of Chaos” term  spread and applied to complex adaptive systems. For example, we may suggest that evolution drives systems to the edge of chaos, and it is one of the reasons of life origin. Life is a very interesting and complex thing in itself. Most likely it could not appear neither in order state and in chaos state. Life exists if there is a stable base, but with possibilities of changes.

Edge of Chaos and Software Development

How can we apply the edge of chaos concept to software development? Obviously, edge of chaos is a state where the development team works with maximum efficiency. Complex processes and explicit rules impede creativity. The team becomes too ordered to think :). On the other hand, no processes and no rules lead to chaos. The development team can’t complete anything useful, and that is something we call hack & fix process.

As a software development folks, we are particularly interested in two questions:

1. How can we drive the team to the edge of chaos?
2. How we will ensure that the team is at the edge of chaos?

The human who knows the answers to these questions most certainly will be rich and famous. However I am not sure whether you can find such a man.

It is obvious that edge of chaos = hyper productivity. This term often used in Scrum. However it is quite hard to define hyper productivity.

Well, currently we may discover hyper productive teams and identify their properties. But if we try to apply CAS to the edge of chaos, we may define the following properties that may lead a development team to hyper productivity:

1. Feedback
2. Information exchange
3. Cooperation
4. Self-organization
5. High Adaptability

Agile software development processes have some of these properties explicitly, and some implicitly. For example, feedback, information exchange and adaptability are roots of agile development. Self-organization is something most are expecting to have in an agile process, but sometimes it is skipped. I think it is a huge fault if, for example, Scrum Master does not include self-organization into development process. All these properties are important and must have to build a highly efficient team. If you have 4 of them, but missed the last one, you do not have hyper productive team.

Clinton Keith identified four important conditions for hyper productive teams:

Let’s try to bind these conditions to CAS properties defined above. Independence is one of the self-organization pre-requisites. Leadership on my opinion is not a must, but likely is an effect of self-organization. The development team got a leader as a result of natural selection Core competency is a result of a feedback, learning and adaptation. The last condition obviously is an information exchange and cooperation.

We may easily define conditions, that block hyper productivity:

• No feedback. That is one of the reasons for waterfall process failure. In a waterfall process feedback is delayed and sometimes even does not exist. Team can’t learn and adapt. Efficiency is low.
• Poor information exchange. Large teams, distributed teams, huge functional divisions, interpersonal problems - all that impedes information exchange.
• No cooperation. Political games inside a company or inside a development team, dumb corporate rules, wrong motivation schemes, communication problems due to personal likes and dislike - just some factors that make cooperation almost impossible.
• Impossibility of self-organization and weak adaptability. Complex corporate rules, hard hierarchy, command and control management, no autonomy make self-organization impossible.

We’ll cover these problems in more details later. Even in such a brief overview it’s getting clear what the main responsibilities of CTO, CIO, PM, Scrum Master, etc. should be. They should remove obstacles listed above (and more) that block a trip of the development team to the edge of chaos.

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

Complexity science is very young. Cybernetics, ecology, sociology, meteorology all study complexity. In general, a complex system consists of interacting components. The result of this interaction can’t be predicted by observing an individual component. For example, human brain consists of neurons. The brain has ‘memory’, while each neuron doesn’t. Ants’ colony behavior can’t be predicted from the behavior of an individual ant. And it is impossible to understand software development process looking at how just one developer works.

We are especially interested in a particular case of complex systems called Complex Adaptive Systems (CAS). The main difference is that CAS may learn and change (adapt) over time based on previous experience. CAS remember the history, and that is the main difference from chaotic systems.

There is no common definition of Complex Adaptive System (CAS). One of the most popular definition was offered by John H. Holland

CAS has quite many common properties. However, hardly a system should have all of them to be CAS. Moreover, researches define different sets of CAS properties. Let’s try to review the most common properties and see how software development process can be described in terms of those properties.

This is just a quick overview of common CAS properties. Later we will go into details and draw more parallels between CAS and software development. But even in this short list we’ve got some nice phrases like self-organization, feedback, information exchange, cooperation, adaptation - does it remind you something?

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

There are quite many agile processes in the world already. We have Extreme Programming, Scrum, Crystal, DSDM, Feature Driven Development, Lean. Some of them are famous, some are maybe dying. The point is, we have plenty of agile processes. In general this is not too bad. We’ve got alternatives, they move agile forward and solve real problems. On the other hand, I feel the lack of information on the roots of agile processes. Most articles available are missing the point on my opinion.

Why iterative development works? Why agile manifesto is the truth? Why traditional project management techniques miserably fail on software development projects? Somehow many people feel the right direction. They have a background that guides their decisions. We feel that agile manifesto brings real value. We has proven this in practice after all! But I personally feel that there is something deep and cool behind most of agile practices. Something that will explain why this all works.

While reading Jeff’s post about Scrum roots, I found a reference to Complex Adaptive Systems. I went digging into more details. I read a lot about CAS and related things and there were many “a-ha” moments. There are so many parallels between CAS and Software Development. It is pretty obvious that Software development is a Complex Adaptive System itself. It has almost all basic properties of CAS and it is interesting to read, for example, about adaptation in various CAS and think about its application to the software development and agile principles.

I believe that a deep understanding of Complex Adaptive Systems will shed some light to the real roots of agile development and will answer important questions. With this knowledge in hand it will be easier to see why this works and why that failed. It will be easier to make decisions and create a better software faster.

However, I am not sure how interesting this will be to the people In the next several posts I will write on why software development is CAS, describe important properties of CAS and apply them to software development (adaptation, feedback, self-organization, emergence, edge of chaos, non linearity, etc). Let’s see how it goes, maybe we’ll find something interesting

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development