2009
Code base of a large project is getting worse over time. I hope there are lucky exceptions, but in general it is true for most projects. The reasons are quite obvious:
- More and more features. It leads to increased complexity.
- Shortcuts and hacks to support “We need this fancy search till August. Period!” features
- Developers rotation. New developers don’t know all the fundamental decisions and ideas behind the architecture. Knowledge gets lost with transition inevitably.
- Development team growth. More people – less communication. Less communication – bad decisions. It leads to code duplication, hacks to make something work without deep understanding of the underlying conditions etc.
Suddenly you can’t add new features easily, you can’t make significant changes easily, you have tons of technical debts and development team is close to bankruptcy. You want to change that and have just two options: refactoring or rewriting everything from scratch.
Refactoring and Punctuated Equilibrium
Punctuated Equilibrium is a theory in biology, but it is applied to other complex adaptive systems like societies as well. It states that the system has almost no changes in significant period of time and then changes very rapidly. Then it acquires the equilibrium state again. Sounds familiar? Sure, refactoring is almost the same.
Any change in the system introduces chaos in the short term. The goal of refactoring is to eliminate chaos, but often you increase it initially. When you are working on changes you make system less stable, but as the change is completed, the system can be more ordered. That is not always true for software development. If you use branches, this makes changes safer. Still significant changes increase the risk of new bugs.
The real danger of refactoring are local optimums. With a full rewrite you may have a better architecture than with refactoring. Solution? If you have a vision of final architecture, use it and try to make a path from current architecture to the new architecture. In general simplicity should emerge from refactoring.
Rewrite and Chaos
When you rewrite from scratch, you add such a large portion of chaos that it is hard to predict the final result. You have a new singularity that will explode to the new product universe. But are you certain that it will be better than the previous universe? How many the ‘same bugs’ will you fix in the new product version?
However, rewrite may look faster. I mean you may release a new version faster with rewrite, but most likely with more bugs and less stable.
I think we may expect something like that:
The green line shows how chaos changes with refactoring. After each refactoring there is a small increase of chaos, but then the system becomes stable and chaos decreases. You see that the final release is quite late, but keep in mind that there have been many releases before, so customers benefit earlier.
Black line is how chaos changes with a full rewrite. We have the old system during rewrite, so chaos is constant. After the public release chaos increases significantly. Quite many new (and old) bugs and quirks are expected, so stabilization period is longer. But the release itself is faster. The reason is that there is no burden behind. For example, there is no need to support all the places when you do a change, while in refactoring it is required to keep system working and stable all the time, and it demands additional effort.
Adaptation vs. Revolution
There is another angle to this problem. Refactoring is adaptation, while full rewrite is revolution. Again, revolution is a chaotic beast. You may slowly adapt the product for new external conditions or make one revolutionary rewrite.
So, Rewrite or Refactor?
I do think there is no universal correct answer to this question. If time to market is paramount, if you feel that you’d lose business if a new version will not be published in 6 months, you may try a full rewrite. But beware side effects! Quality may drop significantly and long stabilization period may hurt existing customers.
In most cases refactoring is preferable. Slow pace, high quality, constant improvements, happy customers. I like it more. But rewrite is so magnetic…
We’re all tied together by things we do. Projects we work on, conferences we go to as a whole team, even bugs we fix together, problems we solve together. Insights we share together.
Some people call it team spirit. Some people call it good collaboration. In any case, this is the light of live communication and talk between people. My deepest belief after all is that no matter how sophisticated telecom stuff gets our days, there’s nothing more valuable – and ROI-generating as well – as the real talk.
I was perplexed the other day as I saw the phrase - “we’re trying to maximize our face time with them” – this was said about some guys from an off-site location visiting the main office. My first thought was – so the rest of the time they work is their a** time? As obviously software guys spend most of their time sitting on their derrier?? Though of course it’s worth a praise that after all they’re still trying to maximize the face time..
As hype is the statement that with globalization you could get the best software developers out of anywhere in the world, as eternal is the truth that to produce people need to REALLY collaborate. This goes for software, for any production. So it’s not about picking up people as vegetables on the market – even if it’s a huge market. People are more than vegetables. They need live communication to feel alive, to collaborate and to be productive (for those who want to count figures, you can sit down now and compare the monetary value of pure programming skills - with no communication skills at all).
I’m not saying that we should all go local in our quest for best products and profits. But what I really see is that people are squeezing their way to follow some of agile communication practices using telecom. As a proof, look at hot discussions at LinkedIn agile groups, and Jean Tabaka’s observations on the reasons of agile adoption failure.
Blasphemy to the religion of remote/distributed teams – but the picture is slowly starting to get clear for me: it really takes good skills to balance the equilibrium of infrastructure and management to activate collaborative agile work in remote teams. Meaning 5 here, 7 there, 6 elsewhere – as one team.
Is it possible to trust without seeing each other? I doubt there’re teams who are absolutely OK with remote customers/product owners etc. I even suspect that waterfall can be the best solution for remote teams-customers that haven’t developed enough trust to each other. If anyone has real-life stories to prove that I’m wrong, speak up.
Everything has its life-cycle. Even stars. Even The Universe. Everything.
It is quite sad for intelligent creatures, but it is just a fact of life. I want to draw some quite obvious parallels to reveal the real danger of Technical Debt for a software product.
Humans
Let’s take a human. When we are young, we are overflowed with energy. Do you know how a 7 year-old boy spends his day? I have a son, so I’ll tell you. He wakes up at 7 am (sometimes before 6 am!) and tries to play some quite silent games (it is too early to make much noise). He can restrain himself till about 8 am, then he starts to jump, does somersets, some boxing etc… By 8.30 we have a real “tornado” in our house. It may take him an hour to exhaust and have a breakfast. Such periods of activity can be easily repeated 2-3 times a day. And he literally turns off around 9 pm – sleep is when he recharges his batteries.
An old man is quite the opposite. He saves every movement, everything is hard and slow. He makes many mistakes and health maybe getting worth. Learning new things is also getting harder at times. Yes, wisdom and tremendous life experience is there, but very often there is no energy to move forward.
Software products
Software products have exactly the same behavior. Young product is energetic. It has an excited development team behind. It moves forward quickly and feels warm sun on the top. It learns lightning fast and grows on new knowledge. Everything is smooth and shiny. Time to market is everything and tradeoffs are so common! Let’s fix it fast and mark with //TODO: refactor this comment. Let’s hack this framework here to release another cool new feature this week. Let’s re-invent the wheel and write some custom javascript code. Gosh! We don’t have time to learn this new cool javascript framework! These are quite common patterns for young and energetic products.
But what’s next? Years later progress stops. Product becomes old with all the bad and good side-effects. Suddenly it is much harder to add new features. Ages of development make the product more complex. Millions of technical debts rotten its body like cancer . There is no clear architecture anymore and there are so many patches that one small change can produce a totally unpredictable impact and bring along new bugs in unexpected areas. Yes, development team has wisdom. But often no courage and energy to revive the product.
Technical Debt Is a Cancer
How long does this cycle last? It really depends. We all visit doctors. Most of us want to live longer and healthier. Early cancer detection gives good chances to fight and win the disease. We should do exactly the same with software. Here are typical symptoms:
- Velocity has dropped significantly over the last several iterations/releases
- A bug fix triggers one or several new bugs too often
- Nobody knows ideas behind the original software architecture
- Team spends more time fixing bugs than developing/improving features
- Automatic tests are red 80% of the time (if it is 100% — the product is most likely in coma)
If you see at least 2 such symptoms, you’ve just discovered a product cancer — Technical Debt. Technical debt is a true killer when you have deadline (time to market). If you have the symptoms, you should fight the disease right now. You may think that it is OK to wait several months, add some more “cool and highly requested” features and then get back to the real problems. It is a wrong decision, believe me. I used to make it and I used to fail with it. It bogs you down. You lose focus and make stupid mistakes. It leads to fear. And fear is a bad ally. You go from one extreme to another to only increase entropy, nothing else.
If you miss the deadline, all the possible actions will not help. Cancer will win. And then you will have just two options: re-write the product completely from scratch or start a new product.
If you see these symptoms, you should stop and think about the attitude of your development team. If you’ve survived over several years, priorities should be changed. Reset your development team and use chemotherapy.
- Focus on quality. Fix the roots of the problems.
- Teach the original architecture to all, pair program, communicate.
- Introduce “No new code without tests” rule.
- Fight fear. Let the knowledge spread. Knowledge eliminates fear.
- Put the most experienced people on fundamental problems solving.
You have to fight the cancer to bring energy back, to bring courage back, to live and produce a great software product. Otherwise it will be as dead as Lotus Notes.
Once upon a time there lived a very brave and adventurous young man by the name Arthur. He lived in a large kingdom with knights and castles and an extremely cunning king who had a beautiful daughter – Caroline. She was so beautiful, that every morning birds would fly through the open window with fresh roses in their beaks. Birds would drop rose petals near her bed and fill the air with pleasant tweets, every five minutes exactly. Then they would fly away to never come back…they would simply die outside, because they couldn’t live without princess’ beauty anymore.
The king was greedy so he held annual competitions during which the contestants had to pay 10 sovereigns each to participate. The winner would marry Caroline, but missions were so hard, that there were no winners ever!
New competition was announced and brave Arthur decided to win the highest prize! He saw the beautiful princess only once before and fell desperately in love with her.
On a sunny day all the contestants gathered together on a central square. The king said: “Brave men! There will be 3 main missions… and a very easy 4th mission to the winner. Today I’ll tell you about the first mission only. You have to build a house in 10 days. It should have 2 rooms, doors, windows and a roof. That’s all you have to do!!”
Mission #1
Is it possible to build a house in 10 days? Arthur was disappointed. He had hoped to kill a dragon, catch a witch or do something else that a brave knight is supposed to do. But he was smart (and brave). He spent all the day doing nothing but thinking. Other contestants laughed at him: “Look at Arthur! He gave up miserably! Arthur, go home, you are such a loser!” But as you know, our Arthur was smart so he decided to build the simplest house possible. And he did it: the ground became a floor; house had only 2 tiny rooms and the light would enter through the 2 windows near the front doors; the roof was there to simply protect from rain and snow – no more no less! Most importantly, he finished it in 8 days! Many, many other brave men tried but failed. Some started to build a large and solid house and ran out of time. Others started building too quickly and their houses would collapse half way through. However, brave Arthur and a dozen other guys completed the first mission successfully.
“I am proud of you! – said the cunning king. “Now here is the second mission for you all. This time, you should build a basement that will be twice as large as the house itself. Also you should build a fireplace. You all have a total of 12 days to get the job done.”
Mission #2
And again Arthur spent the whole day thinking. “I don’t even have a floor in the house… How can I build a basement? The house will simply break down… If I spend time on the floor, I’ll have no time to do the rest.” Then he found a workaround. He decided to dig deeper than usual. “Deep basement should be safe enough and floor is not required in this case! Fireplace will take 2 days to create, and I can definitely make this basement in 9 days!”
Arthur worked like crazy and completed the mission in 11 days. He slept for the whole 12th day – the last day of the mission. At the end, only 5 other brave men have completed this stage of competition.
“You did a fantastic job!” – the king said. – “Now listen to the last mission. You should build an upper level in your house. An upper level may have only one room, but you have only 5 days to do it.”
Mission #3
All the contestants were quite excited. It looked like an easy task. But Arthur was sad. “If I build another level” – he thought – “my house will crash…” This time he started to work immediately. He brought several large logs into the basement and made props. It took him a total of 2 days. Arthur was not sure that it’d be enough, but he had no more time and decided to accept the risk. Luckily, he had a flat roof and therefore it turned out to be relatively easy to create another level right on top of it. Arthur was tired, but extremely happy. After all, he was the only one who made it through the last mission!
“My boy!” – King happily hugged Arthur. – “You did it! I can’t believe, but you did it! You are just one step away from the Caroline’s hand. Your last mission can’t be simpler – You should live in the house you built during the first 2 weeks. This is it. And then you will have my blessings and the hand of my beloved daughter, Caroline!”
“Wow, so easy!” – said Arthur. “Yes, it is” – answered the king with a smile on his face.
Mission #4
The next day brave Arthur was killed by the crashed ceiling. It was a windy and rainy day…