Edge of Chaos | Agile Development Blog

Software penetrates every pore of human existence. We look up the weather info over the web, giving up on outdoor thermometers. We’re driving to destinations with GPS navigator (forget paper maps with their G7 sections on page 59). We turn on RunKeeper when riding a bike to calculate the average speed and run and boast in Twitter. We’re using software every single day of our lives. It seems we’re hugging our dear gadgets a lot more than our loved ones.

No one knows the exact how-to of writing great software fast, that’s the problem. Waterfall passed away at the crossing of 2 centuries, whereas new software development methodologies (agile) fail at solving the fundamental problems so far. We’re living in very interesting times. Software development industry grows fast right here, right now, and the foundation for a quantitative leap is building up.

Today I’ve read two interesting posts: The Cost of Code by @unclebobmartin and Code as a Cause of Project Failure by @DocOnDev. They discuss various arguments to prove that all projects fail because of code. The main argument is that if code is free and light-fast to change, project can’t fail. OK. But this case is rather extreme and obviously impossible. We don’t live in a world with hyper-space jumps, teleportation and free medicine (unfortunately). In real life code costs a lot and it will be true in the next decades for sure, so this argumentation proves nothing. In ideal world there is no such thing as code. In ideal world you have a solution in seconds without computers, software and other complicated things. So I don’t buy this idea. Code is not the main reason for projects failure in reality.

Code is not free. Code is expensive. We do not sell source code though, we sell solutions. If it is possible to create a solution without source code, it would be fantastic. Let’s take an industry that handles tangible objects. Automobile industry does not sell carbon and metall — it sells cars. It sells a solution to transportation problem. Teleportation is an ideal solution, but we can’t teleport nothing but electrons, sadly. We buy cars to get from point A to point B. We buy a solution.

Code != Solution

I think the main problem with projects is that they provide either bad solution or no solution at all. Nobody buys stagecoaches these days, since there are more efficient solutions. If a project does not solve anything, it will fail. If a project solves something, but  in a nasty, unusable way, it will fail. You can create an absolutely beautiful architecture with the cleanest code in the world. You may have 100% test coverage, complete separation of concerns, flat hierarchies and methods without boolean arguments. You may have all that beauty, but still fail miserably if the program does not solve user’s problems efficiently.

You may argue that with clean code you can refactor it fast and change everything. C’mon, it will be a full re-write if it solves the wrong problem. Can you fix a stagecoach to make it a true competitor of a car?

On the other hand, if a projects solves a right problem, but with some issues, clean code is very important. You can’t adopt fast without it. You can’t change things and react to people demands.

Don’t get me wrong, I believe that clean code is an important thing, but it is not the most important asset in software development.

They did it again. Companies are making the same mistake during agile adoption over and over again. We’ve made some of these mistakes as well. If by any chance you see something familiar in the list below, read on and find out why it is a mistake.

1. Start With a Tool

It is not always bad to start with a tool. If you want to dig a pit, you most likely want to find a shovel first. You can dig manually, without a tool, but it will take enormous amount of time.

Agile development is something different. A tool will not provide immediate effect and will not solve most of the problems by the mere fact of its existence. Moreover, you will put effort into tool adoption, shading more important goal — agile adoption.

We encounter this mistake quite often. People come to TargetProcess web site, register for a trial, install the tool and try to use it. They start to ask questions and it’s getting clear that they have no experience in agile development neither agile process established. Sometimes they don’t know what a Burn Down Chart is and how to use it. Sometimes they know nothing about iterative development. It happens. The only piece of advice we can give is to get rid of the tool and dig deeper into agile domain: become familiar with basic concepts and try some process with simplest tools like whiteboard and sticky notes. Then decide whether you need a more sophisticated tool.

2. Start With a Process

Starting with a tool is obviously a bad idea. Most companies start with a process. It is a less serious, but a more common mistake. So you read about Scrum, it looks easy initially. You apply all Scrum practices and after some sprints see that development somewhat improved, but not as drastically as expected. Excitement goes away and process degrades.

What are the reasons? Why has it failed? Most likely people didn’t get the core values of agile development. Process is the mechanics, while values are the core of any agile adoption. The first thing any company should do before trying any process is to focus on agile values such as: communication, collaboration, feedback, trust and passion. It is nearly impossible to apply a good development process if you compromise any of these values. As an agile champion you should enforce these changes. Sometimes these are company-wide changes, sometimes team-wide changes, but still it is absolutely required to apply them.

The fastest way to adopt agile is to start with people and culture. I can’t stress this enough. And it is quite obvious! But why so many companies (including ours) have made this mistake? It is blazingly hard. You will have to change culture, and that will be met with a resistance. You will have to change people and most likely you’ll have to fire some of them. It is a hard and complex process, there are not so many people in the world who know exactly how to apply it. Most of us have no such experience and skills, so we fear this change. We try to start with something easier, something we know. We try to start with a process, with a tool, or with hiring an external consultant (which is good, but not enough). We should fearlessly start with the hardest but the right thing.

Agile manifesto is very deep indeed. If you read it carefully and set the principles as a cornerstone of agile adoption, you will success eventually.

3. Development Practices Are Enough

Extreme Programming is a very cool agile software development process. I think it includes Scrum. Many people may disagree, but in general if you do XP fully, you don’t need Scrum. Interestingly, that developers tend to apply technical practices like TDD, Continuous Integration and even Pair programming, but pay less attention to such things as communication, integral team, having customer on-site and retrospectives.

Why is it like that? Developers are techies, and techies like technical stuff, while often disrespect social stuff. Many of them are introverts, they don’t like meetings, they don’t like to chat with people extensively. They do like to communicate about technical things and do that with passion. However, communication with customer is not about Clojure or fancy lambda expressions, it is about business. It’s business problem domain, and it’s not that interesting for them.

Again, agile is more about people and less about technology. Many practices are focused on people. Pair programming makes for more communications between developers. Customer on-site makes for more communications with end user. Retrospectives boost communication about team, processes, etc. It is absolutely required to apply practices that focus on communication, adoption and fast feedback. It is absolutely required to pay more attention to people, improve their skills, improve technical excellence and shift mindset.

4. Scrum is Enough

Is it possible to adopt agile without technical practices at all? No, it is not. However, it is a less severe mistake than the previous one. If you apply Scrum right, you will inevitably decide to try some technical practices at a retrospective meeting.

The true mistake will be to rely on Scrum solely as a process that will solve all the problems. It can do that, but only if you are open-minded and willing to try various things: from pair programming to BDD. Best coaches believe that Scrum should be adopted in tight pack with XP technical practices. This is a good advice to follow.

5. CSM Knows Everything

Certified Scrum Master is not a demigod. Yes, he has some basic knowledge about Scrum, but in many cases that’s just it. He may have no hands-on experience neither strong theoretical background. He should not act as a PM and prescribe what to do and what not to do. Team should decide. The only real things SM should care about are teams impediments and meta-process. By meta-process I mean a set of rules/procedures that allow team to reflect and improve existing development process.

Common manifestation of this mistake is phrases like “I insist we should change iteration duration from 1 month to 2 weeks” or “I strongly believe we should keep doing code reviews”. It might be that he is right, but it does not matter, the language speaks for itself. There should be no “I” in SM phrases, there should be “We”. Otherwise agile adoption will fail. If SM doesn’t believe in team, the team as such will not gel and will degrade eventually.

Read Part II for last 5 mistakes.

I read Agile development grows UP article written by Mark Lines. It has quite a strange taste… Is it an attempt to advertise new methodology? I hope it is, otherwise Mark is completely missing the point and agile development trends.

I want to review the article. Just can’t resist, sorry.

Many of us don’t have the benefit of co-location, where the entire team works in close proximity (ideally in the same room), with the user representative present at all times to answer questions

Sure we all know that a co-located team is better. It is not a surprise that consultants advise to co-locate team members. If you need to win the race, it is better to drive BMW M3 than Ford Focus. There are so many discussions around distributed agile this year. There are so many ideas and experience reports on how to implement agile remotely. Agile works in remote teams.

Pair programming i.e. 2 programmers sharing 1 keyboard is an expenditure most management will not endorse

It’s a very strange argument. Many managers will not support continuous integration, iterative development and BDD. It does not mean that these practices are bad. Context is everything. In some companies and on some projects pair programming will increase productivity. While on other projects there will be no benefits or even degradation. How many agile coaches insist on pair programming and say “You are not agile without PP”? Not many. It is wrong to spread this argument to the whole agile community.

Delivering a system without moderately detailed requirements (beyond User Stories) is not acceptable for testing, writing training materials, and production support purposes.

C’mon! User stories can be VERY detailed. For example, you may use BDD-style user stories format and write many cases that will describe functionality in some format, that can be directly converted to unit tests! It is an incredible thing, to be honest.

Fixing architecture as you go (i.e. refactoring) without some initial architectural design is not appropriate for large-scale enterprise systems

Most people in agile community believe that it is required to spend some time on architecture. Again, it is so context dependent. If you create a simple application, one day may be enough to nail down all important decisions. If you create a complex system, it may take several months. Common sense is a critical factor here. There is always a danger to over-architect things and working software is the best proof of concept.

Also something called “emergent architecture” is being developed. I like the idea and maybe it is a right direction.

Most projects are not completely independent and cannot be built in isolation. Dependencies and integrations with other systems are required and require some co-ordination.

Agile adoption is spreading rapidly and indeed there are no common/standard approaches to solve this problem. But I believe it will be resolved during agile enterprise adoption (and we already stepping into this phase).

Organizations usually have some governance practices in place (such as PMOs, ISO, CMMI) that necessitate a level of bureaucracy and documentation

Does it all mean that PMO, ISO and CMMI are good and we should give up and follow the rules? Agile community tries to fight bureaucracy and I personally love that. Documented process means NOTHING for success of software development project. In 90% of cases it holds you in the spider’s net and blocks creativity. There is NO software development without creativity. Wake up, Mark.

We live with these standards, but we should fight them.

Deploying software to users every 1 or 2 weeks is usually not practical in most organizations. Just migrating software through development, test, system test, user acceptance, production environments is a major undertaking. Having many projects trying to do this in parallel on a weekly basis simply isn’t feasible.

This argument is so common… and “not practical in most organizations” is a truly masterpiece. Any statistic data? Did Mark try to deploy anything outside the enterprise-scale Java mastodon applications world? Many SaaS services do painless weekly updates. Some services do painless updates on a Daily basis! It is unbelievable, but I like that. Customers like that!

Mark promotes OpenUP methodology (based on Unified Process). Only advantages in the post, no disadvantages or context at all. Folks, I think it is a so long-awaited silver bullet! Finally it is there! Let’s bury Scrum. Let’s bury XP. Let’s burn Kanban and all the other processes that spark agile movement. God bless the OpenUP silver bullet. Amen.

People often ask questions like “What is the difference between Scrum/XP/Lean/Agile/You name it“? I don’t see much difference between XP and Scrum on that matter. If you already have XP, most likely don’t need to switch. Maybe adopt some Scrum specific practices for a better scalability like Scrum-of-Scrums. Almost all the other practices like daily meetings, iterations, roles separation, retrospectives, etc. exist in XP already.

In fact I am not sure if such separation brings benefits. It is better to spot what’s going wrong in your development process during retrospective meetings and apply practices from any agile process (or create own solutions) to your specific problems. XP, Scrum or Lean give you a framework that helps to be adaptive and creative, whereas traditional processes give you a set of rules hindering any creative behavior.

Your team and your project IS special. Think and communicate to sharpen your development process.

I do understand why people are so enthusiastic about Scrum or XP. These processes provide tools and practices out of the box. You may just start doing it and see what happens. It is not the best way, since quite often agile adoption fails. People don’t get the new way, they implement it without proper support and under the pressure. They maybe don’t have a deep understanding how it should work. So it is expected that agile adoption might not be smooth for all.

Agile Development Future

I wish all agile methodologies evolve into one elegant, smart, simple framework. For me it is a natural trend and it should be something like Theory of Everything in physics. This framework should explain why agile processes work. It should based on natural laws of Complex Adaptive Systems. It should provide general guidance on agile adoption and a repository of possible practices that might be used to build your own development process. The framework should be simple and unobtrusive, since complex rules will not work for software development. The framework should empower creativity, communication, learning, feedback and self-organization. The framework should support nonlinearity, emergence and edge of chaos concepts.

Software development is young. Currently we are having transition from waterfall to agile. It is mainly based on practice rather than theory. Practice has proven that agile works better. However, there’s almost no research explaining why. There are some blog posts and short explanations with reference to Complexity Theory. That is a right direction I think, but there’s no depth behind these posts, they are quite superficial.

I expect we will have deeper understanding of the agile roots and why it works. This knowledge will simplify transition and help us to find better practices, patterns and solutions. I expect there will be a unified agile software development framework in the future.

So far Scrum and Lean are closest processes to this framework. Lean has a strong philosophy and nice background. It is general enough. Scrum is simpler and more understandable for mere mortals. XP has a set of best practices and heavier in general. All of them have strong and weak sides. But still I feel there’s a deeper level and all modern processes are just a subset of some hypothetical (at the moment) unified agile software development framework.

I’ve manage one small project (3 developers – 6 months planned). Project was canceled due to some reasons (project progress was great, so the reasons are not in development :-). I briefly describe here what we’ve tried and what was useful for us. I think this information will be interesting. This is a real example after all. The project was on Perl.

XP-style Planning. Small 2 weeks iterations. Small iteration helps to track project progress and create better estimates. Planning is quite easy and can be done in Excel without sophisticated tools like MS Project. We are going to use XP-style planning in future for high-risk projects (with new technologies, new team, or whatever make project risk high). Sure, web based tool for XP-planning can help.

Customer Feedback. Manage a project without client representative is a hell on the earth. You are working like in a vacuum. No response, no feedback and no attention. You can’t establish rapport with a client, and there is a chance that system will not satisfy client expectations. And sure it won’t. Project Manager must make every effort to get a person from the client side for the constant feedback.

Info for Top Managers. All information for top managers should be in a short and clear form. Do not send 20 pages requirements docs or lengthy emails. Top managers do not have enough time to read such docs. It is better to extract most important features and ideas at a single page and attach full document to be on a safe side.

Phone Calls. Phone calls are much better than emails or IMs. Sure, phone is impolite and it is hard to clear express your thoughts, but, anyway, phone is better. Many issues can be resolved faster and easier.

Test-Driven Development. We wrote unit tests for all business objects before actual code, and it worked great. However, we did not have tests for controllers. Controllers are very close to GUI and they changed often (during first month). Maybe we should write unit tests for controllers as well, maybe not. This is unclear. Anyway, TDD is good and we will use it as a mandatory practice.

Acceptance Tests. About 50 tests were created based on SAMIE. We modified tests quite often and spent near half an hour hour per day to keep ‘em working. These tests helped a lot when we performed major refactoring in controllers and GUI layers. But it seems that SAMIE is not the best tool for acceptance testing. It is easy to use, but sometimes unstable and Win32 restricted. But I did not find a better one so far.

Refactoring. It helps to reduce code duplication and keeps system architecture clear. We had doing refactoring constantly, and sometimes we spent whole day on major refactoring. As a result, new functionality implementation appeared easier than we expected. We will use refactoring as a mandatory practice in future.

Walking Skeleton. This is a first, fast and small release, which contains single system feature ‘in depth’. For example, show projects list, add project into the database, check user permissions on project. Walking Skeleton for the system took near 2 weeks. We managed to investigate many architectural decisions and choose the best. Small and fast release is a great thing.

Pair Programming. We created the most important system modules in pair. The code quality and overall system architecture appeared significantly better when pair programming used. Constant code review, communication and experience sharing are really great things. It is not easy to program 7 or more hours in a row, and you feel exhausted in the end of the day. But work satisfaction is great. We will use pair programming for teaching, experience sharing and critical modules implementation in future.

Code Metrics. LOCs dynamic, Unit Tests # dynamic, Acceptance Tests dynamic. Code metrics helped to track project progress and predict team velocity and project size.

Project Site. Helps to organize important project related information (news, documents, actual project state, links) in a single place. However, Project site update should be more simpler. For example, we can use a basic CMS for project site maintenance in future.

O/R Mapping. Tangram was used as an O/R mapping tool. It took about 2-3 days to resolve some tricky issues, but in general O/R mapping speeds up development and supports database abstraction. Maybe OODBMS is a better solution, but if you stick to RDBMS, O/R mapping will help. However, for small systems or web sites O/R mapping is not required.

System Prototypes. Helps to resolve many issues on the early stage. In general, the prototype should be created if project budget allows to do that. But I do not recommend to create a comprehensive prototype, since it just wastes time. Prototype should contain major system modules and maybe some secondary.

Honesty. Just be honest with your teammates. They will trust you, and trust is one of the major factors for gelled teams.

Documentation in Source Code. Almost all system architecture related documentation was in source code. We used a tool to build linked html pages from POD documentation. As a result, documentation maintenance was easier.