Edge of Chaos | Agile Development Blog

I read Agile development grows UP article written by Mark Lines. It has quite a strange taste… Is it an attempt to advertise new methodology? I hope it is, otherwise Mark is completely missing the point and agile development trends.

I want to review the article. Just can’t resist, sorry.

Many of us don’t have the benefit of co-location, where the entire team works in close proximity (ideally in the same room), with the user representative present at all times to answer questions

Sure we all know that a co-located team is better. It is not a surprise that consultants advise to co-locate team members. If you need to win the race, it is better to drive BMW M3 than Ford Focus. There are so many discussions around distributed agile this year. There are so many ideas and experience reports on how to implement agile remotely. Agile works in remote teams.

Pair programming i.e. 2 programmers sharing 1 keyboard is an expenditure most management will not endorse

It’s a very strange argument. Many managers will not support continuous integration, iterative development and BDD. It does not mean that these practices are bad. Context is everything. In some companies and on some projects pair programming will increase productivity. While on other projects there will be no benefits or even degradation. How many agile coaches insist on pair programming and say “You are not agile without PP”? Not many. It is wrong to spread this argument to the whole agile community.

Delivering a system without moderately detailed requirements (beyond User Stories) is not acceptable for testing, writing training materials, and production support purposes.

C’mon! User stories can be VERY detailed. For example, you may use BDD-style user stories format and write many cases that will describe functionality in some format, that can be directly converted to unit tests! It is an incredible thing, to be honest.

Fixing architecture as you go (i.e. refactoring) without some initial architectural design is not appropriate for large-scale enterprise systems

Most people in agile community believe that it is required to spend some time on architecture. Again, it is so context dependent. If you create a simple application, one day may be enough to nail down all important decisions. If you create a complex system, it may take several months. Common sense is a critical factor here. There is always a danger to over-architect things and working software is the best proof of concept.

Also something called “emergent architecture” is being developed. I like the idea and maybe it is a right direction.

Most projects are not completely independent and cannot be built in isolation. Dependencies and integrations with other systems are required and require some co-ordination.

Agile adoption is spreading rapidly and indeed there are no common/standard approaches to solve this problem. But I believe it will be resolved during agile enterprise adoption (and we already stepping into this phase).

Organizations usually have some governance practices in place (such as PMOs, ISO, CMMI) that necessitate a level of bureaucracy and documentation

Does it all mean that PMO, ISO and CMMI are good and we should give up and follow the rules? Agile community tries to fight bureaucracy and I personally love that. Documented process means NOTHING for success of software development project. In 90% of cases it holds you in the spider’s net and blocks creativity. There is NO software development without creativity. Wake up, Mark.

We live with these standards, but we should fight them.

Deploying software to users every 1 or 2 weeks is usually not practical in most organizations. Just migrating software through development, test, system test, user acceptance, production environments is a major undertaking. Having many projects trying to do this in parallel on a weekly basis simply isn’t feasible.

This argument is so common… and “not practical in most organizations” is a truly masterpiece. Any statistic data? Did Mark try to deploy anything outside the enterprise-scale Java mastodon applications world? Many SaaS services do painless weekly updates. Some services do painless updates on a Daily basis! It is unbelievable, but I like that. Customers like that!

Mark promotes OpenUP methodology (based on Unified Process). Only advantages in the post, no disadvantages or context at all. Folks, I think it is a so long-awaited silver bullet! Finally it is there! Let’s bury Scrum. Let’s bury XP. Let’s burn Kanban and all the other processes that spark agile movement. God bless the OpenUP silver bullet. Amen.

People often ask questions like “What is the difference between Scrum/XP/Lean/Agile/You name it“? I don’t see much difference between XP and Scrum on that matter. If you already have XP, most likely don’t need to switch. Maybe adopt some Scrum specific practices for a better scalability like Scrum-of-Scrums. Almost all the other practices like daily meetings, iterations, roles separation, retrospectives, etc. exist in XP already.

In fact I am not sure if such separation brings benefits. It is better to spot what’s going wrong in your development process during retrospective meetings and apply practices from any agile process (or create own solutions) to your specific problems. XP, Scrum or Lean give you a framework that helps to be adaptive and creative, whereas traditional processes give you a set of rules hindering any creative behavior.

Your team and your project IS special. Think and communicate to sharpen your development process.

I do understand why people are so enthusiastic about Scrum or XP. These processes provide tools and practices out of the box. You may just start doing it and see what happens. It is not the best way, since quite often agile adoption fails. People don’t get the new way, they implement it without proper support and under the pressure. They maybe don’t have a deep understanding how it should work. So it is expected that agile adoption might not be smooth for all.

Agile Development Future

I wish all agile methodologies evolve into one elegant, smart, simple framework. For me it is a natural trend and it should be something like Theory of Everything in physics. This framework should explain why agile processes work. It should based on natural laws of Complex Adaptive Systems. It should provide general guidance on agile adoption and a repository of possible practices that might be used to build your own development process. The framework should be simple and unobtrusive, since complex rules will not work for software development. The framework should empower creativity, communication, learning, feedback and self-organization. The framework should support nonlinearity, emergence and edge of chaos concepts.

Software development is young. Currently we are having transition from waterfall to agile. It is mainly based on practice rather than theory. Practice has proven that agile works better. However, there’s almost no research explaining why. There are some blog posts and short explanations with reference to Complexity Theory. That is a right direction I think, but there’s no depth behind these posts, they are quite superficial.

I expect we will have deeper understanding of the agile roots and why it works. This knowledge will simplify transition and help us to find better practices, patterns and solutions. I expect there will be a unified agile software development framework in the future.

So far Scrum and Lean are closest processes to this framework. Lean has a strong philosophy and nice background. It is general enough. Scrum is simpler and more understandable for mere mortals. XP has a set of best practices and heavier in general. All of them have strong and weak sides. But still I feel there’s a deeper level and all modern processes are just a subset of some hypothetical (at the moment) unified agile software development framework.

I’ve manage one small project (3 developers - 6 months planned). Project was canceled due to some reasons (project progress was great, so the reasons are not in development :-). I briefly describe here what we’ve tried and what was useful for us. I think this information will be interesting. This is a real example after all. The project was on Perl.

XP-style Planning. Small 2 weeks iterations. Small iteration helps to track project progress and create better estimates. Planning is quite easy and can be done in Excel without sophisticated tools like MS Project. We are going to use XP-style planning in future for high-risk projects (with new technologies, new team, or whatever make project risk high). Sure, web based tool for XP-planning can help.

Customer Feedback. Manage a project without client representative is a hell on the earth. You are working like in a vacuum. No response, no feedback and no attention. You can’t establish rapport with a client, and there is a chance that system will not satisfy client expectations. And sure it won’t. Project Manager must make every effort to get a person from the client side for the constant feedback.

Info for Top Managers. All information for top managers should be in a short and clear form. Do not send 20 pages requirements docs or lengthy emails. Top managers do not have enough time to read such docs. It is better to extract most important features and ideas at a single page and attach full document to be on a safe side.

Phone Calls. Phone calls are much better than emails or IMs. Sure, phone is impolite and it is hard to clear express your thoughts, but, anyway, phone is better. Many issues can be resolved faster and easier.

Test-Driven Development. We wrote unit tests for all business objects before actual code, and it worked great. However, we did not have tests for controllers. Controllers are very close to GUI and they changed often (during first month). Maybe we should write unit tests for controllers as well, maybe not. This is unclear. Anyway, TDD is good and we will use it as a mandatory practice.

Acceptance Tests. About 50 tests were created based on SAMIE. We modified tests quite often and spent near half an hour hour per day to keep ‘em working. These tests helped a lot when we performed major refactoring in controllers and GUI layers. But it seems that SAMIE is not the best tool for acceptance testing. It is easy to use, but sometimes unstable and Win32 restricted. But I did not find a better one so far.

Refactoring. It helps to reduce code duplication and keeps system architecture clear. We had doing refactoring constantly, and sometimes we spent whole day on major refactoring. As a result, new functionality implementation appeared easier than we expected. We will use refactoring as a mandatory practice in future.

Walking Skeleton. This is a first, fast and small release, which contains single system feature ‘in depth’. For example, show projects list, add project into the database, check user permissions on project. Walking Skeleton for the system took near 2 weeks. We managed to investigate many architectural decisions and choose the best. Small and fast release is a great thing.

Pair Programming. We created the most important system modules in pair. The code quality and overall system architecture appeared significantly better when pair programming used. Constant code review, communication and experience sharing are really great things. It is not easy to program 7 or more hours in a row, and you feel exhausted in the end of the day. But work satisfaction is great. We will use pair programming for teaching, experience sharing and critical modules implementation in future.

Code Metrics. LOCs dynamic, Unit Tests # dynamic, Acceptance Tests dynamic. Code metrics helped to track project progress and predict team velocity and project size.

Project Site. Helps to organize important project related information (news, documents, actual project state, links) in a single place. However, Project site update should be more simpler. For example, we can use a basic CMS for project site maintenance in future.

O/R Mapping. Tangram was used as an O/R mapping tool. It took about 2-3 days to resolve some tricky issues, but in general O/R mapping speeds up development and supports database abstraction. Maybe OODBMS is a better solution, but if you stick to RDBMS, O/R mapping will help. However, for small systems or web sites O/R mapping is not required.

System Prototypes. Helps to resolve many issues on the early stage. In general, the prototype should be created if project budget allows to do that. But I do not recommend to create a comprehensive prototype, since it just wastes time. Prototype should contain major system modules and maybe some secondary.

Honesty. Just be honest with your teammates. They will trust you, and trust is one of the major factors for jelled teams.

Documentation in Source Code. Almost all system architecture related documentation was in source code. We used a tool to build linked html pages from POD documentation. As a result, documentation maintenance was easier.