Agile Software Development Process: 90 Months of Evolution

Three years ago I wrote an article that describes the changes in our Agile software development processes from 2008 to 2012. Three more years have passed by and our processes were not set in stone. Here I want to provide you with 90 months of changes in our product development practices, company culture, structure and engineering practices. Hope you will find it interesting and learn from our mistakes.

We are still working on the Agile project management software Targetprocess. While we started three small side products last year, 90% of the company is focusing on Targetprocess.

Team Structure

There is a clear pattern of more and more cross-functional teams. We started with just developers and testers in 2009, added feature owners later in 2011, added designers in 2014 and now we even added product specialists into development teams to close the gap between customers and developers. Maybe in future these cross-functional teams will transform into independent business units.

Emergency Team

This is a special team we formed in 2013 to fix bugs and address minor improvements. Initially it consisted of newcomers. The idea was that newcomers spend 2-3 months in the emergency team, learn the codebase on bug fixes and then join some development team. It didn’t work well. Lack of experienced team members with a good codebase knowledge in fact impedes learning.

In April 2014 we decided to remove the emergency team and instead rotate this role between all the other development teams. In August 2014 we paused the rotation to focus on new functionality. As a result, it became much harder to put small fixes and improvements into the roadmap. In a retrospective we admitted this role is required and I don’t think our company can live without this for more than 6-8 months. Overall product quality degrades.

Technology

On UI we have completely abandoned ASP.NET and transformed Targetprocess into a single-page application. The new UI is extremely interactive and novel, but a bit more complex than we desired. We struggled with the UI architecture, but now we have a decent component-based architecture with React.js on top.

We have a somewhat pluggable architecture that allows the customer to extend the product according to their project management needs and business domain, with plugins on the server-side and mashups on the client-side.

We feel that the server-side decisions we made 8 years ago are outdated now, so we have to change significant parts of the product core to utilize new approaches and meet new market requirements.

Process

It is hard to fit all the data into a single table, so I split all practices we use into several areas.

Planning practices

This table shows the evolution of practices related to planning.

Orange background - practice change caused significant problems.
Green background - practice change led to nice improvements.

Iterations

We switched to Kanban 6 years ago and in general didn’t miss anything. It is interesting to note that after several years without iterations some teams decided to try them again. The argument was like “Well, maybe we can increase our development speed with iterations since they will be a pacemaker and set the tempo”. However, iterations didn’t stick and the teams get back to Kanban.

Features prioritization

For years it was a responsibility of a single person to create the product backlog and prioritize high-level features. In 2013 we decided to form a cross-functional committee (we call it Product Board) to replace the Product Owner. It consists of several people representing almost all roles: support, design, sales, marketing, development and QA. This solves many problems. More people in the company understand how product decisions are made, what is the market situation, what problems customers face. We removed this single point of responsibility and now we feel more confident for mid-term planning (1 year). However, there is one thing that the board can’t replace so far — the product vision. It appears that Product Board can handle tactics nicely, but innovations is something out of its scope so far.

Last year we introduced a formal model to prioritize features. We estimate several parameters and calculate the feature score. It is interesting that the model actually works and we rely on it. Definitely we don’t take features blindly one by one, but at least we have a clear separation between top 20 problems we should attack soon and all other problems that can be postponed.

Estimates → No Estimates → Estimates

We used to not estimate user stories till 2013, but the real problem with this approach was a forecast. We didn’t find a way to forecast feature duration without estimates, so we got back to user stories estimation with a very basic and quick process. People often say that there is no correlation between Estimate and Cycle Time and it is quite true indeed in our case as well.

Here is the chart that shows the correlation between Cycle Time and estimated Effort for a single development team:

So maybe our desire to forecast better with estimates is an illusion, but it doesn’t take much time and helps to notice and split large stories better, so we keep this practice so far.

Smaller user stories

To split stories is hard. Period.

Tracking

WIP limits

We’ve discovered that it is extremely hard to imply WIP limits to development teams. We tried several times, but they just didn’t stick. In general WIP limit should affect the cycle time. Here is the average cycle time by quarters for all user stories. You see a clear positive trend — Cycle Time is getting shorter. Most likely other practices helped, but a further Cycle Time reduction may be problematic without good WIP limits.

Metrics

We like physical things, and on the kitchen we created a new roadmap that shows major features, releases, new hires, sales indicators and "fuckups". This focus on fuckups was considered destructive by some people, so we are thinking to replace it by "wins".

We have custom reports in Targetprocess now and can track more interesting trends. For example, we see that we complete more and more user stories every quarter, so the company velocity is increasing.

Retrospectives

In 2013 we were so focused on pushing a new major version of Targetprocess to the market that we abandoned retrospectives. When you work without retrospectives for more than a year something bad happens, you start to feel exhausted and the productivity drops. I really don’t recommend to ignore this practice, it has effect on everything, like team morale, productivity and quality. In 2014 we decided to do retrospectives regularly and each team decides how often it will have them. Interestingly, after this huge pause retrospectives in some teams were quite long, while in few teams they were short with a very brief list of action items. Teams are different and it seems some really jelled teams do not need retrospectives since they solve problems on the fly.

In 2014/2015 we tried all-hands meetings to brainstorm ideas for various problems like productivity and development process improvements. It was a very successful practice and in general it is not that hard to have a full day meeting of 60 people.

Daily meetings

This practice lives for years. We never neglect it (however, some discussions about the termination of the practice did happen). In 2013 we had more than 20 people at a single meeting and it started to be long and boring. So we introduced a rule that one person from every team speaks about the team’s accomplishments and plans, thus we shrank the meeting to 5-7 minutes and we keep this practice till now.

Meetings

Just two meetings have survived a several years timeframe: Daily and User Stories kick starts. It is really important to get the whole team together and discuss a user story before implementation. With the company’s growth we have more global meetings now that include people from several teams or even the whole company. We do company-wide feature demos and infrequent retrospectives. We have Development and Product Boards that solve cross-team problems and set the product vision, these committees meet regularly as well.

Our teams are mature enough and there is no need in frequent retrospective meetings. One new meeting we have introduced is for the team and a product specialist to discuss customers’ requests and bring development teams closer to the real users.

UX and Craftsmanship

UX

We started to form cross-functional UX teams a few years ago and in general it works pretty well. Definitely there are gaps in knowledge, but people provide different perspectives and understand problems better. Usually a UX team consists of 4-6 people, including a product specialist, a developer, a tester, a designer, a support engineer and the feature owner.

We used to have a more formal approach to UX with a single UX team, now things are quite unstructured. Sometimes it slows us down. We are trying to bring a bit more formality into our UX process. The main problem is that feature owners need more knowledge about UX processes and now we are starting the formal education to liquidate this gap.

Craftsmanship

In 2010 we experienced a huge shift in craftsmanship practices, but in 2014 we decided to focus on the development of new features and stopped all educational programs, like conferences and 20% time on own projects. The effect was controversial. While we improved the short term productivity and completed more features, the price was a burnout and an overall morale drop. In retrospective I think it is OK to focus, but one year of focus is the limit. Then productivity drops and you might lose some key developers. It seems it is better to alternate focus and relaxation periods to create a healthy environment and teams that can run a 5+ years product development distance.

Here are the company’s focus changes by year:

Development practices

Build Board

We created an internal tool Build Board. It is a software that integrates Targetprocess, GitHub and Jenkins to automate Definition of Done for every User Story or Bug before merging a feature branch to the trunk.

Developers have to keep in mind various information related to a current task. Is it already tested? Have all tests passed? Is an integration branch open for commits? And so on. All this information is spread among different tools that are usually not integrated. The Build Board provides a single place to handle all that.

Code Review and Green Develop Branch

We introduced code review in 2013 and it was a success. In 2014 we introduced a special rule “green trunk”. It means that any commit to the develop must contain at least one tested User Story or Bug. A person who does the commit must ensure that tests are passed on trunk and revert the last commit if tests are failed. It was hard to enforce this practice, but with time it indeed helped to keep the trunk green.

Continuous Deployment and Branches

Our path of branching is Single → Feature → Single → Feature. The last change is interesting. We hoped to introduce continuous deployment and migrated to a single branch, but it appeared our build/test infrastructure was just not ready. The main two problems were slow tests and unstable tests. It takes years to stabilize tests and create a decent build process. Functional tests are quite slow and unstable, so we re-run them several times to ensure correctness. We have more than 60 virtual servers, but it is still not enough to run tests for less than an hour. Single branch development with such slow tests is just impossible. Master is seldom green and we struggled for several months before we reverted to feature branches.

Stats

Some real numbers that help to understand the size and complexity of the product.

Note that we have much more tests in 2015 and almost doubled virtual machines to run them, but still the build time has increased. We have fought unstable tests, but still don’t have a fast and reliable environment to enable Continuous Delivery. The main lesson learned is that it is extremely hard to have CD for a monolith application. We consider using more and more microservices and deploying each service asynchronously, thus decoupling the system and reducing test/build time for every module.

Wrap Up

In the conclusion section three years ago I wrote "Still our development process is far from perfect". I just can repeat this again. We are getting better slowly and some areas even degrade slightly and demand re-enforcement to shine again, like User Experience. In general our UX processes are OK, but not exceptionally great as we want them to be. The software development process has no destination, it is just a journey.

* Header illustration source

P.S. Reddit discussion