Edge of Chaos | Agile Development Blog

I’ve been thinking about  the influences that might affect the team’s velocity recently. Every single product owner wants to have features delivered as soon as possible. It may seem that this race for a better velocity is the wrong goal and can lead to the ugly “Work faster, basterds!” solution. But that’s not the case in a good and healthy environment .

Here is the diagram I’ve sketched in half an hour to generate some ideas. Bubbles are activities, practices and other entities that can make an impact on the development speed. They can have either positive or negative effect. For example, it seems that only two things improve velocity directly: fast feedback and experienced developers. While there are many other things that slow us down, such us unplanned work, interruptions, multi-tasking, rework, high coupling and technical debt.

The cool thing about this diagram is that it asks very specific questions:

• How to deal with customer requests and reduce unplanned work?
• What to do with urgent bugs?
• How to do more training?
• How to have smaller batches?
• What to do about noise and interruptions?

When you ask such questions on a retrospective meeting, you can expect quite many good ideas. If you just ask:  ”How can we work faster?”,  the answer would be silence and confusion.

Almost 2 years ago I published the Flow. Discover Problems and Waste in Kanban post. The idea was quite simple: visualize the flow of a single user story or bug, and track their life cycle to Done:

You can spot such problems like delays and re-work very fast this way:

Now we’ve brought this idea to life. The Flow chart for every user story, bug and feature will be available shortly in TargetProcess v2.22.9.

The chart gives answers to a whole lot of hands-on questions:

• For how many days has this User Story been in this particular state?
• Were there any delays?
• Was there any re-work?
• Who was responsible for the User Story?
• When were Bugs and Tasks added and closed?
• How much time was spent each day?
• Were there any impediments?

Let’s look at some examples. The user story on the chart below has been  in Open state for 25 days (it means, in the Backlog). Then it jumped right into In Progress state. Two developers (Alla and Vladimir) started working on it (so it was pair programming). They’d been working for 3 days and then the story was moved into Re-open state. This is quite surprising, most likely they had to switch to something else (no good). Then they got back and spent 15 days working on this user story. That’s way too long. Most likely there were switches as well,  so this should be investigated.

Starting from Oct-18 the progress was very good: development went smooth, tester found several bugs and they were fixed in 2-3 days. Finally, the user story was released to production with no more delays.

You immediately get a high-level overview: delays and up/down state transitions. It is a clear sign of some problems, the systematic ones or not known so far, but we already have some background info to start an investigation).

Let’s check another example. It looks like the user story on the chart below was taken to development right as it was added. That’s true in fact, since it was a customer request to which we reacted immediately. It was implemented in a single day, and there was a small delay before tester took it to the  testing phase. We found quite many bugs and fixed them in 2 days, everything is fine. But then the completed user story was hanging in Release Branch state for 11 days, and that’s no good.

We’re planning to extend this Flow Chart and put more information there (comments, attachments, etc.) The goal is to provide the complete production timeline uncovering hidden malignant patterns and problems. You should be able to get a high-level overview in an instant and dig into as many details as possible.

Software penetrates every pore of human existence. We look up the weather info over the web, giving up on outdoor thermometers. We’re driving to destinations with GPS navigator (forget paper maps with their G7 sections on page 59). We turn on RunKeeper when riding a bike to calculate the average speed and run and boast in Twitter. We’re using software every single day of our lives. It seems we’re hugging our dear gadgets a lot more than our loved ones.

No one knows the exact how-to of writing great software fast, that’s the problem. Waterfall passed away at the crossing of 2 centuries, whereas new software development methodologies (agile) fail at solving the fundamental problems so far. We’re living in very interesting times. Software development industry grows fast right here, right now, and the foundation for a quantitative leap is building up.

I’m reading books on TRIZ and becoming enthusiastic about  its potential  for software development industry. Yes, it is not clear how to apply it directly, since TRIZ focuses on technical systems, but I believe we can apply general rules and even have solution patterns in the future.

TRIZ has several patterns of evolution. Here are my thoughts about the most interesting patterns and their applicability to software development.

Evolution toward Increased Ideality

Every system generates both useful effects and harmful effects and every system has costs.

Ideality = Benefits/(Cost + Harm)

Software system is not an exception. If we take a project management software, it helps us stay on track,  plan work and see progress. What is the cost? Well, it takes time to add data into the system. It takes time to find useful information. So the system wastes our time. An Ideal Project Management Software (I use big letters to stress its ideality) will add data from various sources fast and provide all information we need in 2-3 clicks.

The other hidden costs? A learning curve is often significant. Migration to other project management software is not easy and painful. Customization sometimes impossible or very expensive. All that should be (and will be) resolved in the future.

Technology Lifecycle

Each new technology follows a S-curve pattern. Slow early adoption, sharp growth with mainstream adoption and finally slow growth to a full saturation.

source: An Introduction to TRIZ

In software development there are plenty of examples. OOP went mainstream years ago and it is not sexy anymore. REST services grows sharply, mobile industry grows sharply, Agile adoption is mainstream now. It is more intresting what is cooking in early adoption phase right now, what will change the future of software development. Is it TRIZ and problem solving techniques? Is it Continuous Delivery? I don’t know, but we’d better keep our eyes wide open and discover trends as early as possible to ride the wave.

Uneven Development of System Parts

“A system encompasses different parts, which will evolve differently, leading to the new contradictions.” Every software system has various parts such as modules, layers and components. If you think about that, you will remember many cases when one part of a system was improved significantly, while other parts of the system stayed as is. Very often this significant improvement leads to new, unexpected innovations and total re-work of existing modules.

Let me provide an example from TargetProcess. We decided to re-write plugins. We were not satisfied with the existing architecture and we were going to find a more efficient approach. We stopped by ServiceBus pattern and implemented the solution. However, then we faced a new problem: how to create UI for new plugins. It was solved via Mashups and it was totally unexpected from the beginning. Now Mashups are a part of TargetProcess and people can do very interesting things via Mashups.

Very often a new module design brings along major changes in system architecture. We decided to re-design Views, and new Javascript architecture evolved (tau js framework). All areas of TargetProcess will be based on this new tau framework as a result. It’s amazing how often this law is actually observed in software development.

Increased Complexity then Simplification (Reduction)

That is my favorite maybe. “There is a tendency for systems to add functions that at first increase complexity but over time collapse into simpler systems that provide the same, or more, functionality.” It is so common for a software system to become more complex, to add new features, more features. There is even a term for that,  coined years ago: bloatware. This law clearly advises the natural path of evolution, and Simplification phase is crucial. However, usually simplification never happens and software product dies.

source: Featuritis vs. the Happy User Peak

I can clearly see this pattern in TargetProcess as well. Initially it was a pretty simple product with a few features. It grew into a quite complex yet powerful agile project management software with many integrations and customization options. In worst times we had about 100 different screens in the system. Everything could be done differently and somethimes you had to visit several screens to complete a single action .

We started Reduction phase last year. We are re-working all the functionality and we have a clear vision on how to shrink all the complexity into 10 screens or even less. It is really cool to see the way and to follow it. It is fun and you have a genuine feeling of the “right way”. We fearlessly removed features that are almost not used, yes. But interestingly enough, new, fewer screens will provide even more functionality than the old ones.

I believe this is the path all software systems should follow. More complexity, more functionality, then Simplification and Reduction.

1. You don’t waste time on estimation

Estimation takes time. Even if you do planning poker and use story points, it still takes time. What do you do to improve estimation accuracy? You gather some data, analyze the data and discuss the results. You are spending time on all that. But are you sure that estimates really provide any value? Most likely not. It is a waste. You’d better spend this time doing something important for the product.

2. You shouldn’t explain to higher managers why it took soooo loooooooong

If you don’t have estimates, you can speak without fear and explain things clearly. You can enumerate problems and explain how they were resolved. You can show that you work really hard and take all the necessary actions to release this story as soon as possible with great quality.

If you have estimates, be ready to hear something like “you screwed up, maaan! You had to deliver this feature till the end of the month! What the f%$k is going on?” as an argument. The team put themselves on a weak side immediately and have to care about deadlines more, not about quality or a better solution. Is it something you really need?

3. You don’t give promises that are hard to keep

You are not relying on people’s optimism (which is built in). People are optimists (almost all of them) and inevitably give optimistic estimates. You can use  complex formulas or very simple rules to have better estimates, but is it really worth it? You can spend many hours defining correct formula for exactly this development team. People will not trust you, since you (rightfully) don’t believe in their estimates. Numbers will look “made up” and still you will have incorrect estimates in the end.

4. You don’t put additional pressure on development team

In some teams Estimate is equal to Deadline. That’s bad. What do you do to meet the estimate? Compromise quality? Tolerate average architectural solution? Abandon polishing? All that leads to poor solutions that end users will not like. It is better (almost always) to spend more time, than planned, and release something really usable and useful.

5. You focus on really important things

Imagine, you have a planning poker session and estimate all stories inside an epic. You sum up all the stories and define it as a 1000 pts epic. You just stick a “fear” label on the epic. People do fear large problems. A 1000pt problem is huge and psychologically it is harder to decide “let’s start it right now”. Product Owner will have a temptation to implement 100 smaller user stories instead of this huge epic. However, it may be a bad decision. This epic may be really important, the morst important thing for the product.

If you don’t estimate it and are not completely sure about its size, you have a better chance to actually start doing it.

Our company is quite small (25 people). Most companies of this size focus on rapid growth and don’t pay much attention to learning. We are different. Fortunately, learning is very important in our company. We boost it in all possible ways:

• We highly encourage people to try and learn new things.
• Every employee can dedicate up to 12% of their time to unstructured learning (5 hrs per week).
• We organize internal trainings twice per months.
• We organize so-called Friday Shows every other Friday where we watch and discuss some interesting videos about UX, development, business, etc.

About a month ago we decided to structure our trainings in a better way. The resulting idea was internal mini-conference. Mini-conference is a full-day event dedicated to sessions and discussions. All sessions are prepared internally. The idea is to have a very focused learning event instead of several trainings over 2 months.

Our first conference will take place this Friday. Here is the program

10:00 Registration, “wake-up” coffee
10:15 Data Visualization Alex Tsayun /45 min
11:00 coffee break /15 min.
11:15 Node.js intro Vadim Gaidukevich /30 min
11:45 MongoDB (NoSQL) intro Oleg Seriaga /30min
12:15 coffee break
12:30 Exploring Good Experience Seth Godin (guest video) /20min
13:00 History of Kanban Anton Marchenko /30 min
13:30 lunch
14:30 Performance Metrics Alex Fomin /1h
15:30 coffee break
16:00 Marketing and Sales @ TargetProcess Andrey Mihailenko /1h
17:00 free discussions.

I am really curious about the outcome. Wil people like it? Will we keep this practice? Not sure. But we are trying new things :)

Today I’ve read two interesting posts: The Cost of Code by @unclebobmartin and Code as a Cause of Project Failure by @DocOnDev. They discuss various arguments to prove that all projects fail because of code. The main argument is that if code is free and light-fast to change, project can’t fail. OK. But this case is rather extreme and obviously impossible. We don’t live in a world with hyper-space jumps, teleportation and free medicine (unfortunately). In real life code costs a lot and it will be true in the next decades for sure, so this argumentation proves nothing. In ideal world there is no such thing as code. In ideal world you have a solution in seconds without computers, software and other complicated things. So I don’t buy this idea. Code is not the main reason for projects failure in reality.

Code is not free. Code is expensive. We do not sell source code though, we sell solutions. If it is possible to create a solution without source code, it would be fantastic. Let’s take an industry that handles tangible objects. Automobile industry does not sell carbon and metall — it sells cars. It sells a solution to transportation problem. Teleportation is an ideal solution, but we can’t teleport nothing but electrons, sadly. We buy cars to get from point A to point B. We buy a solution.

Code != Solution

I think the main problem with projects is that they provide either bad solution or no solution at all. Nobody buys stagecoaches these days, since there are more efficient solutions. If a project does not solve anything, it will fail. If a project solves something, but  in a nasty, unusable way, it will fail. You can create an absolutely beautiful architecture with the cleanest code in the world. You may have 100% test coverage, complete separation of concerns, flat hierarchies and methods without boolean arguments. You may have all that beauty, but still fail miserably if the program does not solve user’s problems efficiently.

You may argue that with clean code you can refactor it fast and change everything. C’mon, it will be a full re-write if it solves the wrong problem. Can you fix a stagecoach to make it a true competitor of a car?

On the other hand, if a projects solves a right problem, but with some issues, clean code is very important. You can’t adopt fast without it. You can’t change things and react to people demands.

Don’t get me wrong, I believe that clean code is an important thing, but it is not the most important asset in software development.

We are using Kanban for product development. It works great. As a Product Owner I can re-prioritize backlog anytime and put things into Planned state when I want to. It works. But. Every Product Owner is passionate about the product. He wants to improve so many things and do that RIGHT NOW. Prioritization is really hard and Planned state has a strong tendency for saturation and overflow. For example, our Planned state has limit of 20 items. In the worst case there were 50 items in it (most of them are bugs and small enhancements). Mea Culpa.

The obvious side effect is that most items are buried in Planned state for weeks.  A more important user story pops out and a small enhancement moves down. As a result, enhancements and bugs were never fixed/implemented. Planned column turned into Backlog column someday. That was a real problem. Yes, bugs were small and quite easy to fix, but more important problems constantly pushed them back.

We got tired of this and decided to try a new practice: The Ultimate Clean Up Day. The rules are quite simple:

1. From the start of the workday everybody who can do programming (including CEO :) take any bug/small story from Planned state and fix it. Repeat till the end of the day.
2. Someone suggested to draw a star on the whiteboard for each fixed bug and find out who is the best bug fixer.
3. Testers verify fixed bugs as soon as possible and cooperate with developers to ensure the fix. Repeat till the end of the day.

We were afraid that the clean up day will take longer than 1 day due to re-opened bugs and so on, but this did not happen. It was a very focused activity resulting in 23 fixed bugs in a single day. It was fun and development team decided to do this monthly. Next clean up day will be in the middle of December and I will re-fresh my .NET skills again :)

There are various ways to support agile team retrospectives. We’ve used all of them, so let me share our experience.

Cadence (usual retrospectives)

If you have iterative development like Scrum or XP, it is very convenient to run usual retrospectives meetings. For example, with 2-week iterations you have such meetings every other week, discuss issues, what worked, what not, brainstorm solutions and new things. We’ve tried mood boards, various formats for issues gathering and for action items tracking. We’ve tried a whole lot of things in 2 years. In general, it worked. But then we switched to Kanban and somehow retrospective meetings faded out…

Is there a better way to improve development process?

Stop-the-Line

We tried to apply stop-the-line practice. It states that as a mistake or malpractice is discovered all responsible people should immediately hold a meeting to resolve/prevent this specific problem. There were several stop-the-line events, but this practice did not survive. Why? There are two main reasons.

1. It is very disruptive. People are working on various unrelated tasks and are in the flow. Suddenly they should switch to a problem resolution brainstorming. Many people don’t like to do that.
2. It may take too long. Sometimes the problem is very hard and it takes much time to find a good solution. People impatiently drink coffee and want to get back to work.

So we dropped stop-the-line practice and replaced it with a pure pull system.

Pull / Issues Board

Issue Board is a very simple concept with 3 basic rules:

1. Every person in a development team can write a problem or a new idea he wants to discuss on a whiteboard (Issues Board).
2. There is a limit of 3 problems on the board.
3. When there are 3 problems on the board, we have a retrospective meeting right after a daily meeting.

We have been using this approach over the last several months and I like it most. It leaves off some problems both of the stop-the-line approach and the cadence approach. First, if there are no issues or ideas, there’s no need to have a meeting :) Second, no interruptions, since daily meeting is interruptive by itself, so it is just natural to have a retrospective meeting right away.

If you have other approaches to retrospectives, go ahead and share them!

In Part I I’ve described 5 mistakes in agile adoption, this part has 5 more.

6. CST Knows Everything

Certified Scrum Trainer is not a God. Yes, he knows a lot and has a decent experience. There is a high chance that he may be a capable person to help with agile adoption, but only help. He may know nothing about your domain, about your unique situation and about root problems of your organization. If you rely solely on CST and delegate agile adoption to him, it will fail.

Everything should start from the culture and people. It means everything should start with you. You should share agile values and support CST with all the passion and force you have to make agile adoption a success.

7. Functional Departments Should Survive

You are starting a new agile project. Design manager delegated one designer to the team, but refused to re-allocate his desk. QA Manager delegated several testers, but refused to re-allocate them. Developers sit together, but all the other team members are separated by walls. Sounds familiar? Most likely there is no cross-functional team there, and agile adoption will suffer. 

There are absolutely no reasons to keep functional departments and functional teams. People should work together as a team whenever it’s possible. Definitely, when you have testers in US and developers in India, it is hardly possible. But it is just plain dumb to NOT have cross-functional team if everybody sits in the same building.

Cross-functional teams have so many positives, and no negatives. With cross-functional teams you improve communication, reduce functional competition, simplify problem solving, enable ad-hock process improvements and creativity. Again, there is NO reason to work with old traditional functional teams.

8. We Can Live Without Customer Feedback

Agile is mostly about fast feedback. Feedback from customers is the most important. If you build something with wrong methods, that’s bad. But if you build a wrong thing, that’s just terrible. Why? You will have to throw it away later. Regular (and fast!) feedback from customer is like a sailing direction in a bay with reefs. You constantly correct the course based on wind change and other factors. Customer is the most valuable team member and should be treated accordingly.

Extreme programming recommends to have customer on-site. While this is a great tip, it is rarely practical. Still, you have customer available remotely all the time to answer questions and communicate about a product. If you can’t have feedback in a reasonable amount of time, agile methods will not work. You may build a technically perfect product, but yield zero customer’s satisfaction in the end.

9. Self-organization is Easy

Scrum heavily advertises self-organization. Complexity theory has something to say about it as well. Self-organization is based on a set of (simple) rules, non-linearity and interactions between agents (in our case between people). As a Scrum Master you should just set some (widely known) rules, protect the team and expect the team to self-organize. Right? In reality it is not that simple and no, it will not work out. I’ve never seen self-organization working with just a set of rules.Self-organization in a software development team needs more, it needs leadership. I believe that self-organization can’t happen without leadership.

Pure self-organization assumes that a leader will emerge. That’s not happen frequently, in many cases team will stagnate and fluctuate around mediocrity without a leader. Leader sets a vision and pushes team to the right direction. Leader empowers confidence, passion and self-reflection. This leads to self-organization eventually. 

What happens when leader leaves the team? In most cases it falls back or degrades slowly. This is a clear sign that self-organization was not there. True self-organized team will keep its values and progress without a leader.

10. False Goal (e.g. Customers asked us to be agile)

If you have a customer who insists on agile process for his project, you should praise Heaven for this gift. Use this chance as a turning point for agile adoption.

Unfortunately, many companies just try to “emulate” agile adoption with a desire to get this contract. They send some people to CSM courses, purchase an agile project management tool and apply Scrum superficially. They do all that without deep goals and culture change. They do all that without passion. They do all that with a false goal. Almost inevitably there will be the following symptoms:

• Sprints fail. Always.
• No commitment. 
• Scrum Master assigns people to user stories.
• Testing phase in each sprint.
• etc.

The result of “false goal” agile adoption is failure and a long-term disappointment with agile software development.

I encourage you to share another mistakes in agile adoption!