Edge of Chaos | Agile Development Blog

I’ve been thinking about  the influences that might affect the team’s velocity recently. Every single product owner wants to have features delivered as soon as possible. It may seem that this race for a better velocity is the wrong goal and can lead to the ugly “Work faster, basterds!” solution. But that’s not the case in a good and healthy environment .

Here is the diagram I’ve sketched in half an hour to generate some ideas. Bubbles are activities, practices and other entities that can make an impact on the development speed. They can have either positive or negative effect. For example, it seems that only two things improve velocity directly: fast feedback and experienced developers. While there are many other things that slow us down, such us unplanned work, interruptions, multi-tasking, rework, high coupling and technical debt.

The cool thing about this diagram is that it asks very specific questions:

• How to deal with customer requests and reduce unplanned work?
• What to do with urgent bugs?
• How to do more training?
• How to have smaller batches?
• What to do about noise and interruptions?

When you ask such questions on a retrospective meeting, you can expect quite many good ideas. If you just ask:  ”How can we work faster?”,  the answer would be silence and confusion.

Software penetrates every pore of human existence. We look up the weather info over the web, giving up on outdoor thermometers. We’re driving to destinations with GPS navigator (forget paper maps with their G7 sections on page 59). We turn on RunKeeper when riding a bike to calculate the average speed and run and boast in Twitter. We’re using software every single day of our lives. It seems we’re hugging our dear gadgets a lot more than our loved ones.

No one knows the exact how-to of writing great software fast, that’s the problem. Waterfall passed away at the crossing of 2 centuries, whereas new software development methodologies (agile) fail at solving the fundamental problems so far. We’re living in very interesting times. Software development industry grows fast right here, right now, and the foundation for a quantitative leap is building up.

I’m reading books on TRIZ and becoming enthusiastic about  its potential  for software development industry. Yes, it is not clear how to apply it directly, since TRIZ focuses on technical systems, but I believe we can apply general rules and even have solution patterns in the future.

TRIZ has several patterns of evolution. Here are my thoughts about the most interesting patterns and their applicability to software development.

Evolution toward Increased Ideality

Every system generates both useful effects and harmful effects and every system has costs.

Ideality = Benefits/(Cost + Harm)

Software system is not an exception. If we take a project management software, it helps us stay on track,  plan work and see progress. What is the cost? Well, it takes time to add data into the system. It takes time to find useful information. So the system wastes our time. An Ideal Project Management Software (I use big letters to stress its ideality) will add data from various sources fast and provide all information we need in 2-3 clicks.

The other hidden costs? A learning curve is often significant. Migration to other project management software is not easy and painful. Customization sometimes impossible or very expensive. All that should be (and will be) resolved in the future.

Technology Lifecycle

Each new technology follows a S-curve pattern. Slow early adoption, sharp growth with mainstream adoption and finally slow growth to a full saturation.

source: An Introduction to TRIZ

In software development there are plenty of examples. OOP went mainstream years ago and it is not sexy anymore. REST services grows sharply, mobile industry grows sharply, Agile adoption is mainstream now. It is more intresting what is cooking in early adoption phase right now, what will change the future of software development. Is it TRIZ and problem solving techniques? Is it Continuous Delivery? I don’t know, but we’d better keep our eyes wide open and discover trends as early as possible to ride the wave.

Uneven Development of System Parts

“A system encompasses different parts, which will evolve differently, leading to the new contradictions.” Every software system has various parts such as modules, layers and components. If you think about that, you will remember many cases when one part of a system was improved significantly, while other parts of the system stayed as is. Very often this significant improvement leads to new, unexpected innovations and total re-work of existing modules.

Let me provide an example from TargetProcess. We decided to re-write plugins. We were not satisfied with the existing architecture and we were going to find a more efficient approach. We stopped by ServiceBus pattern and implemented the solution. However, then we faced a new problem: how to create UI for new plugins. It was solved via Mashups and it was totally unexpected from the beginning. Now Mashups are a part of TargetProcess and people can do very interesting things via Mashups.

Very often a new module design brings along major changes in system architecture. We decided to re-design Views, and new Javascript architecture evolved (tau js framework). All areas of TargetProcess will be based on this new tau framework as a result. It’s amazing how often this law is actually observed in software development.

Increased Complexity then Simplification (Reduction)

That is my favorite maybe. “There is a tendency for systems to add functions that at first increase complexity but over time collapse into simpler systems that provide the same, or more, functionality.” It is so common for a software system to become more complex, to add new features, more features. There is even a term for that,  coined years ago: bloatware. This law clearly advises the natural path of evolution, and Simplification phase is crucial. However, usually simplification never happens and software product dies.

source: Featuritis vs. the Happy User Peak

I can clearly see this pattern in TargetProcess as well. Initially it was a pretty simple product with a few features. It grew into a quite complex yet powerful agile project management software with many integrations and customization options. In worst times we had about 100 different screens in the system. Everything could be done differently and somethimes you had to visit several screens to complete a single action .

We started Reduction phase last year. We are re-working all the functionality and we have a clear vision on how to shrink all the complexity into 10 screens or even less. It is really cool to see the way and to follow it. It is fun and you have a genuine feeling of the “right way”. We fearlessly removed features that are almost not used, yes. But interestingly enough, new, fewer screens will provide even more functionality than the old ones.

I believe this is the path all software systems should follow. More complexity, more functionality, then Simplification and Reduction.

2 weeks ago we made a large roadmap poster out of four A2 sheets of paper and put it up on the wall in a highly popular place, that is kitchen. The poster shows 2 years in a row:

You can see  initiatives (or epics) rows. Red line is for an epic start date, green line marks release date. Yellow lines show previous and future expectations about releases, but since we are not estimating stories, these are just expectations. Blue line indicates beta-release. Stories and features are represented by yellow sticky notes inside each initiative. The current day is marked by a thread with a marker attached (yes, we utilize gravity force). Future initiatives are marked with large orange sticky notes.

Here is a quick overview of 2010. We completed only 2 major epics and started 3 more that are still in progress.

Currently we are working on Plugins (this epic  has been in production for almost 9 months already) and new Views (the current “in progress” time is 6 months). As you see, we expected to release new Views in April initially, but full JS architecture redesign changed the plans. Quick Add UX activities will be started soon.

Finally, an overview photo:

1. You don’t waste time on estimation

Estimation takes time. Even if you do planning poker and use story points, it still takes time. What do you do to improve estimation accuracy? You gather some data, analyze the data and discuss the results. You are spending time on all that. But are you sure that estimates really provide any value? Most likely not. It is a waste. You’d better spend this time doing something important for the product.

2. You shouldn’t explain to higher managers why it took soooo loooooooong

If you don’t have estimates, you can speak without fear and explain things clearly. You can enumerate problems and explain how they were resolved. You can show that you work really hard and take all the necessary actions to release this story as soon as possible with great quality.

If you have estimates, be ready to hear something like “you screwed up, maaan! You had to deliver this feature till the end of the month! What the f%$k is going on?” as an argument. The team put themselves on a weak side immediately and have to care about deadlines more, not about quality or a better solution. Is it something you really need?

3. You don’t give promises that are hard to keep

You are not relying on people’s optimism (which is built in). People are optimists (almost all of them) and inevitably give optimistic estimates. You can use  complex formulas or very simple rules to have better estimates, but is it really worth it? You can spend many hours defining correct formula for exactly this development team. People will not trust you, since you (rightfully) don’t believe in their estimates. Numbers will look “made up” and still you will have incorrect estimates in the end.

4. You don’t put additional pressure on development team

In some teams Estimate is equal to Deadline. That’s bad. What do you do to meet the estimate? Compromise quality? Tolerate average architectural solution? Abandon polishing? All that leads to poor solutions that end users will not like. It is better (almost always) to spend more time, than planned, and release something really usable and useful.

5. You focus on really important things

Imagine, you have a planning poker session and estimate all stories inside an epic. You sum up all the stories and define it as a 1000 pts epic. You just stick a “fear” label on the epic. People do fear large problems. A 1000pt problem is huge and psychologically it is harder to decide “let’s start it right now”. Product Owner will have a temptation to implement 100 smaller user stories instead of this huge epic. However, it may be a bad decision. This epic may be really important, the morst important thing for the product.

If you don’t estimate it and are not completely sure about its size, you have a better chance to actually start doing it.

Hiring is hard. It really is. There are not so many talented and smart developers in the world, but there are lots of inexperienced, boring, exhausted developers. Surprisingly, so many developers can’t even provide a decent architecture of a simple system! I don’t know how a person with 5 years experience can’t create basic designs, but that is a true fact.

Hiring is about matching. You should find a candidate that matches your company and vice versa. It is not easy. We’ve interviewed about 30 developers over the last several months but hired only 3.

Here is our hiring process.

Initial Email / CV

I pay much attention to introductory email and CV. Things like that make me cry:


Holy shit! Does anybody think I really care about this “2-pages-long-list-of-skills” bullshit? I don’t care how much Linux you did. I don’t care about years of experience in some programming language, platform or framework. What I care about is whether developer can do stuff we need. If we state that we are looking for .NET developer and describe his real job, it means we are looking for a person who can do this job perfectly. No more, no less.

As per our job description, .NET developer is supposed to create architecture for independent components, improve business layer etc. I expect a human friendly email with some useful information e.g. previous projects, problems that were solved, reasons to apply for a job with us and experience and roles outside main job. This email should be quite short.

In fact I’ve never seen a good candidate sending a long skills list. I did invite many people that send such emails and all of them were quite weak for our needs. So, this skills table and a boring CV format are the first sign of a candidate level.

The First Interview. Projects History and Practical Task.

We don’t run phone interviews with developers, we’d rather invite them to a live interview right away (if the intro email is good enough).

For about half an hour I ask questions about previous experience, real problems and solutions. The goal is to find out if this person can really solve problems and to get a general feel about his attitude and personality.

Then we give a practical task — design a simple system. It is a pure business layer design that should be abstracted from database and UI. The system should be easily expandable and simple. Candidates have 2-3 hours to create the design and present it to us on whiteboard. We expect basic knowledge of UML to speak the same language, but it is not mandatory. Ideally we want to see a Class Diagram and (pipe dream!) a Sequence Diagram. How many developers from several dozen candidates draw Sequence Diagram? What do you think? Zero.

Usually 3-4 developers from our team take part in design review. We ask questions, raise challenges and see what happens. How candidate fights for his vision, how he accepts critics, how he explains his solutions. If the result is positive and candidate has some spare time, we do pair programming for 1-2 hours and try to implement the design idea.

This first interview does one thing perfectly: it filters out 95% of candidates that can’t work in our company.

The Second Interview. Theoretical Knowledge.

The second interview is slightly superfluous, but still required, I think. We have a second chance to make sure that this person is really good. We ask questions about various technologies to understand knowledge. Typical questions are “Are you familiar with SOLID principles?” and “What is ORM?” with ongoing discussions. This is not as important as practical skills, but still gives a good overview. I don’t think this part can be skipped altogether.

We ask some questions to tap on how this candidate learns new things. My favorite questions are “Which books have you read recently?”, “What are your favorite web resources?”, “Have you learned something new over the last month?”. Answers to these questions are crucial, as we are looking for people who learn continuously and crave for new knowledge.

We also do more of a personal talk to make sure we can trust candidate and he can trust us.

Then we give candidate a chance to ask as many questions as he wants. Usually people ask 3-5 questions, but one candidate asked about 20 questions after the interview :) (We haven’t hired him, but not for this reason).

That is it. We’ve been using this approach for over several months and it works very well.

Humans receive 95% of information through visual perception. We spend countless hours staring into laptops reading, analyzing, interpreting and feeling information flows. I think visualization is something we lack in many disciplines and project management is not a lucky exception.

Interestingly, latest trends in agile project management do care about visualization. Kanban success heavily depends on great visualization. Everybody loves Task Boards and Burn Down charts in Scrum. As you can see, there is some progress, but I think it is not systematic, just a side effect of other activities. It would be fascinating to create an orthogonal movement in agile project management community that focuses on visualization, I call it Visual Project Management.

There is only one really important rule about visualization:

Visualization should reveal problems, states and trends

Sure, we can add more rules, but they’d be secondary. Imagine you can see a Backlog and diagnose a disease right away:

• too large
• contains too many old user stories and
• grows too quickly

Imagine you see a Board, and at the same instant you grasp all the bottlenecks:

• overloaded people
• problematic stories
• quality problems and
• overall development speed

I can’t provide quick visual solutions right away, but I think this approach will change the way  teams make decisions and improve their development process.

I wonder why we still have no good visualization tools for project management. I think the answer is that project managers are not strong in design, and designers are not strong in project management. Separation of responsibilities leads to basic, trivial decisions. PM should learn design and data visualization techniques to really invent new visualization. Designer should learn PM domain to invent new visualization. The ideal mix is a team where PM knows design and Designer knows project management.

Let’s review some examples to get a basic feeling of how visualization re-define things.

Why Kanban Board is a Great Step Forward

Information presentation really affects the ways we run projects. Here are two screenshots that contain exactly the same information. You can easily say which one is better.

This is a simple list of user stories taken from TargetProcess (I am working at TargetProcess, so forgive me biased screenshots). It contains quite many data about user stories like state, assigned people, effort, etc. However, this data is hidden. To make some decisions, you need to dig  and put some effort into it. It means you have a higher cognitive load. It means you’ll miss some details and sometimes make a wrong decision.

Next screenshot is a simple Kanban Board. While it is not the best example of Kanban Board, still it is good enough to reveal the difference. You see the same user stories on this screen. However, you can quickly identify  there’re too many user stories in In Progress state, but it is not critical, since there are no holes in development flow. You can somehow feel that most likely the team is doing pretty good. In some cases you can’t even quickly explain why you think so, you’ll need to analyze your own feelings to provide logical answers.

But the truth is that you have arrived to some conclusion with enough accuracy, really quickly. That is why visualization is so important in any discipline, including project management.

I believe we can have something much better than a simple Kanban Board. Kanban Board visualizes development flow, but misses other important things like people load, local problems, overall progress for all projects etc.

What happens if we have several teams and want to see their work on Kanban Board? Definitely, the board above will not show that. So far,  I haven’t seen good solutions to such problems. Software products are not good in presenting large amount of data in an interesting, meaningful manner. This should be changed.

Why Gantt Chart is Misused All The Time

Gantt Chart is a great tool as well. It is heavily used in traditional project management, but often with poor results.

Most of the project charts look the same and make the same mistakes: analytically thin, bureaucratic grid prison, not annotated, little quantitative data.
E. Tufte

This Gantt Chart is quite good and useful:

This Gantt Chart is bad and useless:

Stop. Think a little bit. Can you define the difference? Can you say why I’ve made such a conclusion?

First chart shows quite large project phases. Second chart shows very granular tasks. Gantt Chart is not a good tool to handle granularity. It works best to visualize long project phases, like releases or iterations. It just plain fails to visualize 500+ tasks in a project.

Gantt Chart has a large white space, it lacks information density. Thus it should be used carefully to visualize important information, not everything you have.

I think the frustration with Gantt charts arises more because of tool issues. People’s contexts of use may require information that is obscured by a Gantt chart. But Gantt charts are the dominant (sometimes only) visualization in many tools, and it’s difficult to impossible to extract and present the data in other forms.
Brian de Alwis

These were just two examples of visualization in project management. I believe we can be more creative to provide better tools and concepts, invent new ways to present project information and improve transparency. We definitely can do better. And we should.

I will post more about visual project management. This topic is really intriguing and promising to me.

Why do teams gel? Why some teams are trustful, enthusiastic and passionate; while other teams are apathetic and boring? There is no recipe to build a great team. You can’t add 5 grams of trust, 3 grams of fire, pour in some communication and boil till ready.

Why we need teams at all?

Working Alone

Team adds some new qualities in comparison with individual. When you program alone, you don’t need trust. Of course, you most likely trust yourself, otherwise you need  medical help. Besides you don’t need communication skills, if you speak to yourself very often you need medical help as well. All you need as a lonely programmer is problem solving, technical skills and passion to move forward. However, these are not enough to work effectively as a team.

If you work alone, you are free to make any decision fast. You have zero overhead: no meetings, no discussions, no phone calls, no questions.

If you work as a group of people (I mean, team), everything is not that shiny. Suddenly, you have to visit meetings. You have to participate in various discussions about technical issues and solutions. You have to answer questions and sometimes you have to do things you disagree with. Is that all so awful? Yes, it is.

So, why people do form teams? Simply, to solve problems that can’t be solved by a single person. Team is a necessity and it suddenly brings more formality to problem solving:

You have to communicate to have aligned vision. Otherwise you will solve two parts of the problem differently and solutions will not fit.
You have to coordinate work. Otherwise you will have huge delays and inefficient process.
You have to find common language to understand each other.

Team brings an additional overhead.

Working as a Team

In general, an effective team reduces overhead to minimum and lets people really solve problems and focus on main activities (I mean development). If we think about team from this interesting angle, we can easily find good practices to  improve team’s efficiency:

• Less people. Small team has less overhead.
• Common language. All should understand each other as quickly as possible. That is why offshore teams have huge problems.
• Less meetings. Ban all meetings and activities that don’t help to solve problems.
• Minimum people on meetings. Invite only those people who can really bring value to problem resolution
• Team isolation. It is generally a bad idea to have 2 teams working on different projects sit in one room.
• Fast communication channel. In general, the goal is to have less communication. Really. It is. It means if you have to discuss something, you should do that using the fastest available way, and it means in person talk most likely near a whiteboard.

But. Improving team’s efficiency, we can easily decrease team’s creativity. It may happen that team will solve problems quickly, but these solutions will not be the best. It may happen, that some unstructured chats about stupid things cause a genius idea. We can’t predict what will help us to find a really cool solution. Most likely we should not strive for complete rigid team that communicates only about problems and solutions. There should be a slack, and it is always there. People instinctively feel that unstructured chats are good and helpful sometimes.

Trust

You can’t have a good team without trust. Why? Simply because trust reduces communication time and helps to solve problems quickly. Imagine a team where people don’t trust each other. You immediately have political games, suspicious questions, indirect wording and rounded phrases. You have hidden conflicts, but polite discussions on a surface.

If you come and see this team in action, you will be astonished by stupid decisions and dumb solutions. Everybody’s goal is to cover his ass in such an environment.

Trustful team doesn’t spend time on all that shit. Discussions are hot and straight to the point. People can even scream on each other sometimes, but if they trust each other it does not really matter. They critique bad decisions fearlessly and don’t hesitate to provide fresh ideas.

Trust saves time on communications and leaves more time to do the real job.

Passion

We all saw people with extinct eyes. We all saw boring and semi-dead teams that work from 9 to 7 and can’t wait to leave the office. Nobody wants to work in this place. But still many do. I personally can’t understand why people do that. I don’t accept any usual arguments like stability, money, habit. I can’t work in a boring place on boring projects. It is not fun, it is not interesting, it is not valuable.

Passionate team is built from passionate people. That is it. They really care about what they do, they focus on real problems, they do everything to improve things. Passion is an ultimate team’s engine. No passion — no drive.

Wrap Up

Great software development team is built from passionate skilled professionals that trust each other. They collaborate effectively to solve problems and improve team work.

Today I’ve read two interesting posts: The Cost of Code by @unclebobmartin and Code as a Cause of Project Failure by @DocOnDev. They discuss various arguments to prove that all projects fail because of code. The main argument is that if code is free and light-fast to change, project can’t fail. OK. But this case is rather extreme and obviously impossible. We don’t live in a world with hyper-space jumps, teleportation and free medicine (unfortunately). In real life code costs a lot and it will be true in the next decades for sure, so this argumentation proves nothing. In ideal world there is no such thing as code. In ideal world you have a solution in seconds without computers, software and other complicated things. So I don’t buy this idea. Code is not the main reason for projects failure in reality.

Code is not free. Code is expensive. We do not sell source code though, we sell solutions. If it is possible to create a solution without source code, it would be fantastic. Let’s take an industry that handles tangible objects. Automobile industry does not sell carbon and metall — it sells cars. It sells a solution to transportation problem. Teleportation is an ideal solution, but we can’t teleport nothing but electrons, sadly. We buy cars to get from point A to point B. We buy a solution.

Code != Solution

I think the main problem with projects is that they provide either bad solution or no solution at all. Nobody buys stagecoaches these days, since there are more efficient solutions. If a project does not solve anything, it will fail. If a project solves something, but  in a nasty, unusable way, it will fail. You can create an absolutely beautiful architecture with the cleanest code in the world. You may have 100% test coverage, complete separation of concerns, flat hierarchies and methods without boolean arguments. You may have all that beauty, but still fail miserably if the program does not solve user’s problems efficiently.

You may argue that with clean code you can refactor it fast and change everything. C’mon, it will be a full re-write if it solves the wrong problem. Can you fix a stagecoach to make it a true competitor of a car?

On the other hand, if a projects solves a right problem, but with some issues, clean code is very important. You can’t adopt fast without it. You can’t change things and react to people demands.

Don’t get me wrong, I believe that clean code is an important thing, but it is not the most important asset in software development.