Edge of Chaos | Agile Development Blog

I’m reading books on TRIZ and becoming enthusiastic about  its potential  for software development industry. Yes, it is not clear how to apply it directly, since TRIZ focuses on technical systems, but I believe we can apply general rules and even have solution patterns in the future.

TRIZ has several patterns of evolution. Here are my thoughts about the most interesting patterns and their applicability to software development.

Evolution toward Increased Ideality

Every system generates both useful effects and harmful effects and every system has costs.

Ideality = Benefits/(Cost + Harm)

Software system is not an exception. If we take a project management software, it helps us stay on track,  plan work and see progress. What is the cost? Well, it takes time to add data into the system. It takes time to find useful information. So the system wastes our time. An Ideal Project Management Software (I use big letters to stress its ideality) will add data from various sources fast and provide all information we need in 2-3 clicks.

The other hidden costs? A learning curve is often significant. Migration to other project management software is not easy and painful. Customization sometimes impossible or very expensive. All that should be (and will be) resolved in the future.

Technology Lifecycle

Each new technology follows a S-curve pattern. Slow early adoption, sharp growth with mainstream adoption and finally slow growth to a full saturation.

source: An Introduction to TRIZ

In software development there are plenty of examples. OOP went mainstream years ago and it is not sexy anymore. REST services grows sharply, mobile industry grows sharply, Agile adoption is mainstream now. It is more intresting what is cooking in early adoption phase right now, what will change the future of software development. Is it TRIZ and problem solving techniques? Is it Continuous Delivery? I don’t know, but we’d better keep our eyes wide open and discover trends as early as possible to ride the wave.

Uneven Development of System Parts

“A system encompasses different parts, which will evolve differently, leading to the new contradictions.” Every software system has various parts such as modules, layers and components. If you think about that, you will remember many cases when one part of a system was improved significantly, while other parts of the system stayed as is. Very often this significant improvement leads to new, unexpected innovations and total re-work of existing modules.

Let me provide an example from TargetProcess. We decided to re-write plugins. We were not satisfied with the existing architecture and we were going to find a more efficient approach. We stopped by ServiceBus pattern and implemented the solution. However, then we faced a new problem: how to create UI for new plugins. It was solved via Mashups and it was totally unexpected from the beginning. Now Mashups are a part of TargetProcess and people can do very interesting things via Mashups.

Very often a new module design brings along major changes in system architecture. We decided to re-design Views, and new Javascript architecture evolved (tau js framework). All areas of TargetProcess will be based on this new tau framework as a result. It’s amazing how often this law is actually observed in software development.

Increased Complexity then Simplification (Reduction)

That is my favorite maybe. “There is a tendency for systems to add functions that at first increase complexity but over time collapse into simpler systems that provide the same, or more, functionality.” It is so common for a software system to become more complex, to add new features, more features. There is even a term for that,  coined years ago: bloatware. This law clearly advises the natural path of evolution, and Simplification phase is crucial. However, usually simplification never happens and software product dies.

source: Featuritis vs. the Happy User Peak

I can clearly see this pattern in TargetProcess as well. Initially it was a pretty simple product with a few features. It grew into a quite complex yet powerful agile project management software with many integrations and customization options. In worst times we had about 100 different screens in the system. Everything could be done differently and somethimes you had to visit several screens to complete a single action .

We started Reduction phase last year. We are re-working all the functionality and we have a clear vision on how to shrink all the complexity into 10 screens or even less. It is really cool to see the way and to follow it. It is fun and you have a genuine feeling of the “right way”. We fearlessly removed features that are almost not used, yes. But interestingly enough, new, fewer screens will provide even more functionality than the old ones.

I believe this is the path all software systems should follow. More complexity, more functionality, then Simplification and Reduction.

At TargetProcess we develop by feature. This means that all the user stories and bugs have multiple branches. We maintain quite many builds, as branches should be tested one by one. Visual Builds Board shows statuses for all the builds. This simple .NET application retrieves data from Jenkins and TargetProcess and makes it visual.

See the screen below. Passed builds are green, orange builds are running now, failed builds are red.

The board shows where exactly the build is failing. For this one, it’s Unit Tests:

The build below is in progress and 3 phases have already passed. Functional tests are running now:

Here’s a full build history for a particular branch:

That’s how it looks live:

The Visual Builds Board has been created by @AlexSane, @eugenekha and @AlexTsayun

2 weeks ago we made a large roadmap poster out of four A2 sheets of paper and put it up on the wall in a highly popular place, that is kitchen. The poster shows 2 years in a row:

You can see  initiatives (or epics) rows. Red line is for an epic start date, green line marks release date. Yellow lines show previous and future expectations about releases, but since we are not estimating stories, these are just expectations. Blue line indicates beta-release. Stories and features are represented by yellow sticky notes inside each initiative. The current day is marked by a thread with a marker attached (yes, we utilize gravity force). Future initiatives are marked with large orange sticky notes.

Here is a quick overview of 2010. We completed only 2 major epics and started 3 more that are still in progress.

Currently we are working on Plugins (this epic  has been in production for almost 9 months already) and new Views (the current “in progress” time is 6 months). As you see, we expected to release new Views in April initially, but full JS architecture redesign changed the plans. Quick Add UX activities will be started soon.

Finally, an overview photo:

Feedback cycles are very important in software development process. To create cool products we need to get feedback as early as possible, and we need this feedback fast. Design Ideas Board is an interesting way to collect feedback from co-workers on design and UI decisions.

How it works? Mount a large cork board in a busy office location with good people traffic. Anyone can pin a design, a wireframe or anything like that. Then anyone can use sticky notes to provide feedback (make sure that marker and sticky notes are available right there).

Here are some examples: a new web site product page design idea with several critical notes:

A view area redesign sketch with ideas on tags improvements:

Here’s the full view of  our Design Ideas Board:

Why is this board so good? It works as a live information radiator, improving design WIP visibility and fostering team collaboration.

Do you know how to run a really productive meeting? I don’t. I’m learning, and I run meetings with various success so far. My most recent insight is related to the size of a meeting group. Let’s evaluate various sizes and identify strong and weak sides.

Presentation

If you have hundreds of people all you can do as a single group is to listen. You have one presenter that broadcasts information, all the other people consume the information and digest it inside. On the picture below presenter is marked with  red color as a sign that he is the most (and the only) active person there. There is a huge diversity in this group, but there is no feedback, so decision making is impossible. All you can do is listen, think of some new ideas and write them down to discuss later.

Large Meeting

Ten or more people sitting at a single table make a large meeting. Again, there is a great diversity among people, but… Usually several people are almost completely silent, they just listen and do not give anything back. There are several very active people that lead the discussion mainly. The real problem with such a large group is that nobody can consume information from more than 1 person simultaneously. It means when somebody talks, all the others should shut up and listen. Often a large group splits into smaller groups randomly and neighbors discuss something  ignoring the rest of the group. I believe this happens when people ‘feel’ the need for a better communication format. Overall, it is very hard to have any meaningful decision in the end and very often the result of such a meeting is … Yes! To schedule another meeting!

Small Meeting

This is the real problem solver. 3-4 people communicating intensively have more chances to really get the problem and nail the decision. Communication is very dense and there are many feedback loops. You say something, I catch new ideas and say them back. Broadcast-consume-broadcast cycle repeats many times. All people are active. Everybody feels the pace and there is no place for boredom.

Duo Meeting

This is not a meeting, but rather just a discussion. This type of communication happens all the time in pair programming, for example. Communication is very rich and dense, however, sometimes there is a lack of diversity. That is why pair sometimes brings a third person into their discussion to solve a doubtful problem. 3 people can solve problems more effectively than 2 people.

Solo Talk

If you talk alone, probably it is better to visit a doctor. It is much more efficient to have an “inner dialog” than thinking aloud. Surprisingly, thinking is a quite good method to solve problems. For example, I like to think alone. However, sometimes you get stuck and see no fresh ideas coming to mind. Moreover, solutions that you invented may be really bad (or not as good as you think). Group discussion is still the best way to evaluate ideas and select the best.

Now let’s have a look at the resulting chart. It shows meetings by 2 axis: intensiveness of communication and groups diversity.

I think it is better to run meetings with 3-4 people to solve problems effectively and split large groups into many small groups. A small group has enough diversity and intensive collaboration to be effective. Several small groups have even greater diversity and potentially can create an even better solution than a single small group. Sure, it will be more expensive and slow process if you invite many people. Just consider several possible options for your next meeting.

Design Studio is a quite simple and efficient way to run UX meetings. Yesterday we tried it for the first time. There are several variations of Design Studio, we made it simple for the first run and set the following rules:

1. Define problem.
2. Sketch 5 ideas individually. No more than 5 minutes per idea.
3. Present and categorize ideas to the team.
4. Discuss positive and negative sides of the ideas.
5. Select interesting ideas and create 2 versions for each final final solutions.

We ran Design Studio for the first time to design Email Integration plugin UI. This plugin allows to setup emails integration and define some rules like “bind email to project and create request”. This way TargetProcess retrieves emails and does some interesting things.

Alex, UX designer, is asking some questions.

Sketches categorized by ideas: 1 vertical line = 1 category.

Sticky notes show votes for each idea. 3 ideas have been selected as top ideas.

Hot discussion.

2 most repeated ideas: define email rules via drop downs and via drag and drop

The wall overview

Resulting Sketches

Setup email rules via drop downs

Setup email rules via drag and drop

In general, the meeting went fine. It took about 2 hours to generate, discuss and select ideas. 8 people participated in the meeting, and it seems too many. I think it is much better to have 4-5 people, no more. Also the ideas convergence phase should be improved. Too many people generate too much buzz, so it is harder to make decisions. Anyway, we are going to try Design Studio again and adjust it to our needs.

4 people from TargetProcess attended London UX Conference this year. It was a decent event and here are mine “take-aways”.

UX + Agile

Alan Cooper read a solid visionary lecture about UX future and adoption. The main trend is to mix agile software development practices with user experience teams. There is a new initiative called Balanced Team. He did not spend much time on this topic, but that is something I am curious about. We’ve been adopting UX + Agile at TargetProcess since 2009 and definitely want to do that efficiently.

It was nice to be reassured that we are going in the right direction as a company. We already do many things  Alan mentioned in his speech.

Visual Meetings

Our meetings might be better. Sometimes they are too long. Sometimes too boring. Sometimes there is no clear outcome. Quite many workshops and sessions at London UX 2011 were dedicated to better, more productive and more fun meetings. Sunni Brown ran a fantastic workshop:  The Art of Graphic facilitation (a.k.a. “How to Run A Workshop with Pictures”). It was very funny and useful. We had several short creative meetings and drew a lot of sketches. I highly recommend her Gamestorming book.

Kate Rutter’s session Design Patterns for Fantabulous Collaborations was very informative and I’ve learned new ways of running efficient meetings, but the practical part was a bit boring and out of context. Overall, after this conference I have a clear understanding how we can improve our meetings which is good. Best Practices for Facilitation looks like a “must read” book, but it seems you can barely purchase it.

We often do ad-hoc meetings, without any preparation. We go like “Let’s discuss this user story with filters”. However, such meetings should be planned. All attendees should be prepared. There should be a right balance of order and chaos. Our meetings are on the chaotic end, so we have a huge space to improve.

Sketching, Sketching, Sketching

Everybody should be able to sketch. It means we should train people to not be afraid to draw something and express information with words AND pictures. Adults often prefer to write and don’t like to draw their thoughts. Mostly because they’re wary of looking  “unprofessional” and “ridiculous”. We should break this prejudice. Sketching is a very powerful technique that allows people to try and express ideas blazingly fast. Bill Buxton wrote an incredible book Sketching User Experience. Read it and you will understand why sketching is so cool.

1. You don’t waste time on estimation

Estimation takes time. Even if you do planning poker and use story points, it still takes time. What do you do to improve estimation accuracy? You gather some data, analyze the data and discuss the results. You are spending time on all that. But are you sure that estimates really provide any value? Most likely not. It is a waste. You’d better spend this time doing something important for the product.

2. You shouldn’t explain to higher managers why it took soooo loooooooong

If you don’t have estimates, you can speak without fear and explain things clearly. You can enumerate problems and explain how they were resolved. You can show that you work really hard and take all the necessary actions to release this story as soon as possible with great quality.

If you have estimates, be ready to hear something like “you screwed up, maaan! You had to deliver this feature till the end of the month! What the f%$k is going on?” as an argument. The team put themselves on a weak side immediately and have to care about deadlines more, not about quality or a better solution. Is it something you really need?

3. You don’t give promises that are hard to keep

You are not relying on people’s optimism (which is built in). People are optimists (almost all of them) and inevitably give optimistic estimates. You can use  complex formulas or very simple rules to have better estimates, but is it really worth it? You can spend many hours defining correct formula for exactly this development team. People will not trust you, since you (rightfully) don’t believe in their estimates. Numbers will look “made up” and still you will have incorrect estimates in the end.

4. You don’t put additional pressure on development team

In some teams Estimate is equal to Deadline. That’s bad. What do you do to meet the estimate? Compromise quality? Tolerate average architectural solution? Abandon polishing? All that leads to poor solutions that end users will not like. It is better (almost always) to spend more time, than planned, and release something really usable and useful.

5. You focus on really important things

Imagine, you have a planning poker session and estimate all stories inside an epic. You sum up all the stories and define it as a 1000 pts epic. You just stick a “fear” label on the epic. People do fear large problems. A 1000pt problem is huge and psychologically it is harder to decide “let’s start it right now”. Product Owner will have a temptation to implement 100 smaller user stories instead of this huge epic. However, it may be a bad decision. This epic may be really important, the morst important thing for the product.

If you don’t estimate it and are not completely sure about its size, you have a better chance to actually start doing it.

What is the recipe of a successful company? How to become an expert in any discipline? People are struggling to get answers to these questions. We have thousands of books full of business advice. We have terabytes of information on any subject. (I believe 80% of information is pretty useless. It contains no real value, just a compilation of other sources.) OK, back to the topic.

Skills

I am reading a very interesting book (in fact it is a list of research papers)  Nature of Expertise. This research is dedicated to human expertise. What is the difference between an expert and a novice? How can we train novices to become experts faster? The general notion about expertise is that it takes about 10,000 hours (~10 years) to become an expert in a complex discipline like programming, chess etc. Yes, there are exceptions, but you have to spend ~10 years on average to become a great software developer.

I’ve contemplated that for over several weeks and tend to think that this is nearly true. Only now I really feel I can do something significant. I wanted this 5 years ago, but obviously was not ready enough. I lacked some skills such as User Experience and Leadership. I was quite bad as a CEO, but improved a lot over the last years.

Success

An even more interesting question is whether we can apply the same rule to companies. It seems we can. If it takes 10 years for a person to become an expert, it seems it takes about 5 years for a company to find its way. It means that on average a company is struggling for the first 5 years, gains some experience and is then capable of doing something significant. Let’s take some random examples. I’m not picking companies that meet the criteria above, just some companies I know and for which I was able to find history records:

OK, it seems there are 2 groups of companies: some of them are really quick and did something cool right from the start. Some of the companies follow the 5-years rule. Also, it seems that fresh startups are more apt to succeed quicker. Does it mean that the years-to-success number is decreasing? Maybe. It will be compelling to verify this tendency. It can also mean that years-to-failure is decreasing as well.

There is a very interesting data about 100 largest publicly traded software companies that shows how fast they hit $50m revenue.

There are 3 groups of companies: Rocket Ships ($50m in 6 years or less), Hot companies ($50m in 12 years or less) and Slow Burners. On average, 8 years is required to have $50m revenue. I believe it is somewhat consistent with the 5-years milestone. Obviously, it takes time to grow when you have created something cool. Usually it is not an immediate effect.

Rocket Ships

I think it takes a good deal of time to research many companies, identify their first key milestones and find out typical patterns. There are so many factors like company size, management style, environmental changes etc. Still it will be very cool research to do and I believe it will be possible to evaluate companies based on its results.

Does the “10-5 law” exist? I don’t know. But. If you can’t master something in 10 years, maybe you should focus on something else. If you can’t do something great as a company in 5-6 years, maybe you should change your business goal.

Editor: Olga Kouzina