Edge of Chaos | Agile Development Blog

Bringing together agile philosophy and software development outsourcing has been one of the hottest topics over the last decade.  Lots of blog posts, discussions in social networks — people try to figure out for themselves if it’s at all possible to align agile methodology with the existing reality of outsourced projects.

How can vendors who practice agile gain new customers? How can customers as newly-converted adepts paste their vision of agile thinking on their existing outsourcing partners?

First of all, let’s look back into when and why actually did it all start with IT outsourcing. Manufacturing outsourcing existed since long ago, but IT offshoring only started at the end of 80’s - beginning of 90’s. Companies jumped at the opportunity to save bucks and use cheap labor not only for producing goods, but for producing software. So, the main point is that from the very beginning outsourcing has all been about saving money. No other notable motivation - just saving money and using cheaper labor.

Next, along comes agile manifesto. People start seeing that the waterfall approach  they’ve been using with their outsourcing vendors is not that good after all.  Fixed price contracts do not guarantee real value (Scott Ambler writes very well on that in this article). Next, the more labor is outsourced to some country, the higher are the costs, so the main point for outsourcing which is cost savings makes no sense any more.

There’re other even deeper-lying consequences. On the one hand, the country which outsources - or businesses in this country not the country itself - they save bucks but lose in the long run as they do not grow their own engineering minds, let alone all the problems that you have working with remote teams - yes, we have all this telecom in place, but nothing ever will replace face-to-face communication. If you often go on business trips to the outsourced destination to talk to your team - again, it’s more costs.  On the other hand, outsourcing makes a dangerous long-term impact on the recipient economies as well.

But the point is not about how bad or good outsourcing is.  Vendors have written an array of blog posts on how to align agile and outsourcing - and that’s natural - they want to keep going, so they do everything to back up their point of view proving that agile goes well with outsourcing. This might work true in some cases.

The companies that outsource on the other hand - they have legacy outsourcing teams. They need to get going with them as well, to stand up to all the funds already invested in their outsourcing center/provider.

So with all this outsourcing in our hands - what do we do?

As an agile outsourcing vendor, you should be ready to invest lots of time and effort to educating your new customers on the value of agile and to building a solid relationship. This might be a very difficult task since some people just don’t want to get educated and prefer to stick to good old fixed price bids, logging and billing for gazillion of change requests, lack of communication and other “joys” of classical outsourced development.

As a company with established outsourcing facility, you’re better off. But perhaps you could be even better off if you started this relationship not with an overseas company, but with a guy next door at least.

Anyway,  you are where you are - so you would need to work with your outsourcing partner to practice the agile approach, since at the end of it all work with agile methodology brings real value, as opposed to counting short-term waterfall pennies and losing long-term gain.

We’re all tied together by things we do. Projects we work on, conferences we go to as a whole team, even bugs we fix together, problems we solve together. Insights we share together.

Some people call it team spirit. Some people call it good collaboration. In any case, this is the light of live communication and talk between people.  My deepest belief after all is that no matter how sophisticated telecom stuff gets our days, there’s nothing more valuable - and ROI-generating as well - as the real talk.

I was perplexed the other day as I saw the phrase  - “we’re trying to maximize our face time with them” - this was said about some guys from an off-site location visiting the main office. My first thought was - so the rest of the time they work  is their a** time? As obviously software guys spend most of their time sitting on their derrier?? Though of course it’s worth a praise that after all they’re still trying to maximize the face time..

As hype is the statement that with globalization you could get the best software developers out of anywhere in the world, as eternal is the truth that to produce people need to REALLY collaborate. This goes for software, for any production.  So it’s not about picking up people as vegetables on the market - even if it’s a huge market. People are more than vegetables. They need live communication to feel alive, to collaborate and to be productive (for those who want to count figures, you can sit down now and compare the monetary value of pure programming skills  - with no communication skills at all).

I’m not saying that we should all go local in our quest for best products and profits. But what I really see is that people are squeezing their way to follow some of agile communication practices using telecom. As a proof, look at hot discussions at LinkedIn agile groups, and Jean Tabaka’s observations on the reasons of agile adoption failure.

Blasphemy to the religion of remote/distributed teams - but the picture is slowly starting to get clear for me:  it really takes good skills to balance the equilibrium of infrastructure and management to activate collaborative agile work in remote teams. Meaning 5 here, 7 there, 6 elsewhere - as one team.

Is it possible to trust without seeing each other? I doubt there’re teams who are absolutely OK with remote customers/product owners etc. I even suspect that waterfall can be the best solution for remote teams-customers that haven’t developed enough trust to each other. If anyone has real-life stories to prove that I’m wrong, speak up.

Kanban is becoming a popular agile tool. Indeed it is very good for software projects of certain types. However, there is a danger of false reasons behind Kanban adoption.

#1. User Stories Diversity

“Our stories vary in size a lot from 1 point to 40 points. Large stories just do not fit into an iteration”

It is very easy to say you can’t split user stories, thus iterations should be abandoned. An easy solution is not the best one. There is something important behind the fact you can’t split stories. Most likely you don’t know how to split stories right. It is quite hard from the beginning and demands creativity.

Moreover, according to Queueing Theory, it is better to have small stories with roughly equal size. Small batches with equal size improve flow (and you have a better and more predictive cycle time). So in Kanban you still have to split stories to smaller stories and try to make them equal size.

#2. Failed Iterations

“We can’t complete most stories in a single iteration”

There may be many, many reasons behind. Mini-waterfall approach, velocity greed, large stories, manual testing, poor stories description, etc. You may even have a wrong iteration length. For example, in large projects 1 week iteration may be an overhead with a huge transaction cost. So 1 month iteration may be much better in this case.

You need to analyze true reasons, the roots of the problem.

#3. Failed Retrospective Meetings

“Retrospective meetings are waste, they do not help in process improvement and we want to remove them”

You just don’t do these meetings right. One popular failure is no Action Items after the meeting. Without action items a Retrospective Meeting is just a kind of informal chat that you may have over a glass of beer. You meet, chat about your problems, and get back on the same track next day. Rule #1 is to collect and write the list of Action Items that you will try to accomplish during the next Sprint [or any other time box].

One more  common pitfall is to skip action items execution. You may collect them, but not really try. You may even try them, but abandon too quickly. Almost all new rules or practices put you off the comfort zone. It takes time to learn them, use them and like them (or dislike), but it should be an expert opinion, not just a gut feeling.

If you expect that you will replace failed retrospective meetings with a nice and simple “stop the line” rule, you’ll fail, since it is even harder than scheduled regular retrospectives and demands even higher level of self-discipline.

#4. Shared People / Functional Departments

“We have a single pool of developers and share them between projects. We can’t form stable project teams”

Do you really think that Kanban will solve your social and organizational problems? C’mon! It helps to visualize flow and find bottlenecks, but if you don’t have a cross-functional team — you have hard times. It is proven in so many sources and researches that cross-functional teams perform better, produce better results and better software.

If you are experiencing difficulties planning sprints with shared pool of developers, try to fix the root of the problem first - switch to cross-functional teams and eliminate multi-tasking.

#5. Simplicity

“Kanban is so simple! No plans, no estimations, no iterations, no overhead”

Indeed Kanban looks simple. But it provides nothing interesting by itself. To ensure a successful Kanban adoption, you need to apply Lean principles first. This new buzzword may sound like a silver bullet, but obviously it  is not. Hard work, discipline, target on perfection and constant improvements - all that is required to apply any agile methodology.

Don’t get me wrong. I am not saying that Kanban is bad. We use it within TargetProcess and I personally believe it works way better than iterative development (for us).  I just want to make a point that you’d hardly resolve all your underlying problems as you switch to Kanban.

P.S. I usually don’t read posts like “10 reasons to …” Can’t believe I wrote such post myself!
P.P.S. Read next post 5 right reasons to apply Kanban.

I am really glad that finally we released v.2.14. Check the press release: TargetProcess Announced v.2.14 Release With Full Waterfall Support.

Everybody knows that multitasking is bad in general. Time is wasted due to context switching and more errors due to insufficient attention. There are many opinions and articles about multitasking problems and general advice is to eliminate it when possible. However, it is required to define what the task is in multitasking before making any conclusions. For example, is it possible to consider waterfall a multitasking process?

For simplicity purposes, let’s assume that our development process has 4 stages: specification, design, implementation, testing. We have a bunch of requirements that may be implemented in the project. In the design stage we focus on the system design only. We think about objects, patterns, communication, API, etc. Looks like quite a focused activity, doesn’t? However, at the same time we are designing many features. We are trying to consider them all and create a design that supports all of the features at once.

There are two facets to this problem:

Activity-based:

From this point of view we have a single focused task - system design.

Feature-based:

Here, we have many tasks (design feature 1, design feature 2, etc.).

What is a better view? I think it is required to focus on delivery. In the end our ultimate goal is to ship working software with a set of features. We do not ship system design, requirements documents or acceptance tests. We ship functionality (features). Feature that solves a real problem is the most important thing in software development. That is what we produce. We should focus on features (that solve real problems in a most efficient way). We should not focus on activities; they are just a part of development process. If you can release working bug-free software without testing stage - that is perfect. If you can release it without design stage - fantastic. Following common/defined/approved/considered-best process is not a rule of thumb. If you can release working, usable, high quality, efficient software without any formal process - that is fine. I am not trying to diminish the importance of processes in the software engineering, but too many people developing software using waterfall, loose focus on their ultimate goal – to ship a product that user wants!

It is better to consider feature as a task (in a multitasking term, it means that multitasking and “multi-feature” are interchangeable). Having this in hands we can say that waterfall is a kind of multitasking. It leads to many features in production cycle, thus eliminating focus. Development team tries to follow defined stages with the best possible output, but the goal is often blurred. People focus on specification, design, implementation, testing, but often they loose focus on features. Why? Well, two reasons: context switching and insufficient attention.

For example, in the design stage you have many features to design and can’t prove all design concepts for all of them. During implementation stage you face many problems with wrong initial design decisions and have to change and adapt design quickly. That is natural, since design can be proved only during the implementation, there is no way to get it right the first time. However, such quick decisions may be uncontrolled and uncoordinated. They ruin the original fancy design. Insufficient attention? Yes, since it is impossible to design and prove ALL the features during the design stage.

The same is true for the testing phase where we have many features to test. We can’t focus on one feature, we feel that there are many more features to test and most likely will not complete full testing of one feature. We feel that it is better to test all features briefly than test some features deeply. We briefly test feature A, then jump and briefly test feature B hoping to get back to it later and complete the testing. We have many features to test causing insufficient attention. Often under time pressure you can’t get back to briefly tested features, and that leads to bugs in post-production period. Large testing stage blurs focus on features testing. It may sound strange, since it is the testing stage after all - but think about it once again.

Iterative development (especially using short iterations) solves these problems in the roots. You have few features in the production and may focus on them. It is easy to fully test several features. It is quite easy to design and prove the design right away (in waterfall you may have several months gap between feature design and implementation). You have a clear focus by not having to switch to other features outside of the iteration scope.

Iterative development is a vertical slicing. You implement “by Feature”. Waterfall development is a horizontal slicing. You implement “by Activity”. What do you prefer?

Hammer and Warm Wax

See Part I

.

Let’s take an average software project with full plan created up-front. It appeast that it is hard to support project plan in actual state using MS Project. It is hard to reflect real progress. But why? MS Project is great software, it should be easy. And it is easy in many cases, but not in software development. MS Project is useful when:

1. Project is very small (a month or two). But in this case maybe it is not required. Use Excel, make simple features list with estimates and that is it.
2. Project is very large and requires high-level plan. You may create very good project plan for Bridge building, Airplane or Space Shuttle construction. PERT is great in this case and project plan reflects phases, not tasks.

But most software projects have specifics:

• Average project duration is several months
• They change a lot and require fine-grained plans (”This feature will be implemented by two people. Joe will create GUI screen within 1d and Matt will program controller within 2d”).
• As a result, it is very hard to keep fine-grained plan in actual state for medium and large projects for WHOLE project life-cycle (the keyword is WHOLE of course).

No matter what tool you are trying to use. Most managers and executives improperly think that Waterfall requires such detailed plan somewhere in the beginning. Waterfall is not the best way for software development, but incorrectly understood and implemented it becomes a real pain in the neck.

You may say that MS Project is just a tool and all I wrote above do not directly depend on MS Project, but on methodology and process. I partially agree, but, once again, MS Project was not designed to support anything but Waterfall or its modifications. It is used for Waterfall and not so many people tried to use it for something else. MS Project + Waterfall = Hammer, but software development in most cases is not a nail, it is a warm wax. You can’t make anything but a flat from wax using a hammer.

As you see, there are plenty more reasons to find a better way. Let’s step back and look at another process.

Iterative Development

If we can’t have fine-grained plan for whole medium/large software project, let’s split it on chunks and create fine-grained plan for each chunk when it will be necessary. This idea is not new and called Iterative Development. Whole project split on iterations (exact iteration duration depends on many factors, but usually it varies from 1 week to 3 months) and iteration planned just before start.

This may sound astonishing for some executives. “So we don’t know project end date??? We don’t have a plan??? Our customers will not deal with our company! What the hell you are talking me?” How can anyone answer these questions? There are two ways to go.

On the first road there is no exact project end date and this road called Customer Satisfaction. You respect them, trust them and produce software they want. Feature by feature, iteration by iteration. Exact date of final release is not required since customer knows that you’ll implement as many features as possible based on defined priorities. It is real trust and it is quite rare.

On another road there are many signs with a big red “Deadline” in the end. While everyone wants to be on Customer Satisfaction road, most of us are on Deadline road in fact. This is a real world and, while we changing it, we have to drive some projects on this road. Can we apply iterative development? Yes, but with some restrictions, since there is less trust between developer and customer. Customer expects a fancy project plan, so we will create one, but low-grained. MS Project is good for low-grained plans. The plan will contain project phases and major milestones and, of course, it will contain end date. If customer asks why there are no tasks, you answer that you don’t like micro-management and fine-grained plan will be developed later. In fact, customer doesn’t care about exact plan, it does care about delivery dates and you’ve already provided them. Then you may break down each release on iterations and follow first road.

Some off topic. The other trick is to leave project/release scope flexible and this is really hard to achieve. It is harder than make a sculpture from warm wax with a hammer. If customer insists that he should get these features till that date without any deviations - you are in trouble. All you may vary in this case are people and process. And often you can’t vary people as well. So the process becomes a single life-saver for you and your team. While it is powerful life-saver, it may not help. Once again, do your best to leave scope or date flexible and focus on Business Value whenever it possible. If not, use iterative development anyway and try to release something useful as early as possible. Maybe customer will appreciate your effort, trust you and you slowly turn to Customer Satisfaction road.

“OK, we may use MS Project for low-grained plans. But what should we use for fine-grained plans?”

I can’t point Best-of-the-Best solution, but some important things that your tool of choice should do:

• Fine-grained plan for iteration should be visible and understandable for all developers without any restrictions
• Person who performs a task should track time spent on the task. PM shouldn’t ask everyone how many times he/she spent on this and that.
• Plan should be easy-to-change (no dependencies, easy re-allocation)
• Iteration status should be visible for all (and preferably tracked automatically)
• Iterative planning should be supported by the tool

In short, the tool should be opposite to MS Project, it should be strong in those areas where MS Project becomes frustrating.

Stop! I Can Plan Iterations in MS Project!

Yes, go ahead and try. You’ll quickly discover how it is useful for iterative development and how it is designed for iterative development.

I’ve planed iterations in MS Project for several projects and it somewhat worked, but this method lacks visibility. Developers don’t see plan and don’t see progress. You should update actual effort manually and assign tasks using Outlook.

How many features in MS Project you will use for Iterative Development? Gantt Chart, Resources and Tasks. That’s it. Tasks will be independent, so critical path is useless and Gantt chart is not very helpful. Leveling… OK, sometimes it helps, but in most cases conflicts may be resolved very easily for one iteration. Reports? Useless in most cases, but you may need something for executives. It appears, that MS Project is too complex and don’t bring enough value to justify its usage in iterative planning.

On the other hand, there are many features that you’d better have for iterative planning:

• Product/Release backlog. The main issue. If you use MS Project, you will have to store backlog in Excel or somewhere else and copy/paste features when required.
• Integrated time tracking. May be resolved programmatically and with help of MS Project Server, but this is an additional effort and MSP Server used not so often.
• Work remaining tracking. This practice is very useful to know actual iteration state. Can be resolved the same way as previous item.
• Iteration velocity concept. If you create an iteration in MS Project, Effort will be Iteration Velocity in fact. But you don’t have clear velocity progress picture

So you really may plan iterations in MS Project, but it will be not so simple and natural. MS Project just will not help you much.