Managing or prioritizing backlog, re-prioritizing work in progress — these activities make up the most of product owner’s work. That’s not a breaking news, obviously. Product Owners have lots of things to consider as they:
1. keep some work in the backlog (as unsorted as a heap of wooden logs)
2. sign off some work to go into production
3. re-prioritize work that is already in progress.
At times, the boundaries between a pure production flow (Work In Progress, as in Kanban) and backlog are quite blurred. Some items might stall, as it turns out that they need to be re-prioritized and/or returned back to the “heap of logs”. You might wonder, as a product owner, or a project manager, when is it OK to go with the 2D map which plots work against In Progress states (like in the classical Kanban board), or when you need a 3D concept map to balance your decisions against multiple priorities? Are there any workarounds you can use for a 2D map to make it work as 3D, when there are more things to consider than merely putting work through the production flow?
Some of you might be familiar with the concept of story maps vs. process maps, but I’m using a bit different perspective. A part of it is covered by the story map concept of “walking skeleton”, but what I’m trying to highlight is the speed and convenience with the 1-2-3 product owner actions.
Here’s a 2D Kanban board, where Planned is some prioritized, “ready to go to In Progress” work:
You might need to introduce yet another buffer for the sorted out Planned state. That’s when you most probably add the Open state:
Backlog -> Open -> Planned -> Work In Progress -> Done
The Open and Planned states are not the sequential production states. The Planned state is a product of of applying some other 3d dimension. Some set of business or any other priorities. As a product owner, you need to sift the work through many deciding criteria, bouncing it between Open and Planned as often as needed. That’s what we call “grooming the backlog“.
If you operate within this 2D environment, on a physical or electronic Kanban board, you might use tags, or colors for the cards, or add more non-production states which are essentially priorities (as Open – Planned states).
Now take a look at this visual. It’s a little bit different from what is considered a story map concept, as I’m using another set of coordinates, where X-axis is work, Y-axis is progress, the X-Y plane is any work in progress AND Z-axis is any other 3d dimension, or lens, through which you filter the backlog or work in progress. The red lines in the Z-plane against the X-Y plane represent those various influences, or filtering criteria.
If you’re using some tool to switch and rotate the criteria on the Z-axis fast, you’re much better off grooming your backlog, prioritizing work, and tracking work in progress. I’ve mentioned some workarounds, such as using tags, introducing the states that are not actually “production” but rather “prioritizing” states, colors for cards. Another traditional technique is prioritizing work in lists, not on a board. But all those practices are quite limiting. They don’t provide enough versatility.
Z-axis are essentially swimlanes, if you look closer, and flexibility with swimlanes is a huge thing. If 2D workarounds are limiting, or your current backlog management tools do not give you enough freedom with swimlanes, we have an unbelievably flexible 3D backlog management experience in store for you, coming soon in a newer version of our tool. It will work exactly as on the visual above: rapidly switchable Z-axis influences (swimlanes) that highlight certain backlog items and work in progress.
Today, as a part of the DataViz 101 series (check out the previous installments: Visualization: Understated or Overrated? and Data Visualization 101: Basic Guidance), I will write about one tiny graph that can be used as a report in agile project management, be it Kanban, or Scrum, or any other custom mix of those.
What is a sparkline graph?
In the non-project management world, sparkline graphs work as quick time-span mini-reports featuring the dynamics of certain states or activities. The sparkline graphs usually look like tiny ragged lines. Here’s the definition, the direct quote:
Sparklines are data-intense, design-simple, word-sized graphics… Sparklines mean that graphics are no longer cartoonish special occasions with captions and boxes, but rather sparkline graphic can be everywhere a word or number can be: embedded in a sentence, table, headline, map, spreadsheet, graphic.
Edward Tufte, Beautiful Evidence
Why sparkline graphs are good?
In a few words, sparklines are used if there’s no room, nor time, nor space for visual bells and whistles. What would be the best graph for a broker to track the stock dynamics with one brisk look? Or, for anyone who tracks stocks in newspapers? Exactly, a sparkline graph. This one pretty much explains itself:
*the image is taken from the book “Beautiful Evidence” by E. Tufte
Next, take clinical tests. A patient is recovering from some bad health condition, and doctors need to assess the recovery dynamics by taking just one look at some data. That’s where sparklines can help, too. Anything that goes beyond the grey strip, at the bottom or at the top, is alarming, not within the norm.
*the image is taken from the book “Beautiful Evidence” by E. Tufte
What else is good about the sparklines, apart from their extraordinary time- and space-saving qualities? They are very effective in tackling the phenomenon that Edward Tufte calls “recency bias“.
Compare these 2 reports:
*the images are taken from the book “Beautiful Evidence” by E. Tufte
The upper report shows the yearly dynamics, as a table. The sparkline part of the lower report covers more than 2500 numbers! Thus, it reduces this very recency bias as the year-long daily history (from the last year) makes the analyst pay attention not only to the yearly fluctuations, but to more subtle daily movements in the context of the whole year. If an analyst were to look at the daily dynamics in a table format, she would have missed it. It would be possible to track the dynamics by week, or by day, but it would just take too much space to have such a report as a table, for 2500 numbers, and for every day of the year, hence “the recency bias” would come in charge with a table report for fewer days. They are not able to cover such a huge 365-day span with the pluses (“+”) and minuses (“-”). That’s where the sparkline really helps.
Sparklines for Agile Project Management
So much for the intro to sparklines, let me now show how they can be used for agile project management. Take a look:
On the left you can see the list of teams, along with the avatars of people from those teams. Here’s the logic behind the sparklines for user stories and bugs:
-Any sparkline covers a time span of 16 weeks. Why 16 weeks? It’s convenient in terms of iterations and releases, as in agile project management iterations usually take 2 weeks. So, this stands for ~ 8 most recent iterations. If you count each tiny sub-bar, this would total to 16 (in the sparkline for Designers and Philadelphia Flyers teams. The Support team has less than 16 weeks history, most likely). The 16 weeks time-span seems to be working well for iterationless development, too, as in Kanban.
-The zero line is the vague blue line in the middle (or, the red one, for bugs).
- The actual numbers shown on the top and on the bottom represent the max. number of user stories done or added, respectively, at any given week out of those 16. If you see the max. number, for a week that goes shortly after the 4th week, then you get an idea of how many user stories have been done in the previous or in the following weeks. There’s no need to show the numbers for each week, as it would make this tiny graph too clogged. The same logic stands true for bugs.
-The numbers on the very right, in the sparkline, relate to the current week.
-The “total” numbers show how many user stories were done in those 16 weeks (on the top) and how many were added (at the bottom). Sames for bugs: the total fixed number is on the top, and the total added — at the bottom.
If you’re a stakeholder, or a Scrum Master, or VP of Development, or anyone who wants to keep an eye on how those several teams are doing, this report is indispensable. With the least time spent, you get the maximum info. And – yes – there’s no recency bias. You are able to embrace the whole span of 16 weeks in that little space.
It’s not only about the counts of bugs and user stories. For example, if you see that your sparkline for user stories is keeping low, close to zero, and your bug numbers are pulsating with action, this means that this team is currently working to make the release stable (most probably).
Finally, this logic which has taken me quite a bit of space to explain here, is condensed to a mouse-over text tooltip.
“Where can I take a look at this sparkline in action?” you may ask. This is something we will release in a newer version of our tool. I tried to explain here why we decided to introduce sparklines, and why we consider them a useful metrics for agile project management.
Data Visualization 101: Basic Guidance
My 12 Visualization Books
There are two extreme views on requirements documentation (most use user stories to describe requirements these days):
- User story is just an invitation to communication. It should be very short.
- We should document requirements fully, with all known details.
Most likely, you accept option 1 and reject option 2. If you ask me, what option I prefer, I will answer “both”. Hold on your tomatoes and read on.
Let’s say, you start a project and have a backlog with 100 user stories. This is the stage where you should keep user stories descriptions short. Don’t waste time, many of these stories will never make it into production, they will die.
You pick a user story and ready to start development. At this moment you have a solution and this solution should be documented. Short description is not enough, since some details may be lost and implemented incorrectly, this will lead to re-work and time waste. At this stage you shouldn’t worry that detailed description of a user story is a waste, this story is going to development phase anyway.
Simply speaking, each user story has a life-cycle: Birth → Solution → Implementation → Release. Let’s review it:
Someone has an idea and user story is added into a backlog. Just a short description is fine in this case.
There are various methods to create a solution for the problem that user story describes. Maybe you have a special UX phase and special UX teams that brainstorm possible solutions. Maybe you discuss and invent solution in development team just before implementation. Maybe Product Owner creates a solution alone (worst case). Still somehow you have a solution.
Then you should document the solution with quite many details to make sure that solution will not be misinterpreted. Here you will argue that live communication is better than specification. I agree and will follow on below, but if you have a gap between Solution and Implementation phases (which is often the case), you have a good chance to forget many things about the solution. You will end up with sub-cycle: Forget → Remember → Lost Some Details → Re-Work. This sub-cycle generates nothing but waste.
Implementation may take several hours, or may take several weeks (happens). If a solution was invented outside a development team, it’s extremely important to have a kick-start meeting, discuss the solution again and find all weaknesses in it. It may happen that solution is unideal and can be improved. Kick-start meeting brings everyone on the same page and every team member understands how it should work.
Right after the kick-start meeting user story should have as much details as possible, including positive and negative flows, texts and labels, final design, etc. Memory is a good thing, but it fails quite often. Don’t rely on your own memory and on memory of other people, put everything you know into text and images.
When user story is done, there is not much value in documentation. If there are some minor discrepancies — that is not a problem. Sure, it can be used as a reference for technical writer, for example, so, depending on your context, you may want to have it corrected.
The main idea is that importance of documentation is low initially, highest in the implementation phase and falls down again after the release. Full documentation just in time is a good practice.
You may worry that it sounds heavy, but in real life a user story can pass all these phases in a single day. For such quick stories full documentation looks like an overhead, but usually it’s very short as well and may take 10 minutes to write.
I personally like when user story is fully documented (at the right time!) with all important cases, has final design attached, has all labels nailed and provides a good vision about the solution.
With many user stories you have rolling waves. Like that:
Now fish out your tomatoes and throw as many as you want.
There’s more and more buzz around estimates and #noestimates in software development. People like to write bold statements and go extreme about things in blogs. Usually, personal dialogues are much more balanced. Some hate estimates and believe it’s a useless activity. Some defend it with arguments of arguable truth.
I want to dig into intrinsic estimates complications, what people mean by “estimate” and what future directions we may attack → Read the article.
We are excited to announce a new major spring 2013 release!
TargetProcess 2.24.0 includes quite a few small and big improvements. Check what we’ve done to help you work better, faster, more comfortably.
10x faster search
We’re close to less than 1 sec search time, but this new search is not only fast. The search results look nicer now, and you can open them both in new tabs and as a pop-up.
Read more about the new fast search in our Help portal.
Relations and Relations Network Diagram
Quite often one piece of work is “informally” tied to another piece of work. For example, the core team is working on an API, and the other teams are dependent on the core team’s progress. Keeping such informal dependencies in your head can be tiresome…
Starting from 2.24 you can track such informal dependencies right in TargetProcess as Relations!
Relations Network Diagram represents a network of related entities:
Relations can be created for any entity (i.e., user story, task, bug, etc.), and they can be of great help when planning or prioritizing. Be sure to check the article about Relations in TargetProcess Help Portal. It has more screens, and explains how Relations are helpful in planning and progress tracking.
New Visual Reports:Process Control Chart and Lead and Cycle Time Distribution Report
The Process Control Chart shows cycle time distribution for completed entities (user stories, features, bugs, tasks, requests) within a certain time frame. Check the screen below. If you see user stories between the warning limit and the control limit lines, this is suspicious. If a user story goes beyond the control limit line, this is really a bad thing, and you should investigate why it took so many days to complete. These limits are calculated statistically; they depend on the overall distribution of stories in the chart.
You can find a comprehensive description of the Process Control Chart here. It includes more screenshots and some examples on how the chart can be used.
Note the new powerful filters. You’ll see more of them later. They can now be used to filter out the entities in the Relations Network Diagram and in the new visual reports.
While the Process Control Chart quickly spots the entities that took too long to get done, the Lead and Cycle Time Distribution report helps you make realistic forecasts. You can filter out any entities, just as in the Process Control Chart:
Read more about Lead and Cycle Time Distribution Chart.
There’s a bunch of nice improvements to the views: quick convert (all the properties are kept intact for the converted entities), assign 2+ people, switch project quickly in the Info box for an assignment, auto-convert URLs to clickable links.
Besides, starting from TargetProcess 2.24 you can add up to 60 custom fields to any entity.
New History REST API
Check out the new History API in TargetProcess REST API.
The new API will track state changes for Bugs, Feature, Impediments, Requests, Tasks and User Stories.
More info on the new History API in TargetProcess Help Portal
Our company is almost 6 years old now. It was founded on agile principles and grew up on them. We’ve used Extreme Programming from day 1, immixed some Scrum ideas later on and switched to Kanban in the end. Here below I’ve tried to review our process changes for the last 4 years.
Almost 2 years ago I published the Flow. Discover Problems and Waste in Kanban post. The idea was quite simple: visualize the flow of a single user story or bug, and track their life cycle to Done:
You can spot such problems like delays and re-work very fast this way:
Now we’ve brought this idea to life. The Flow chart for every user story, bug and feature will be available shortly in TargetProcess v2.22.9.
The chart gives answers to a whole lot of hands-on questions:
- For how many days has this User Story been in this particular state?
- Were there any delays?
- Was there any re-work?
- Who was responsible for the User Story?
- When were Bugs and Tasks added and closed?
- How much time was spent each day?
- Were there any impediments?
Let’s look at some examples. The user story on the chart below has been in Open state for 25 days (it means, in the Backlog). Then it jumped right into In Progress state. Two developers (Alla and Vladimir) started working on it (so it was pair programming). They’d been working for 3 days and then the story was moved into Re-open state. This is quite surprising, most likely they had to switch to something else (no good). Then they got back and spent 15 days working on this user story. That’s way too long. Most likely there were switches as well, so this should be investigated.
Starting from Oct-18 the progress was very good: development went smooth, tester found several bugs and they were fixed in 2-3 days. Finally, the user story was released to production with no more delays.
You immediately get a high-level overview: delays and up/down state transitions. It is a clear sign of some problems, the systematic ones or not known so far, but we already have some background info to start an investigation).
Let’s check another example. It looks like the user story on the chart below was taken to development right as it was added. That’s true in fact, since it was a customer request to which we reacted immediately. It was implemented in a single day, and there was a small delay before tester took it to the testing phase. We found quite many bugs and fixed them in 2 days, everything is fine. But then the completed user story was hanging in Release Branch state for 11 days, and that’s no good.
We’re planning to extend this Flow Chart and put more information there (comments, attachments, etc.) The goal is to provide the complete production timeline uncovering hidden malignant patterns and problems. You should be able to get a high-level overview in an instant and dig into as many details as possible.
Here is the snapshot of our real TargetProcess production Kanban Board taken today (November 18). You can see what we are working on right now, what is in progress, what is in testing and what will be released soon.
1.5 years ago I wrote a post that described our development process
. It is interesting to see what’s changed.
- Product development (quite large web application)
Two development teams: Core and Integration Several Mini-teams.
- Quite mature product:
- Technology: C#,
- Development team has all the power to try new practices, keep working practices and remove bad practices. There’s
a little no push from Scrum Master side.
- We form mini-teams for every Epic. A mini-team includes 2-3 developers, tester and designer (half-time). It is fully responsible for Epic implementation. Usually mini-teams live for several months.
- We do not use iterations.
We do use releases, but often it is just a set of all the features implemented last month. We do not plan releases.
- We do not have iteration planning meeting, but discuss every user story JIT before implementation on a meeting with PO + Testers + Developers.
- We do not estimate stories and bugs, just a very brief estimate like (“oh, this will be a large story, let’s split it”).
- We split stories, but sometimes not as aggressively as required. It is still our weak side. Sometimes user stories are too big.
- We release new builds weekly (sometimes 2 builds per week). The goal is to release every day, but we do not have a good automated functional tests coverage so far. Automated tests have been improved significantly over the last 18 months. We’ve migrated to Jenkins recently and parallelize tests aggressively. Now all the tests run ~40 minutes and test runs on every push.
- Smoke testing of the build still takes 3 hours (
We have a limit of 3 user stories or bugs in progress.
We have retrospective meetings every 2 weeks. We run JIT meetings instead of retrospectives. We have an Issue Board, limited to 3 items. When the limit is hit, we run a meeting.
- We sometimes have technical user stories (for example, remove jQuery from application), but that is rare.
- We use Kanban Board in TargetProcess heavily and have a large screen in development room.
- We have a flat hierarchy in the company. There are only two layers and no boundaries between the layers. No formalism at all. Also we do not have positions like Senior Developers, Junior Developers, etc. All people are equal (however, there are informal leaders).
We track spent/remaining time on tasks and stories.We don’t track time anymore
- We develop each user story or bug in a separate branch. Master is always ready for release (after smoke testing). We use Git.
- Pair programming for
all user stories and complex bug fixes. Sometimes people work alone on smaller/simpler problems. Pair programming is used for complex tasks only.
Developers switch pairs every day, even if user story is not completed yet. The goal is to a have fresh look, knowledge spreading and real collective code ownership. We eliminated pair swapping, it appeared to be distracting and annoying for people.
- Before implementing a story, pairs discuss possible solutions. It is quite formal.
- TDD. All the new code is covered by unit tests.
We use Cruise Control and have quite many different setups. Smoke builds, release builds, etc. Functional tests run in parallel and it takes 1 hour to run them all. We use Jenkins now and it takes about 40 minutes to have a new build with all the tests completed.
- Automated functional tests are based on Selenium. We’ve created a framework in C# especially for our application to make tests easier. Tests coverage is not good so far, but we are working on it (there are 2 automation testers in our team).
- We still want to add performance tests into automated builds. We have performance tests running already, but not as a part of new build so far.
- We want to split user stories better. Much better.
- We are incorporating User Experience into development process. This is our main goal for the next year. This includes running a UX community for customers, interactive prototypes for all user stories, usability testing. We are making steady progress here and we have defined our UX process. However, UX injection takes more time than expected. As usual.
1. You don’t waste time on estimation
Estimation takes time. Even if you do planning poker and use story points, it still takes time. What do you do to improve estimation accuracy? You gather some data, analyze the data and discuss the results. You are spending time on all that. But are you sure that estimates really provide any value? Most likely not. It is a waste. You’d better spend this time doing something important for the product.
2. You shouldn’t explain to higher managers why it took soooo loooooooong
If you don’t have estimates, you can speak without fear and explain things clearly. You can enumerate problems and explain how they were resolved. You can show that you work really hard and take all the necessary actions to release this story as soon as possible with great quality.
If you have estimates, be ready to hear something like “you screwed up, maaan! You had to deliver this feature till the end of the month! What the f%$k is going on?” as an argument. The team put themselves on a weak side immediately and have to care about deadlines more, not about quality or a better solution. Is it something you really need?
3. You don’t give promises that are hard to keep
You are not relying on people’s optimism (which is built in). People are optimists (almost all of them) and inevitably give optimistic estimates. You can use complex formulas or very simple rules to have better estimates, but is it really worth it? You can spend many hours defining correct formula for exactly this development team. People will not trust you, since you (rightfully) don’t believe in their estimates. Numbers will look “made up” and still you will have incorrect estimates in the end.
4. You don’t put additional pressure on development team
In some teams Estimate is equal to Deadline. That’s bad. What do you do to meet the estimate? Compromise quality? Tolerate average architectural solution? Abandon polishing? All that leads to poor solutions that end users will not like. It is better (almost always) to spend more time, than planned, and release something really usable and useful.
5. You focus on really important things
Imagine, you have a planning poker session and estimate all stories inside an epic. You sum up all the stories and define it as a 1000 pts epic. You just stick a “fear” label on the epic. People do fear large problems. A 1000pt problem is huge and psychologically it is harder to decide “let’s start it right now”. Product Owner will have a temptation to implement 100 smaller user stories instead of this huge epic. However, it may be a bad decision. This epic may be really important, the morst important thing for the product.
If you don’t estimate it and are not completely sure about its size, you have a better chance to actually start doing it.