I've been contemplating recently about agile movement as something that has risen from software developers who felt that the challenges of their work are not addressed by the waterfall approaches of the industrial production, and shared my thoughts in the Back to the Future of Agile Development essay. Then, as another layer of this thinking paradigm, I saw that Kanban as a method in agile software development stems from the human need to get rid of deadlines (check out the article Kanban as Multiban?) Now, there's another concept in project management that falls into the same pattern. I'm talking of what is known as "project portfolio management" in the enterprise lingo, and what we call "multi-project prioritization" for smaller agile companies. So, this time I want to look deeper into prioritization, and give one more example of how a promising new trend stems nowhere else as from human nature.
On to project portfolio management. Just as waterfall was copy-pasted from industrial production, and turned into the Rational Unified Process for the sake of software development, in the same fashion, project portfolio management has its roots in the financial investment industry. As always, there's a human need behind it. Someone must have been tired of looking for the ways to manage risks in software development projects, and resorted to what seemed the closest available counterpart — financial investment. But if we look deeper, would there be any difference between financial investment and following through many projects to completion in an organization? For sure, yes. First of all, projects are not shares or stocks, and in investment management one is looking to lower the risks, and compile the investment portfolio in such a way, so as to mitigate them, and optimize the financial gain. Only that and nothing else. It's not that this person is concerned with being strategically involved with a public company whose stocks or shares they're buying.
It's far more complicated when we deal with projects, and especially with software development projects. One key difference is that the projects are meant to be followed through to completion. Of course, a lot depends on the organization. I figure that IT and software development projects could be shuffled as investment stocks — the closest match — in budgeted research fields, like in the military or in the government projects. Most IT companies, though, have nothing to do with picking up or giving up on projects. While in the finance industry the only value indicator is financial gain, there are many more value indicators in software development. Usually, the question is not if the project should be picked up or given up. The questions are:
1. Are we fitting in the budget? Do we need to secure a leeway to complete our projects?
2. Are we lax in terms of time? Do we need to sacrifice some parts of the project, and skip them, in order to ship some workable software in time?
3. How about people? Are they all balanced well in the projects? Have we made sure that the team's collective energy is sent in the right direction?
4. This one is the closest to where we might come at portfolio management. Let's say you work in a large organization and you have many projects, as an IT director, to supervise and to report on their health to someone standing higher. Or, in a smaller product dev company, one might have this multitude not with the projects per se, but with what we call "product features". Ironically, for a smaller organization, this would seem the closest approximation to portfolio management. We have to prioritize and decide, whether we skip this feature, or follow through.
On all of those 4 levels, it's about prioritization. That's what it's all about. Prioritization is the toughest job of all in the world, be it in personal life, or at work. Sacrificing is the most daunting challenge, and it imposes a huge load on a person or a group of people who are supposed to decide and prioritize. By now, the buzz in the industry says that the concept of "project portfolio management" has something missing in it. The tools for multi-project prioritization are not universal, and they don't do the magic instead of this tired human being. Either the tools have to be customized (at big costs), or they miss some instrument that is crucial for this particular organization. In a nutshell, the project portfolio management concept has outlived itself for effective prioritization, just as RUP had outlived itself previously, and was replaced by agile as a methodology in software development. But still, as a product owner, or a project manager you need to have a birds-eye view on all the risks. Still, you want to get this burden off of your shoulders, and finally get a tool do the bulk of the prioritization for you, as this is the hardest job in the world. What happens usually when some methodology is not working out as expected for those human beings? Right. They're on the lookout for new, better — and what's most important — easier ways to prioritize.
Voila: enter Big Data. There's much talk going on about it now. This is the next big thing coming, as it is supposed to make a productivity breakthrough in prioritization and decision-making. If we draw a parallel with the previous occurrences of groundbreaking phenomena (like, the way agile appeared in software development, or how people resorted to Kanban within the agile paradigm, or how they looked to use investment portfolio methods for project management), this prioritization seems to be the hardest job that IT professionals want to make easier for themselves, intrinsically.
Big Data is a trend that can be briefly described as follows: all the huge data about past performance and work is stored, and can be retrieved then to see how past trends can recur in the current trends, thus helping decide and prioritize. Considering the meta-law of things developing in cycles, this might work, to some extent. There is a certain probability that past trends would help one prioritize efficiently in the present. There's some financial software developed that calculates those trends. But, from what I've been able to see, the big fish and the big gain in stocks usually come at random. This all boils down to the intuitive feeling. Something outside data and calculations. This is not to diminish the importance of data analysis. Any data is a huge asset. Even more this is an asset as we live in the age of information, and we need to learn to get the best of those assets. But, just as the term "project portfolio" has the trace of hope instilled in copying this practice from financial investment to software development, in that it would set us free from prioritization problems, there's something to watch out with the Big Data trend. Yes, we always strive to put burdens off of our shoulders, as Homo Sapiens, and that's why we started with sticks as tools, then shovels, and on. Same with the heavy duty prioritization and Big Data. To a certain extent, we can be sure that it would help us move forward, and give more sophisticated tooling for effective prioritization. But ultimately, there's something even beyond Big Data. Some other infotech miracle. All in all, not that we should lose hope in freeing ourselves from prioritization work altogether, but I don't think that even Big Data would become the silver bullet for the IT professionals in prioritizing their project choices. We will have to carry personal responsibility all the same. But this would definitely be a step ahead in the evolution of methods and tools for effective prioritization across many projects and initiatives.