I continue exploring deep reasons behind various phenomena in agile software development. It's amazing, how often the universal principle of entropy manifests itself in the relatively short ~ 10-year mainstream history of this movement. It looks like I have another eye-opener.
Entropy in software development
What is entropy, in simple terms? This is the principle of bouncebacks as in a pendulum. If there's too much emphasis on one side, a social or ideological pendulum bounces to the opposite point, as if to compensate this overdoing. We have too many decisions to make and we're tired of that? Let's invent the panacea, called Big Data. We got tired of deadlines in waterfall software development, and even Scrum didn't help, because we got tired of those time-boxed iterations? All hail Kanban now, because it has no deadlines. We are tired of managers treating us as numb tools? We want to break free, so we delve into the tempting freedom of self-organization and no managers at all. I sketched the visual for this pendulum, to make the metaphor clearer.
Now, the universal law of bouncebacks is not a problem per se. It's a given. We can't get away from seasons changing; and they say that if you laugh too much, you will soon cry. Anything in this world has a bounceback, if overdone. What all those bouncebacks have to do with problems of my company, you might ask? That's what I'm getting at. There's a composite of forces that start a trend or a movement. If the critical mass of companies are having similar problems, then the pendulum gains the powerful momentum to the opposite side. That's how it happened with agile. Some companies hopped on this vessel as early adopters, some have not yet accumulated enough energies for that move, some are in the process of swooshing with the pendulum swing. Each and every company has their own unique place in this swing, the unique combination of energies and driving forces. The way I see it, the art of technical leadership has nothing to do with catching the swing and rushing headlong where everyone else is rushing. To follow the "join everyone else" call is the easiest thing to do, and a certain component of luck might help here as well, if a company has caught the early tide flow, and benefited from the swoosh. The hardest thing is to sense where your organization is now, in which particular point of the swing, and how standing there influences your org's productivity and ability to deliver software to your customers.
I've spent enough years in software development, with customers, developers and managers, and I speak from personal experience. It feels good swinging along with the pendulum as it bounces from the left to the right, as on the picture. I felt that the agile movement and self-organization in a team is what I really needed at some point of time. However, later on I started connecting the dots in the bigger picture and saw some signs that self-organization and several other values declared in the agile manifesto are not universal laws. They rather signify the points of bounceback. There's no universal proof that self-organization, or face-to-face communication yield the highest productivity for any company. Keep in mind that agile manifesto signatories are software craftsmen. They've postulated the agile principles to declare that software development is a craft, and they are a noble guild that wants to serve customers. By declaring those principles, they mentioned nothing about constraints, times-to-market, expectations of stakeholders, and financial goals.
Is it a human instinct to deliver projects on time?
Some complexity theory studies compare the laws by which organizations function with those of colonies of ants or flocks of birds (e.g. check this article dated 2009). According to those theories, ants and birds know instinctively how to self-organize to fetch food, or to fly safely to their destination, and the researchers somehow concluded that IT companies must follow the same pattern. I've shared this belief until recently, when a realization struck me. I beg your pardon, but is it our natural human instinct to deliver a release on time? Or to a meet a harsh deadline? Can you imagine the Great Wall of China built by a self-organized team? Of course, no. Why am I asking this? Well, software developers are not ants and birds, and, speaking of instincts, their basic high-profile instinct as software craftsmen is to craft what they do to the point of perfection. If given endless time, they would craft it at their own pace, be it a UX design, or a piece of code, and that's where they indeed are able to self-organize and run a flat organization as e. g. Valve or 37 Signals do. Lucky is a software craftsman, and lucky is a team that can afford the luxury to release at their own pace, with this deep feeling inside that they have crafted this piece to perfection. However, we live in the world that is full of constraints. A pizza-sized team of craftsmen wants to grow big, because they now want to come up with yet another awesome software. This move requires more hands, and coordination with stakeholders, internal and external ones. The noble castle of software craftsmanship is invaded by merchants, who want deadlines met and sales goals hit.
So, when the harsh reality of the market hits the noble craftsmen, they follow inertia and think that self-organization (or flat structure) will let them achieve those very non-instinctive deadlines and sales goals. Looks like that's what happened to the guys from Everpix who did not have practical guts to comprehend that they have to take merchants into account, and kept having their heads in the clouds. If they were smart enough to sense in due time that the pendulum wants them to swing backwards, down-to-earth, to pragmatism and discipline, Everpix might have still been alive delivering some nice service to people. That didn't happen, however.
It's time now to take a pragmatic look at the eulogies they sing to flat organizations and self-organized teams. The opposition of hierarchy and flat goes first in the list of pendulum bouncebacks (see the sketch above). As for the other three pairs, I've covered them elsewhere. The flat company structure is related to the concept of laissez-faire leadership. While this is an excellent approach to leading a company from the standpoint of individual learning and skills acquisition of team members, it does have some faults if what a company needs most of all at this very moment is maximum productivity. Quoting the article that I linked to above: "Researchers have found that this is generally the leadership style that leads to the lowest productivity among group members." Again, this leadership is a bounceback from rigid hierarchy. It does allow people to take decisions and contribute to the activities that are beyond their direct responsibilities, but people have only so much energy, and it strongly depends on individual working styles. It could be that someone who has a lot to contribute to good decisions is not OK with the combined need to make decisions and to produce deliverables. Organizing and decision-making is work, the chore that drains energy from the performance component of such an individual. If a UX designer or a senior developer who is at the same time responsible for the implementation of a feature or a design spends too much time in decision-making activities unrelated to their main work, this is a serious hindrance to productivity. Oh, and don't forget that personal discipline and responsibility is a must for that style, as well as high cross-competence in all aspects of software development. What happens then if team members lack competence, and what they do is learn? The momentum spent on learning is subtracted from production. Here's a visual to help explain the idea quickly:
The background colored shapes show how each of the components would take from the other two, if overdone.
Balance learning, time and productivity?
There is a way, though, to get the best of productivity and keep the learning in place. But it does require smart thinking behind the daily setup of company operations. Back to the pendulum, the tech leader has to keep the company somewhere around the pivot point, with no rush to extremes. Someone has yet to develop the balanced mix of agile and hierarchical management. This mode of pivotal operation implies razor-sharp intensity and focus in anything and everything, the foresight in knowing and defining which issues need whose attention, and when, and setting the boundaries as far as to the point of worshiping individual productive flows, that of decision-makers and that of performers. It takes some hard thinking, and taking hard choices: can we afford hitting 5 meetings with 7 people to discuss this issue? Is it worth it? Does it really matter for productivity that everyone reconciles and has the same viewpoint on any given problem? How the skills and competence of people can be applied optimally, to keep them working and have them learn new things? As I read stories about companies that seem to be successful with flat structure, I noted their discipline and foresight with work-related interactions. So, if tech leaders want to copy-paste the laissez-faire style because everyone says that it's cool, they need to be realistic in the assessment of where their company is in the pendulum, and which priorities overrule at this very moment. Hands-off leadership, which is the same as self-organization, is a luxury that not everyone can afford.
I hope to share more thoughts on the modus operandi of tech leadership in the future.