Waterfall Mindset Limitations
IBM Labs ‘Cynefin’ research defined complicated and complex as follows.
Complicated are known unknowns… events that have happened before and which are foreseeable. The causes and effects are clearly understood.. Any prudent planning process should assume they will occur and make provision for same. Waterfall is good at this.
Unknown unknowns are not foreseeable. Cause and effect can only be deduced in retrospect, and there are no right answers. This is complex. In the vernacular ‘sh%^&t happens. Waterfall plans cannot accommodate these events beyond the catch all ‘20% risk provision’.
Building object oriented software is a very difficult thing to do well. Deploying software in containers where microservices engage one another via API’s in cloud infrastructures is not an exact science just yet. New architectural terrain is complex and waterfall mindset is bad at this.
Unknown unknowns are outages that could not be foreseen or unending merge conflicts and churn on code within a pipeline. These occur for a myriad of reasons such as poor design decisions, repurposing software mid build, poor definition of roles and responsibilities, high attrition rates, inexperienced management. And that is just the start of a very long list, but the bottom line is that you won’t know any of this until after the event.
This blog focuses on ‘waterfall mindset’ as a set of behaviours that impedes our ability to succeed in managing complex deliverables. Let’s start with what waterfall mindset is and then explore some of its limitations.
Waterfall is a command and control system. It says that teams can plan up front when they will deliver working software and what features are included. To do this requires significant effort to foresee what could go wrong, and experience of having done many similar projects. There is rightly a focus on risk identification and an assumption that a majority of risks can be identified if not upfront, than at least in flight. Waterfall mindset says ‘IT is in control’.
But we know from Cynefin that unknowns can occur and a waterfall process can only accommodate this after the event when cause and effect have been deduced. Waterfall says we recalculate a later delivery date and an increased budget for the business.
As more unknowns are revealed the cycle of recalculation recurs. Credibility is lost, the team is demoralised and the business accepts less than it wanted to staunch the haemorrhaging or attempt to live within an established budget.
Waterfall mindset remains intact however because the process worked. We couldn’t foresee unknowns and we accommodated them in the prescribed fashion and ‘retained control’ of the project throughout its course. Yet these projects are deemed failures. Frequently IT will bear the brunt of it and since business funds IT their perception quickly becomes reality.
But what if we told the business the truth – we don’t know what we don’t know. Business people are used to making decisions and taking risks with incomplete data sets. They frequently have to execute to some extent before they can gather enough data to make a fully informed decision. They have read the books about Black Swan events.
Why do smart IT Professionals feel obliged to convince the business they are in control especially if historical evidence is to the contrary? The most recent Standish Group’s Chaos Report reveals only 29% of IT project implementations are successful, and 19 percent are considered utter failures.
What if we budget to deliver enough working software based on prioritised value to the business? When we hit an unknown we let the business know this has occurred. And because we have warned them upfront that this might occur we have not undermined ourselves.
In fact we are more in control because we have not committed beyond our first ship date for our first features in agreement with the business’s priorities.
When we now call out a risk that the feature will be delayed, the business can determine whether to continue with the development or not. If the delivery proceeds in this fashion we start to develop a better understanding of that particular system and we learn more about the velocity of the delivery team thereby increasing short term predictability.
We are not saying we are better equipped to foresee unknowns. We are saying that by being more open and collaborative with the business we increase the probability that we deliver more value sooner while increasing the probability that the business retain confidence in the team.
To achieve this we have to unlearn the waterfall mindset and accept we are not in control of complex projects. We need to work more closely with the business to prioritise delivery around customer value or cost savings. We need to help them understand risk on software projects. It’s important that we don’t try to explain technicalities to the business but some of the language used herein could be useful.