Why is agile so much more successful than waterfall?

Agile's willingness to fail is, ironically, one of the reasons for its greater success.
139 readers like this.
diagram of planning a cloud

Opensource.com

Agile continues to take the world by the storm. The latest report from the Standish Group Chaos Study presents interesting findings: Projects based on agile principles have significantly higher success rates than traditional projects based on the waterfall methodology.

Comparing success and failure of agile vs. waterfall

In this article, I will try to describe some of the factors that contribute to these high-performance metrics exhibited by agile teams.

What are the reasons for higher success rates in agile?

Almost all waterfall projects I've ever been involved with were strongly biased toward risk-aversion. Waterfall projects typically have sizeable goals in mind and get organized around such goals. Meaning, sizeable resources (monetary and human) get assembled and put into the strictly controlled funnel.

A waterfall project usually begins with: "Listen up, we have high hopes for this new initiative; we have hired awesome experts, invested a lot of money to make sure this goes smoothly. Failure is not an option!"

The team then commences work in all earnestness and is eager to show early results to justify such potentially risky investment. And, as is typically the case, the work commences by focusing on what we call "precision work." What we mean by precision work is early commitments to various aspects of the software product that the team is building. The specifications, the infrastructure, the architecture, the design, all those aspects get firmed up and solidified at the very outset of the project. Once that's vetted and signed off, requirements go into the deep freeze, and the team rushes into writing the code. Eventually, the code is complete (usually after months of frantic work), then comes the code freeze, and now testing commences.

And it is at this stage, when we move into testing, that interesting discoveries start popping up. More often than not, it turns out that the early, eager commitments that cemented the team and product infrastructure only managed to paint the entire project into a proverbial corner. Late (and unavoidable) failure comes at an exorbitant price.

The process I've just described is often called "premature optimization."

Agile turns the above approach on its head. It embraces risks, because of the recognition that, try as we might, risks cannot be avoided. Agile focuses on early feedback (make that extremely early feedback). Replacing the arrogant "we know it all" attitude, agile adopts a humble "we learn along the way" approach.

So, instead of firming up and solidifying the requirements upfront, agile refuses to freeze the scope. Quite the reverse, the agile approach remains fully open to any scope creep. In other words, in an agile project, there is no such thing as scope creep.

Scope in agile is managed by responding to feedback. But how to solicit that precious feedback?

Feedback is always difficult to obtain. Everyone is busy, and no one has the time to volunteer feedback. Also, even if someone manages to find the time to offer their feedback, it often is offered in the spirit of "if you don't have something positive to say, at least abstain from saying anything negative." So, most volunteered feedback ends up being in the form of lukewarm praise.

We are, however, more interested in finding out the truth. So, the only reliable way to solicit valuable feedback is to fail. Failure is a great motivator. It makes all involved parties respond with urgency.

In agile, we value frequent and early failure. We want to expose any significant risks as early as possible. We are open to brutally honest criticism.

Armed with such an attitude, agile follows the continuous integration (CI) methodology. As we're building a solution, we must attempt to integrate constituent components with each code transformation. Every step of the way is followed by integration testing.

It's the exact opposite with the waterfall approach. Integration is left for the very last step, once everything else has been prematurely optimized, solidified, and tested. It is then at that last stage (integration) where significant risks get discovered. At which point, it is often too late to successfully handle the damage.

How to integrate often and early

The reasons why waterfall methodology is not as successful as agile seem clear. But the underlying causes are not necessarily down to reckless approaches to managing the software development project. The waterfall approach does not arrogantly dismiss early and frequent integration testing. Everyone would love to be able to detect significant risks as early as possible. The issue is with the inability to integrate services and components that are not ready for testing (yet).

As we progress on a project, we prefer to utilize the divide-and-conquer approach. Instead of doing the development and building sequentially (one thing at a time), we naturally prefer to save time by doing as many things as possible in parallel. So, we split the teams into smaller units that specialize in performing dedicated tasks.

However, as those specialized teams are working, they are aware of or are discovering various dependencies. However, as Michael Nygard says in Architecture without an end state: "The problem with dependencies is that you can't depend on them." So the project starts slowing down as it gets bogged down by various dependencies that are not available for integration testing.

The agile approach removes any blockages and allows autonomous teams to continue development activities without having to wait on anyone. This way, integration testing can commence as soon as teams start iterating/sprinting. Agile, in this way, is a huge step forward for IT organizations.

What to read next

4 myths about agile

After a recent presentation about agile development, Jen Krieger , an agile coach at Red Hat, received that she calls "the most epic conference feedback ever: 'Your talk was…

Tags
User profile image.
Alex has been doing software development since 1990. His current passion is how to bring soft back into software. He firmly believes that our industry has reached the level of sophistication where this lofty goal (i.e. bringing soft back into software) is fully achievable.

6 Comments

The comparison the success rates just by counting the projects and classifying them as agile or waterfall oversimplifies the picture. Actually, marking out the projects that are not Agile (whatever it means) as waterfall completely disregards the many other practices and organizations of iterative and incremental product delivery. The original report also compares the size of the projects, but still does not compare the what actually is success in terms of delivered scope (compared to the original scope), the time to deliver and cost.

In my practice I worked for a company where we delivered incrementally and iteratively for 15 years with a stable pace of 6 weeks release cycles, delivering working software, with permanent collaboration with the business analysts and product management, definitely addressing the change and adapting to it, having a defined process still updating and adapting it with the collected practices and observations. This process was in the spirit of Agile manifesto, just not claiming it was Agile - it was our process and it worked. According to this article it would be classified as waterfall, what it definitely was not.

Thus I would not accept the general statement of the report and even less I would accept the generalization made in this article, based just on part of it. A more valuable anaysis would be to analyze the effects of applying the each of the principles of Agile on the project success as of above.

First of all waterfall is miss understood. For some reason people think that waterfall means planning project ahead for year. Actually I read blueprints and there is no information about duration. In fact if you apply waterfall for 2 weeks, you have new agile methodology

Duration of the project/activity is not the factor that differentiates waterfall from agile. You are right -- you can have agile projects that last for years, and you can have a waterfall project that lasts only half a day.

What differentiates waterfall from agile is two factors:

1. Big upfront plan/design
2. Gated phases

In waterfall, we do not attempt to build anything until we first get a hold of a completed detailed blueprint. We call those blueprints 'requirement documents'. The delivery of a requirement document signals the end of conversation. Those documents get frozen, and only once that phase has reached its completion, do we move into the building phase.

Building phase is also gated; it must reach its completion and only then do we move into the next phase -- testing. And so on.

Agile is the exact opposite. No one in agile is waiting on the big upfront plan/design. Specifying requirements happens in parallel with building, testing, and so on.

You can see that there is a huge insurmountable difference between waterfall and agile.

In reply to by Ren (not verified)

We still use Structured Design and Analysis, Yourdon and Demarco ... We are Big Batch processes and have several separate system interfaces. All have to be loaded at the same time to work and the coding/testing phase is usually more than a month or two for everyone to be in sync.
I suppose WaterFall is best for us.

Not sure I understand your reasoning here. Why is waterfall the best for you? What is waterfall solving for you that agile couldn't solve?

In reply to by Mickey

water fall is always easy

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.