Explaining Agile, Part 1

I promised awhile back to put some content on this blog explaining what the heck Agile is, for those who come to this site but don't work in an Agile environment. I've procrastinated on that effort for too long, partially because I was trying to get my head around just what to blog. Tonight it occurred to me that I didn't need to bite off the whole thing all at once, but could rather break it into parts (rather like Agile uses iterative development, but more on that later).

So the basic stuff first... What's Agile? While there are many excellent books on the topic that explain it better than I ever could, here's a thumbnail sketch of Agile, starting with how it differs from the traditional software development approach, the latter of which is usually referred to (in Agile circles, at least) as "Waterfall."

A Waterfall project is so named because the different stages of it - specification gathering, design, programming, testing, installation - are more or less serialized such that one follows another, like water running down a set of steps. Those of us who've been in the software development business had long assumed that was the only way to deliver features. You get someone to write up the requirements/specifications, and you lock them down as much as possible before starting the design, and you lock that down before starting the coding, and so on. The reason for doing it in this fashion is simple: it's more difficult to hit a moving target than a stationary one. To use a real world example that more people might identify with: imagine trying to build a house when the blueprints are still changing in significant ways after the foundation has been poured and much of the framing is already done. It seems obvious that you wouldn't want that, and so we all worked Waterfall for decades without even considering that there might be a better way to operate.

So what's wrong with Waterfall (because if the answer is "nothing" then why bother learning about Agile in the first place)? Well, it has several things working against it, including (at a sort of "meta level") the fact that virtually all studies of traditional software projects reveal that the majority end in failure, when "failure" is defined as any of: running over budget, coming in late, or delivering less than 100% of the promised features. The actual numbers vary from report to report, but it's often as high as 70 - 80% of projects that fall into that rather embarrassing category. Why do so many software projects fail? There are lots of factors, including (potentially) incompetency among the project managers, analysts, developers or testers on the project, but usually it boils down to the basic human difficulty in predicting the future.

Software projects typically involve "guessing" how long a whole bunch of things will take to do for the first time. When you put it like that, it sounds like pure folly, and yet it happens on a daily basis, across the planet! Most computer programming is, after all, the process of a person (or group of people) sitting down to create functionality that he, she or they have never created before (because, if they have done it before, chances are that they won't need to do it again... the customer will just use the old application or screen or button and no new project is required).

So my theory as to why most Waterfall projects fail - and I didn't make this up: I've lived it and read about it - is its flawed central concept: Before anyone actually does the work, I'll ask the customer to predict exactly what they need or want, then I'll predict exactly which steps are required to deliver all of what the customer just asked for, and exactly how long each step will take to complete. Next I'll produce a project plan with all of that information in it, and we (the development team) will execute against it for __ months, at the end of which we'll magically deliver exactly what I said we would... or fail to do so (the majority of the time). And perhaps we'll cure cancer while we're at it...

If you made it this far, congratulations! I'll pick up from this point in the not too distant future.