Agile Development

Why is Software Development so Hard?

We have all heard horror stories of software development projects that have gone horribly wrong.  Budget and schedule overruns, software that is unstable and unreliable, solutions that don’t meet the users’ needs and expectations.

As software development professionals, we would like to pretend that these types of projects are the exception, that software development has received a reputation for failure that is undeserved.  Unfortunately, that does not seem to be the case.

In just one of many industry studies, Standish Group’s recently released report, CHAOS Summary 2009, has some all-too-grim statistics:

“This year’s results show a marked decrease in project success rates, with 32% of all projects succeeding which are delivered on time, on budget, with required features and functions” says Jim Johnson, chairman of The Standish Group, “44% were challenged which are late, over budget, and/or with less than the required features and functions and 24% failed which are cancelled prior to completion or delivered and never used.”

“These numbers represent a downtick in the success rates from the previous study, as well as a significant increase in the number of failures”, says Jim Crear, Standish Group CIO, “They are low point in the last five study periods. This year’s results represent the highest failure rate in over a decade”

So the problem is not just one of perception: it is very real.  Why is that?

It can be hard to identify the causes for these statistics, and any assumption that there exists a single underlying cause is probably naïve. 

Some of the reasons include the reality that software development is one of the most complex activities that man engages in, the erroneous comparison of software development to engineering and construction and trying to manage the process as such, and the lack of communication between the parties involved.

In some ways, the history of project failures (both real and folklore) tends to become a self-fulfilling prophecy, with the business demanding unrealistic deadlines in a misguided attempt to control the process and development teams unwilling to expose the true status of a project’s progress for fear of over-reaction.

And yet, some organisations (including Internet Business Systems Australia) are consistently able to bring in software development projects successfully – on time, on budget and meeting the actual needs of the business.

So the question then becomes: if someone can do it, what can we learn and how can we apply it to achieve the same outcomes?

That is exactly what agile development seeks to do. 

While in no way holding itself out as a silver bullet, agile development methodologies focus on doing more of what has been shown to contribute to project success while avoiding what has been shown to contribute to project failure.  The specific practices vary significantly across the different methodologies because each is tailored for a different type of project, but a set of common principles underlies them all.

NEXT: What is Agile Development?