Rebecca Parsons gave a wonderful talk on why agile is not the easy way out, but is the only way. Having worked on multiple projects agile/lean/scrum, I can only agree with her. Most of the organizations adopting agile feel the pain of traditional methodologies and think agile is the silver bullet which can solve all their problems. Other organizations say they are following agile, without even knowing that there exists an agile manifesto, simply because they don't want themselves to be excluded in the race of adopting agile. Some organizations openly admit that they are still following traditional methodologies, because they are not comfortable with agile. Agile is usually associated with a lack of disciplined development efforts and loss of control. Nothing wrong with a traditional waterfall based SDLC, but time and again we have seen that a lot of software projects either go beyond their allocated budgets, time-lines or both. Though I strongly suspect the methodology employed in the recent Standish group, which reported more success with agile projects, I believe that if a more methodical survey is carried out, we would still arrive at the same conclusion. VersionOne recently published some statistics on agile adoption. Though the percentage of agile projects which did not fail was about 16%, the top category of failure was "do not know" - 13%. These are some of my thoughts on why agile adoptions and transformation fail
Failure to understand agile principles, but just implement the practices
Agile principles are like roots of the tree, they hold the tree together. The trunk and branches represent different processes (XP, Scrum,Kanban, etc) and practices (TDD,CI, Pair programming,etc) respectively. Most of the times, we neglect what we don't understand (the agile principles) and instead focus on what is easy (the practices). And the results are disastrous. We can see that from the survey that at least 78% of the respondents practiced at least one of the agile practices (daily stand-up being the most widely adopted practice at 78%), but 84% of the agile projects failed for one reason or the other. What would you think of a doctor who treated you for the symptoms, instead of understanding the underlying cause and treating the cause ? I can only attribute this kind of failure statistics to the fact that people adopt the practices without understanding why they need a certain practice.
Lack of clear understanding on what is agile
Agile, to me, is a journey. Not a destination. It is a journey striving towards continuous improvement. And it goes beyond the development team. Unfortunately most of the adoptions stop with the IT department. And to fully realize the benefits of agile development, other departments should also be educated about agile principles and why it helps development teams. These include sales/marketing, Human Resources and facilities as well. When sales/marketing is not fully involved with the transition, they promise unrealistic features and delivery dates to their customers. They associate agility with flexibility and speed (and this is true) and that the dev team should match their expectations (which, realistically might be beyond the teams' capability). Regardless of how good the team is, such unrealistic expectations set the foundation for failed projects. No methodology can fix these kind of issues.
Lack of education
One of the organizations I worked with was ready to transition their very well experienced Project Manager (who also had a very good understanding of the domain) to the role of a product owner (in fact, I suggested it) but refused to send him to a Product Owner Training (citing cost reasons). It is easy to cut down on things which you can quantitatively measure (like cost), but hard to derive ROI on things which are subjective/qualitative, like improved product management. Granted, if that person had seen how a good product owner works, I would not have had much concerns, but that was not the case. The product owners were themselves new to scrum and were half way around the globe. Now, I don't mean to say that a simple two day training would make a good product owner/scrum master out of you, but that is just a starting point. If you have a coach who can guide you and your team, that is equally good. Regardless, learning is a continuous process and you learn from conferences, books, blogs, subscribing to distribution lists (Yahoo groups, LinkedIn) and from peers.
Adopting agile and expecting that it is going to solve your problems
Agile is about transparency and visibility. Agile gives visibility to your problems. It will not solve problems for you. If you hire bad programmers, it is going to make it more evident, but it is up to you to hire better programmers. If the business stakeholder is sitting half way across the globe and is hard to collaborate with him/her (and it is true for even someone sitting in the next
cubicle desk, but does not want to collaborate), agile is going to make this problem more visible, but does not solve this problem. If you are building the wrong thing, but skipping sprint reviews, "your customized" agile adoption will not help you. If you have customized scrum to accommodate change requests during sprints, say goodbye to good processes and say hello to the devil. If you pile up technical debt, you will eventually pay a price for it. You are better off paying it early than paying it later.
We are resistant to change
I once read a story about an old dog sitting on a nail and whining all night. A traveller asks a wise man why the dog does not move away to which the wise man replied "It does not pain enough for the dog to move away". A lot of us do not want to change the way we think, simply because we are used to it, it feels comfortable and changing any of that will make us uncomfortable. We justify it saying that "a known devil is better than an unknown angel". Some of us are like ostritch which burries its head in the sand. We don't want to acknowledge a problem because if we do acknowledge it, we will be forced to take action. But change is the only constant. And We resist change because we lack the discipline to follow new practices.