Possible new scrum extension

Scrum is evolving and changing. Product Backlog Items evolved from prioritized to ordered. Product Backlog grooming is becoming a Generally Accepted Scrum Practice (GASP). Charles Bradley has also blogged about some new direction scrum is taking. Burndown is no longer an artifact, but increment is. Ron Jeffries frequently writes (in scrum development yahoo groups) about small, similar sized user stories and working in smaller chunks instead of large user stories.

David Starr has already proposed an extension (Feature focused daily scrum ). Along these lines, I want to propose a new extension "JIT Planning"

Proposed Name
Just In Time (JIT) Planning

Background

The PO already has an ordered list of backlog items.The team starts the spring with a planning meeting where they together to discuss the features that will be taken up during the current sprint.  The planning meeting consists of two parts, the "what" part where the PO and the development team understands the scope of the backlog items which need to be considered for the current sprint (mostly in the form of user stories), and the acceptance criteria for each of those backlog item. The Product Owner should be present during this part of the meeting. The second part consists of the "how" part where the team decides how to implement the the stories and the tasks associated with it. Usually, the sprint backlog is a set of tasks associated with those product backlog considered for that sprint. Though the product owner's presence is not usually required for the second part of the planning meeting, he should be able to help the development team if any clarification is required. For teams where the sprint duration is 30 days, the spring planning meeting can take a whole day (8 hours, split as 4 hours for the first part and 4 hours for the second part). For teams having a shorter sprint duration (say 2 weeks), the sprint planning meeting is proportionately short (4 hours for a 2 week sprint).

Proposed Change

The scrum team maintains the cadence by having their regular "what" part of the sprint planning meeting. These backlog items conveys the theme of the current sprint.

The "how" part  is broken down into a series of "JIT planning" meeting for each PBI.  The development team initially addressed only for the top-most backlog item. It then works on turning this PBI into a completed feature. Once it is complete, the team picks up the next backlog item, discusses different tasks associated with it and discusses how to turn that to a shippable increment. This is repeated for all the backlog items taken up for that sprint.

Assumptions:
1. The team is co-located or has a very high bandwidth communication channel so that the team can swarm over/plan an impromptu "JIT planning" meeting once a feature is implemented and tested.

2. The backlog items are well groomed, estimated (story points) and/or diced to small stories which can be implemented in a day or two.

Advantages

1. One Piece Flow helps the team deliver as fast as possible: The development team explicitly focus on one PBI at a time without switching context between backlog items. This should help team complete the feature faster. The impediments for completing that particular feature become much more evident (since the team cannot work on completing the feature) and those responsible for removing the impediments (the scrum master, product owner and management) are much more proactive in identifying and removing any impediments. Also, no one, including the team members can gloss over  the impediment by saying something along the lines of  "I have this impediment for this top priority feature, but that's OK because I am focusing on implementing some other product backlog item".  Also, the scrum master (and who ever is responsible for removing the impediment) becomes more effective as he is focused on removing all impediments related to a feature than just shot-gunning on impediments of multiple backlog items. With a good backlog with well defined acceptance criteria, backlog items should be implementable in 1 or 2 days. With stories that small, no explicit tracking is necessary ( Conversely, if the story is large, which may take 3-4 days, it might be a better idea to track the remaining efforts for the feature using a burn-down. Also see #6)

Johanna Rothman modified the daily stand-up question to "What feature did we complete yesterday ? What feature are we going to work on today?  What are the impediments to implement this feature?" JIT planning and one piece flow (execution) sets a boundary for the team (sprint backlog being implemented), and helps answer the modified daily stand-up questions.

2. Encourages team becomes more cross-functional: Since the focus is on completing the feature, it challenges the team members to get out of their comfort zone and learn new skills (if required, or hone their existing skills) to complete the feature at the earliest. A tester can no longer say "I am done writing the automated test case for this feature, but since the DB work is not done, I cannot fully test it" . Instead he focuses on contributing what ever he can to complete the database related task.

3. JIT planning eliminates waste:  The development team forecasts the product backlog items which it can deliver during a sprint. As the team gains more experience (with scrum, with the product, with technology and knowing each other), their estimates get better. But they are still estimates. By only targeting ( tasking out) one PBI at a time (and completing it), it avoids planning for features which may not be completed during the current sprint. Why plan for something which you might never need?  Remember- one of the agile principles is " Simplicity -- the art of maximizing the amount of work not done -- is essential"

4. Plan at the Last Responsible Moment (decide as late as possible): The team continuously learns about the product as well as about the technology used to build the product. Though the tasks associated with the sprint backlog can change, with JIT planning, we plan for the PBIs after gaining as much as much knowledge as possible. Since JIT planning focuses on the implementation for only one PBI, chances that the tasks associated with the PBI being missed out are much less. If the sprint duration is long (say 30 days), planning out the tasks just before the execution helps the team in easily remembering the design/execution strategy. On the corollary, it is very hard to remember a discussion about the tasks, designs and possible execution strategy (for a particular backlog item) which happened 20-25 days ago.

5. Development team member does not lose much if he he away during spring planning: Face it, in real life there are unforeseen emergencies, vacations and off-days. And if that happens to be on the day of sprint planning, he/she is missing out a lot. Though I would not like to call it an advantage, JIT planning minimizes the impact of a team member being away during the "how to" part of planning by distributing the "how to" for all the backlog items throughout the sprint. I assume that, in any team, the other team members will bring this person up to speed once he is back and the "burden" of getting that person up to speed is much less. Disclaimer: This point, in no way, encourages team members to stay away during planning, instead , I just want to convey the the impact of that event is reduced for the person and for the team.

6. Focuses on feature implementation explicitly rather than tracking efforts: Most of the commercial tool vendors have some form of sprint burn-down charts where the "ideal" trend is mapped along with the "actual" effort remaining. As bad as it might be, this brings up comparison between ideal burn-down, the actual and the variance. If JIT planning is used an ideal trend-line for the sprint backlog would not be possible as not all tasks and efforts have been identified for all the sprint backlog items.  If the stories cannot be split to smaller stories/features which can be implemented in a day or two, but takes three, four or five days to complete, it might be worth while to track the remaining efforts (for the tasks associated with that feature) using a burn-down chart (only for that feature). Considering this, the sprint can have as many "feature burn-downs charts" as the number of features taken up for implementation for the current sprint.

These are some of the ideas which, I hope, will help the team be more efficient. Please feel free to share your thoughts.

Reasons why your agile adoption might fail

Rebecca Parsons gave a wonderful talk on why agile is not the easy way out, but is the only way. Having worked on multiple projects agile/lean/scrum, I can only agree with her. Most of the organizations adopting agile feel the pain of traditional methodologies and think agile is the silver bullet which can solve all their problems. Other organizations say they are following agile, without even knowing that there exists an agile manifesto, simply because they don't want themselves to be excluded in the race of adopting agile. Some organizations openly admit that they are still following traditional methodologies, because they are not comfortable with agile. Agile is usually associated with a lack of disciplined development efforts and loss of control. Nothing wrong with a traditional waterfall based SDLC, but time and again we have seen that a lot of software projects either go beyond their allocated budgets, time-lines or both. Though I strongly suspect the methodology employed in the recent Standish group, which reported more success with agile projects, I believe that if a more methodical survey is carried out, we would still arrive at the same conclusion. VersionOne recently published some statistics on agile adoption. Though the percentage of agile projects which did not fail was about 16%, the top category of failure was "do not know" - 13%. These are some of my thoughts on why agile adoptions and transformation fail

Failure to understand agile principles, but just implement the practices

Agile principles are like roots of the tree, they hold the tree together. The trunk and branches represent different processes (XP, Scrum,Kanban, etc) and practices (TDD,CI, Pair programming,etc) respectively. Most of the times, we neglect what we don't understand (the agile principles) and instead focus on what is easy (the practices). And the results are disastrous. We can see that from the survey that at least 78% of the respondents practiced at least one of the agile practices (daily stand-up being the most widely adopted practice at 78%), but 84% of the agile projects failed for one reason or the other. What would you think of a doctor who treated you for the symptoms, instead of understanding the underlying cause and treating the cause ? I can only attribute this kind of failure statistics to the fact that people adopt the practices without understanding why they need a certain practice.

Lack of clear understanding on what is agile

Agile, to me, is a journey. Not a destination. It is a journey striving towards continuous improvement. And it goes beyond the development team. Unfortunately most of the adoptions stop with the IT department. And to fully realize the benefits of agile development, other departments should also be educated about agile principles and why it helps development teams. These include sales/marketing, Human Resources and facilities as well. When sales/marketing is not fully involved with the transition, they promise unrealistic features and delivery dates to their customers. They associate agility with flexibility and speed (and this is true) and that the dev team should match their expectations (which, realistically might be beyond the teams' capability). Regardless of how good the team is, such unrealistic expectations set the foundation for failed projects. No methodology can fix these kind of issues.

Lack of education

One of the organizations I worked with was ready to transition their very well experienced Project Manager (who also had a very good understanding of the domain) to the role of a product owner (in fact, I suggested it) but refused to send him to a Product Owner Training (citing cost reasons). It is easy to cut down on things which you can quantitatively measure (like cost), but hard to derive ROI on things which are subjective/qualitative, like improved product management. Granted, if that person had seen how a good product owner works, I would not have had much concerns, but that was not the case. The product owners were themselves new to scrum and were half way around the globe. Now, I don't mean to say that a simple two day training would make a good product owner/scrum master out of you, but that is just a starting point. If you have a coach who can guide you and your team, that is equally good. Regardless, learning is a continuous process and you learn from conferences, books, blogs, subscribing to distribution lists (Yahoo groups, LinkedIn) and from peers.

Adopting agile and expecting that it is going to solve your problems

Agile is about transparency and visibility. Agile gives visibility to your problems. It will not solve problems for you. If you hire bad programmers, it is going to make it more evident, but it is up to you to hire better programmers. If the business stakeholder is sitting half way across the globe and is hard to collaborate with him/her (and it is true for even someone sitting in the next cubicle desk, but does not want to collaborate), agile is going to make this problem more visible, but does not solve this problem. If you are building the wrong thing, but skipping sprint reviews, "your customized" agile adoption will not help you. If you have customized scrum to accommodate change requests during sprints, say goodbye to good processes and say hello to the devil. If you pile up technical debt, you will eventually pay a price for it. You are better off paying it early than paying it later.

We are resistant to change

I once read a story about an old dog sitting on a nail and whining all night. A traveller asks a wise man why the dog does not move away to which the wise man replied "It does not pain enough for the dog to move away". A lot of us do not want to change the way we think, simply because we are used to it, it feels comfortable and changing any of that will make us uncomfortable. We justify it saying that "a known devil is better than an unknown angel". Some of us are like ostritch which burries its head in the sand. We don't want to acknowledge a problem because if we do acknowledge it, we will be forced to take action. But change is the only constant. And We resist change because we lack the discipline to follow new practices.