This article will explain what Wideband Delphi and Monte Carlo Simulation is (Part 1 & 2), how you can use them together for Scrum Project Estimations (Part 3), and gives you a tool that I build in Excel to guide you through the Wideband Delphi estimation and run the Monte Carlo Simulations easily yourself (Part 4).
I named this tool the Oracle Planning & Estimation Toolkit. This because the Wideband Delphi method was named after the greek oracle and the last 6 letters of Monte Carlo contain the word oracle as well.
Part 1: Wideband Delphi Method
Barry Boehm and John A. Farquhar originated the Wideband variant of the Delphi method in the 1970s. They called it "wideband" because, compared to the existing delphi method, the new method involved greater interaction and more communication between those participating. The method was popularized by Boehm's book Software Engineering Economics (1981). Boehm's original steps from this book were:Step 1 Coordinator presents each expert with a specification and an estimation form.
Step 2 Coordinator calls a group meeting in which the experts discuss estimation issues with the coordinator and each other.
Step 3 Experts fill out forms anonymously.
Step 4 Coordinator prepares and distributes a summary of the estimates
Step 5 Coordinator calls a group meeting, specifically focusing on having the experts discuss points where their estimates vary widely
Step 6 Experts fill out forms, again anonymously, and steps 4 to 6 are iterated for as many rounds as appropriate.
Step 1 Choose the team: The project manager selects the estimation team and a moderator. The team should consist of 3 to 7 project team members. The team should include representatives from every engineering group that will be involved in the development of the work product being estimated.
Step 2 Kickoff meeting: The moderator prepares the team and leads a discussion to brainstorm assumptions, generate a WBS and decide on the units of estimation.
Step 3 Individual preparation: After the kickoff meeting, each team member individually generates the initial estimates for each task in the WBS, documenting any changes to the WBS and missing assumptions.
Step 4 Estimation session: The moderator leads the team through a series of iterative steps to gain consensus on the estimates. At the start of the iteration, the moderator charts the estimates on the whiteboard so the estimators can see the range of estimates. The team resolves issues and revises estimates without revealing specific numbers. The cycle repeats until either no estimator wants to change his or her estimate or the estimators agree that the range is acceptable.
Step 5 Assemble tasks: The project manager works with the team to collect the estimates from the team members at the end of the meeting and compiles the final task list, estimates and assumptions.
Step 6 Review results: The project manager reviews the final task list with the estimation team.
You see that this process fits perfectly for any Scrum Team or Teams. The "project manager" is the Product Owner, the "moderator" the Scrum Master and the "team" the Scrum team(s) or representatives of these teams. The "WBS" is the Product Backlog. This is not practicable nor useful for a Sprint Planning, but at the start of a new project this is extremely helpful to get an idea bout size of the whole product and bigger epics/stories.
Part 2: Monte Carlo Simulation
Uncertainty in Forecasting Models
When you develop a forecasting model – any model that plans ahead for the future – you make certain assumptions. These might be assumptions about the investment return on a portfolio, the cost of a construction project, or how long it will take to complete a certain task. Because these are projections into the future, the best you can do is estimate the expected value.
You can't know with certainty what the actual value will be, but based on historical data, or expertise in the field, or past experience, you can draw an estimate. While this estimate is useful for developing a model, it contains some inherent uncertainty and risk, because it's an estimate of an unknown value.
Estimating Ranges of Values
In some cases, it's possible to estimate a range of values. In a software development project, you might estimate the time it will take to complete a particular Product Backlog Item; based on some expert knowledge, you can also estimate the absolute maximum time it might take, in the worst possible case, and the absolute minimum time, in the best possible case. The same could be done for IT infrastructure costs.
By using a range of possible values, instead of a single guess, you can create a more realistic picture of what might happen in the future. When a model is based on ranges of estimates, the output of the model will also be a range.
This is different from a normal project estimation model, in which you start with some fixed estimates – say the time it will take to complete each of the Product Backlog Items of a project – and end up with another value – the total estimated time for the project. If the same model were based on ranges of estimates for each of the Product Backlog Items of the project, the result would be a range of times it might take to complete the project. When each part has a minimum and maximum estimate, we can use those values to estimate the total minimum and maximum time for the project.
How it works
When you have a range of values as a result, you are beginning to understand the risk and uncertainty in the estimation. The key feature of a Monte Carlo simulation is that it can tell you – based on how you create the ranges of estimates – how likely the resulting outcomes are.
In a Monte Carlo simulation, a random value is selected for each of the tasks, based on the range of estimates. The model is calculated based on this random value. The result of the model is recorded, and the process is repeated. A typical Monte Carlo simulation calculates the model hundreds or thousands of times, each time using different randomly-selected values. When the simulation is complete, we have a large number of results from the model, each based on random input values. These results are used to describe the likelihood, or probability, of reaching various results in the model.
Part 3: Combine the two methods for a Scrum project estimation
With a few small adaptions to the Wideband Delphi process it- fits perfectly in a (large scale) Scrum setup.
- we generate the input we need for a Monte Carlo Simulation.
The steps for the whole process would look like this
Step 1 Choose the team: The Product Owner, the Scrum Team and a Scrum Master form the group for the next steps, but only the Scrum Team will estimate. The Scrum Master is the moderator. The Scrum team should consist of 3 to 8 team members. On a large scale project or product group where multiple Scrum Teams follow the same goal each Scrum team will select an X number of representatives so that the total number of people is not bigger as 10. This is the maximum in my opinion to handle the following process effectively.
Step 2 Kickoff meeting: The Scrum Master prepares the team and moderates a discussion to brainstorm assumptions based on the initial Product Backlog items and decide on the units of estimation. I prefer to estimate in number of Sprints but you can also estimate in days, months or story points. Note that 4 week sprints is almost the same as estimating in months.
Step 3 Individual preparation: After the kickoff meeting, each Scrum Team member individually generates the initial estimates for Product Backlog items, documenting any changes to the necessary tasks and missing assumptions.
Step 4 Estimation session: The Scrum Master leads the team through a series of iterative steps to gain consensus on the estimates. At the start of the iteration, the Scrum Master adds the estimates to Oracle Planning & Estimation Toolkit so the group can see the range of estimates. For each Product Backlog Item we will need the following three estimations:
- Best Case Estimate
- Most Likely Estimate
- Worst Case Estimate
The Oracle Planning & Estimation Toolkit creates these three values automatically based on the single estimations of the Scrum team member. The lowest estimation is the Best Case Estimate, the highest estimation is the Worst Case Estimate and the average off all estimations is the Most Likely Estimate. Note that the average is not always good estimation of the most likely estimation, but it serves us well to start the discussion.
The Scrum Team see the three estimates on the screen and will discuss and update primarily the Most Likely Estimate. But also the Best Case and Worst Case Estimates can be changed. this mosten happens when underlying assumptions change, or the item was not clear to everybody. Updates will be entered directly in the Oracle Planning & Estimation Toolkit for everybody to see, The cycle repeats until either no one wants to change the Most Likely Estimate or the Scrum team agrees that the range is covering difference on the most likely outcome. Note that the Product Owner needs to be available during the whole meeting for answering questions and clarifications.
Step 5 Run Monte Carlo Simulation: The Scrum Master runs the Monte Carlo Simulation with a beta-PERT Distribution and the Oracle Planning & Estimation Toolkit will produce and analysis of the results.
Step 6 Review results: The Scrum Master reviews the final Product Backlog, their estimations and uncertainties (i.e. Risks) with the Scrum Team and Product owner.
As you can see the process is very straight forward and will need only two meetings.
Part 4 Oracle Planning & Estimation Toolkit
At the hand of an example I will explain the the Oracle Planning & Estimation Toolkit. Consider a simple Product Backlog like the one for the tool itself. This Product Backlog contains 8 "large" items. Remind that we are at the beginning of the project and the items are so big that they will not fit into one Sprint. It is about project estimating, not Sprint estimating.
Let's further assume that these items must be completed in sequence and cannot be done in parallel. Based on the prioritization of the Product Owner the row order is of course flexible. How long will the project take to complete? During the Kickoff meeting, we will discuss all Items and will change, split or delete them until we agree on a common understanding. This is of course only possible when the Product Owner is in the room. After this is done all Team members will prepare themselves for the Estimation meeting. It is up to the team to agree on the date of the next meeting.
During the Estimation meeting, we will start with the Wideband Delphi Input Screen (click to enlarge).
The first Team Member will give their estimations in public without commenting them. The estimations in this example are in number of 1 week Sprints.
Then the second Team Member
And so on
After all estimations are putted into the screen we will go the Wideband Delphi Estimation Screen which will look like this.
It consists out of 5 sections. The white section is the Product Backlog. The greens section contains the values of the Wideband Delphi Input Screen but are rounded to whole numbers.
The red section is where we continue our meeting. This where we will change the estimations as described in Step 4 of our method until we reach consensus.
The focus will be on reaching consensus on the Most Likely Duration, but the Best Case and Worst Case can be changed as well. As soon as changes are made the model will recalculate the values in the orange and the grey section. The orange section contains a random value between 0 and 1 and the PERT Value coming from the input (Probability, Min, ML, Max).
The grey section contains the Three-Point Estimate that is used often when using PERT but no Monte Carlo simulation.
After a consensus is reached the Scrum Master will run the Monte Carlo simulation by running the Macro RunSimulations(). The Toolkit will ask for the desired number of iterations. Typically 1000 is a good number. The results of the simulation can be found in the sheet "Montecarlo Simulation Results". They will look like this (but then 1000 rows instead of 10).
These results can then be visualized and analyzed by using the Data Toolpak from Excel. You can compute Average, Standard Deviation, Variance, etc. But the most helpful tool is the Histogram combined with the Cumulative Percentage curve.
This diagram will tell you that based on the current 6 expert opinions the probability to be done in 29 sprints is above 80%. At the same time, it tells us that the probability of finishing in 26 sprints is not even 30%. This kind of analysis will help us combining our expert input with risk simulation and will give us this way a better feeling for the estimations itself.
Currently, the Oracle Planning & Estimation Toolkit is in Beta and contains only the functionality described in this post.
When you are interested in the Oracle Planning & Estimation Toolkit and want to receive a copy just contact me. Posted on Sunday, October 18, 2015 by Henrico Dolfing