Thursday, July 30, 2015

Scrum Project Success Metrics

Last week I was reading „The 2015 State of Scrum Report“ from Scrum Alliance. In February 2015, Scrum Alliance surveyed almost 5000 people about their use of Scrum. The survey respondents make up a diverse group, representing 108 countries and 14 industries. They reflect a range of functional areas, including IT software development, product development, operations, human resources, executives and sales and marketing (you can download the full report here).

Whilst reading, two questions and their responses caught my attention.

1) What were some of the challenges faced by your organisation in achieving its goals with Scrum? 

The number one answer with 52% was “We did not have clearly identified metrics to identify and measure the success of Scrum projects and delivery”.

2) Considering all the projects in your organization that were managed using Scrum, what percent of the time would you estimate they were delivered successfully?

The answers were as following:
a. 75%+ of time: 42%
b. 60-75% of time: 32%
c. 25-50% of time: 14%
d. 0-25% of time: 12%

Regarding the first question it is hard for me to understand that organizations are not able to measure the success of a project/delivery. For me this is independent of Scrum, cause the same measurement can be used by any other project methodology.

Yes the sentence “full scope, on time, in budget and high quality” will not work with Scrum, but it does not for Waterfall either. This only states that your planning was correct (or that your project adapted to the planning). It does not say anything about the successful outcome of the project itself.

Depending on your project you can use a set of metrics like:
- Employee / Project Member satisfaction
- Stakeholder satisfaction
- Customer satisfaction
- Quality delivered to customer
- Time to market
- Return on Investment
- Meeting business case objectives
- Customer adaption
- Meeting governance criteria
- Benefits realization

When you look at this list of metrics you will notice that none of the typical Scrum Project metrics like Techncial Debt, Retrospective Items or Velocity are listed. This is because those metrics measure the quality of the Scrum process and are focused on improving Scrum teams. They do not say anything about the success of the project itself. When defining your own organization's metrics it very important to make this difference.

What you can also see is that none of the metrics is Scrum specific. Nor specific for Waterfall or any other method. So when the answer is “We did not have clearly identified metrics to identify and measure the success of Scrum projects and delivery” this is the same as saying "I have no metrics in place to identify success of any project or delivery in my organization". Which is pretty scary.

The reason why the second question caught my attention is the relation with the first. When you have no project success definition how can you answer the question regarding project success rates?

Read more…

Monday, July 13, 2015

14 Principles of Agile Actuarial Modeling

I recently worked on a challenging actuarial modeling project for a large insurance company. Actuarial modeling applies mathematics and statistics to assess risk in the insurance and finance industries (and in some other industries as well). Education and experience both help give actuaries professional qualification, and passage of professional exams is required in many countries.

Actuarial modeling includes a number of interrelated subjects, including probability, mathematics, statistics, finance, economics, financial economics, and computer programming. Historically, actuarial modeling used deterministic models in the construction of tables and premiums. The science has gone through revolutionary changes during the last 30 years due to the proliferation of high-speed computers and the union of stochastic actuarial models with modern financial theory (see Wikipedia).

Building actuarial models for clients, including internal clients, is a difficult and complex task. Much has been written on how to approach the challenge of managing a modeling project, and almost all of this guidance describes, more or less, a Waterfall model. They may use different terms, but it is the same: specification, design, development, testing, and implementation.

It's no surprise that actuarial modeling projects that follow this method suffer from similar issues as software development projects that are managed this way. My recent project was no exception. Therefore, we decided to implement Scrum for managing the modeling project.

I was charged with helping to make the transition. During the transition and after the first few months of running it with Scrum, I learned many new things. Although actuarial modeling has some similarities to software development, it is not quite the same. As a result, I adopted the principles of the Agile Manifesto to actuarial modeling:

14 Principles of Agile Actuarial Modeling

1. Our highest priority is to satisfy the customer through early and continuous delivery of a working actuarial model.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver a working actuarial model frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale.

4. Actuaries, developers, data specialists, and business analysts must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a modeling team is face-to-face conversation.

7. Without the necessary input data available, the model is useless.

8. A working model that can be evaluated with real data is the primary measure of progress.

9. Agile processes promote sustainable development. The sponsors, modelers, developers, and actuaries should be able to maintain a constant pace indefinitely.

10. Continuous attention to the best model practices and good design enhances agility.

11. Simplicity -- the art of maximizing the amount of work not done -- is essential.

12. Models are self-documenting. Everything needed to understand the model is in the model. This allows models to be shared, understood, changed, adapted, and used.

13. The best models emerge from self-organizing teams.

14. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

This was my first Scrum implementation outside of a software development or a business intelligence project, and I could not be happier with the outcome.

This article appeared first on 

Read more…