This is the eighth article in our agile project portfolio management series. The first article explored the concept of project portfolio management and how it relates to projects that are executed with agile delivery methods. The following articles discussed demand management, categorization, evaluation, funding, and alignment. The previous article was about portfolio authorization and the role of the Portfolio Owner. This article is about monitoring your portfolio.
We already discussed the need for some information at the project proposal phase in our article about demand management, but at the latest in the evaluation and alignment phase of the portfolio management process it becomes clear that we need information about ongoing projects in our portfolio in order to make informed decisions about these projects and the portfolio as a whole.
Anything that can be measured can be improved. However, organizations don’t always do sufficient monitoring. Few organizations actually track project and portfolio performance against benchmarks. Worse, strategic multiyear initiatives are the least likely to be tracked in a quantitative, objective manner. For smaller organizations, the absence of such a process might be understandable, but for a large organization, this is a must.
Not monitoring project results create a vicious circle: If results are not tracked, then how can the portfolio management and strategic planning process have credibility? It is likely that it doesn’t, and over time, the risk is that estimates are used more as a means of making a project appear worthy of funding than as a mechanism for robust estimation of future results. Without tracking, there is no mechanism to make sure initial estimates of costs and benefits are realistic.
This is crucial because there can be a tendency to provide cost estimates that are artificially low in order to gain project approval, especially where there is no monitoring process in place to capture this. What’s to stop the winner’s curse of the most inflated projections getting funded based on elaborate but dubious projections, rather than the best project with credible and realizable, if more modest, strategic benefit? If you are not monitoring performance, then not only will execution performance inevitably deteriorate but your selection process will lose legitimacy and will ultimately become a time-consuming charade.
Project Metrics
In my opinion, the minimal set of metrics that should be monitored for each project in execution are:
- Estimated end date (for resource allocation)
- Costs till date (exact number)
- Expected costs till finish (rough estimation)
- Remaining budget (exact number)
- Delivered value since last status update (YES/NO question)
- Expected gain on investment
- Risks
- Issues
- Allocated resources
Depending on your company and project you can use additional metrics like:
- Employee / Project Member satisfaction
- Stakeholder satisfaction
- Customer satisfaction
- Quality delivered to customer
- Time to market
- Return on Investment
- Meeting business case objectives
- Customer adoption
- Meeting governance criteria
- Benefits realization
When you look at this list of metrics you will notice that none of the typical Scrum project metrics like Technical Debt, Retrospective Items or Velocity are listed. This is because those metrics measure the quality of the Scrum process and are focused on improving Scrum teams. They do not say anything about the success of the project itself. When defining your own organization's metrics it very important to make this difference. What you can also see is that none of the metrics is Scrum specific. Nor specific for Waterfall or any other method.
Monitoring needs to be balanced. Not monitoring project success can cause sloppy execution, which is costly. However, monitoring too much can cause a project to drown under the weight of its own status updates. Collect data frequently, but only at the level required to make a decision.
Budgeting is one example. It is important to define what constitutes a meaningful budget overrun and measure to that level. If you only care about the aggregate budget, then breaking out the full budget by line item on a monthly basis is overkill and can distract from the decision and monitoring process that you’re trying to follow. Where data collection is not automated, track only what you intend to act on; more detail can always be captured subsequently should an issue arise (see Exception Based Reviewing)
Reporting
Reporting is critical to an effective project portfolio management system. Whereas collecting up-to-date and accurate data is a challenge, building an effective reporting system is relatively easy. Reports should answer specific questions and be tied to specific processes. Reports that simply look appealing or provide a host of data without underlying meaning are ineffective. Therefore, reports should be the outcome of your project portfolio management processes. It is important to consider the following key report types:
Single-page project report: Using a consistent format for project status reports is important. Doing so means less time can be spent on the layout and more attention can be paid to the data within the report. Each project summary should be contained on a single page or screen, with top-level details and the contact information for the project manager and project owner provided should questions arise. The data should also be time-stamped.
Financial information: This provides aggregate roll-up on budget and costs across projects with the ability to drill-down into individual projects.
Project dashboard: One-line summaries of the projects currently in execution, filterable by important grouping such as division or project sponsor (see Categorization). Often these lists can run to thousands of projects, so by default, filter on the poorly performing projects (such as no value delivered on the status indicator) first, as it is likely those projects are of most interest and require the most attention.
Resource allocation view: This is a view of current resource availability and utilization for the coming 12 months. This means people, machines, money, and anything else that could be a bottleneck in realizing your projects.
Strategic alignment: This provides a view of the strategic goals of the business and how the current projects contribute to them.
Flexible pivot table capabilities: No project report or set of reports will capture everything that is needed, even where drill-down capabilities are available. Using a drag-and-drop pivot table-style analysis on the inventory of projects or proposals can help executives and others find the information that they need to answer a particular question and also lead to creative analysis to produce new report types to further support decision making.
Creating an effective dashboard
Once you know what you are looking to measure, it helps to create an on-screen dashboard that captures the relevant information. The term dashboard uses an automotive analogy to convey the ideas of getting all relevant information simply summarized in one place. This gives an opportunity to effectively summarize project data in an attractive and transparent fashion. Usage of color and graphical indicators can highlight areas of importance, and the best dashboards combine current and trend information to identify patterns.
Online dashboards can also be created to enable drilling down so that the summary data are presented but if more information is needed, the user can double-click on a project to see full detail on risks, issues, budget, the project manager, team, and current tasks. This type of drill-down setup helps to ensure that the initial dashboard is concise but the information is all easily accessible when needed. See for example the Office 365 Project Portfolio Dashboard below.
Not all dashboard metrics need to be quantitative. Team morale is often hard to define and cannot easily be pulled from a reporting system like the most recent cost data, but it is important and can be very useful in monitoring how the portfolio is progressing. Ensuring that the right data are captured on a dashboard is critical, regardless of whether that data can be pulled out of a database.
Also, with broad survey technology, it does become easier to leverage the wisdom of crowds across the project team. No one individual or data point may be the single source of truth on the morale of project team, but by anonymously surveying the project team and aggregating the results, team morale can be tracked on an ongoing basis. Technology makes the collecting of this sort of data increasingly simple.
Transparency
Often, organizations lock down the permissions or access rights that are given to their employees too finely. Security is based on minimizing the negative consequences of information leakage, rather than maximizing the positive benefits of information sharing within the organization among employees. Consider making various project key performance indicators (KPIs) public to a broad set of employees within the organization. This will help foster new ideas for projects and improve project collaboration.
Ideally, disclosure at the KPI level should not reveal confidential information and more granular permissions can be set on drill-down financials (like individual wages and rates for external consultants). Generally, providing more information on project performance and processes can help with the goal of avoiding duplication of effort, and provide a further opportunity to emphasize the goals of the organization.
Portfolio Review Meetings
The Portfolio Owner will schedule regular (I recommend monthly) portfolio review meetings. Participants in these meeting are the Portfolio Owner, Portfolio Committee and invited guests (for example the Product Owner of a large initiative to give a detailed update).
This monthly portfolio review meeting helps make the collection of data actionable. The meeting can review the dashboard, and particularly any significant changes related to the dashboard, and help address them. Knowing that the dashboard will be used in this way makes the dashboard meaningful. Simply creating a well-designed dashboard in the hope that someone will find and act on it is wishful thinking if there is no process to support it.
Ahead of creating the dashboard, consider decision thresholds and the resulting actions. If an indicator turns red for a particular project, what would be the result? If no action would be taken, then why is that piece of information on the dashboard? If the resulting decision is ambiguous, then clarifying the course of action ahead of time can be enormously helpful, given that making decisions at the moment can become a lot more heated. It is also important to consider what other information could result in portfolio actions being taken. If the information is material, consider adding it to the dashboard, even if the data in question are not quantitative. For example, project management might have a perception of project performance.
Ideally, the dashboard should capture all aspects of project status. If a project could be killed despite having a completely positive dashboard performance, then either the dashboard or the organization is not working effectively.
Exception-based reviewing
It may not be necessary to go into detail on all projects, but ensure that those projects in the portfolio that are not delivering the expected value are followed up on. A process that includes all projects but only invests significant time in those projects that are off target is exception-based reviewing. Exception-based reviewing means that only projects that pass a certain threshold are reviewed. For example, projects that are more than 10 percent over budget, or are not delivering value for over two months, are highlighted and require a decision as to whether corrective action is needed. This saves time in not having to review all projects in a detailed manner, as the projects that are on track are not analyzed to the same level of detail. It also means that attention can be drawn to projects when problems occur, not necessarily when the review meeting happens.
Having alerts set up should a project go off target means that stakeholders can be notified in real-time, rather than having to check the dashboard or waiting to receive the data within a particular review meeting. This can improve the speed of response, which can help drive more agility in portfolio management. It also creates a feedback loop, which can help drive improvement. Driving the process by alerts rather than by scheduled meetings can be helpful, even if the review meeting is monthly, if a project starts to go off target the day after the review meeting. By the time of the next review meeting, it might be a month behind schedule, which for a six-month project likely means the project has severe issues.
Alerts also work well for the late stages of projects—as delivery is close, more monitoring attention is often needed, and alerts can be configured to support that, without requiring the need to scrutinize all projects in detail as they approach delivery. An efficient system of alerts ensures that management focuses on the right issues systematically, rather than spreading themselves too thinly across the whole portfolio.
Other articles in this series
1. Agile Project Portfolio Management?2. Agile Project Portfolio Management? Demand Management
3. Agile Project Portfolio Management? How to categorize your project backlog
4. Agile Project Portfolio Management? How to evaluate your portfolio
5. Agile Project Portfolio Management? How to fund your projects
6. Agile Project Portfolio Management? How to align your portfolio
7. Agile Project Portfolio Management? How to authorize your portfolio
8. Agile Project Portfolio Management? How to monitor your portfolio
9. Agile Project Portfolio Management? Conclusion
You can buy my books The Art of Technology Project Portfolio Management and The Science of Technology Project Portfolio Management separate or as a bundle in my store. Just click here or on the image.
