Metrics that matter – how we track our digital transformation
At Microsoft Core Services Engineering and Operations (CSEO), we’ve developed a solution that allows us to summarize and track our most important business processes and systems. CSEO Scorecard has transformed the way that we understand and manage our business. We’ve integrated it with our regular processes and it provides the most definitive source of operational truth for our organization. We’ve taken large amounts of system and process data and refined it to provide accurate and actionable information for our entire organization. With CSEO Scorecard, we’re improving employee satisfaction, enabling growth, and driving operational effectiveness on our road to digital transformation.
The state of data at Microsoft
At Microsoft, data is everywhere. As we continue our progression to digital transformation, the data produced by our digital systems and processes borders on overwhelming. Our business processes produce logs and event data ranging from employee satisfaction to financial results to security compliance. At any point in time our systems present thousands of individual points of data on running processes. There is simply no feasible way to represent each of those individual data points in a complete view of system process status for our organization. Even if the data is gathered, aggregated, and summarized, important data and information is lost in the immense number of averages and medians.
Knowing what data is important presents another challenge to the modern IT organization. In our global organization that employs over 120,000 people, determining which data is important to our organization as a whole and our individual business groups is an immense task in itself.
How can we gather and aggregate relevant data? How do we decide which data provides visibility into our organization’s most critical business processes, so we can better monitor, understand, and manage the way we do business? Our executive leadership asked us this very question. This paper examines our responses and the results we’ve generated.
Finding the data that matters most
When our executive leadership came to us looking for better insight into how our organization operates, they came with a simple request: provide them with a dashboard displaying the top 35 metrics that demonstrated how well our organization was running. While the request was simple enough, fulfilling that request was not. Selecting 35 metrics that properly summarized the hundreds of thousands of individual metrics that exist within the CSEO environment was huge. We had to ask ourselves some important questions, such as: What gets done at CSEO and how is it measured? How do we determine what metrics are the right ones or the most important ones to put on our dashboard? Which metrics will strategically drive our business forward into digital transformation? It was this last question that provided the direction for assembling what our dashboard might look like.
Making sense of our metrics
We were already collecting a large amount of data throughout CSEO, and many of our business groups were already using that data to examine their own metrics and create their own dashboards and insights. After examining the dashboards and metrics situations, we identified several opportunities for improvement:
- Most business groups only analyzed data and metrics from within their organization. While this makes sense from a departmental perspective, CSEO’s overall performance depends on all of our business groups and how they interact with each other.
- Most dashboards provided high-level views of data for their respective business groups but couldn’t drill-down into the data. As a result, in meeting reviews, presenting the data usually required pasting the relevant dashboard screenshots and other more detailed reports into a Microsoft PowerPoint slide deck, rather than presenting from the dashboard itself.
- Examining the data required extra steps. In most cases, closer examination of metrics or underlying data required further data manipulation using reporting tools to drill down to specific metrics. As a result, our data analysts couldn’t provide relevant information to our leadership in a timely and reactive fashion.
- We did not have a single source of truth. If we assembled the different metrics reporting and dashboard solutions being used, they did not combine to provide an accurate and complete idea of what was happening at CSEO.
We realized that from both a metric and solution perspective that we were going to have to start at the beginning by determining which metrics mattered. Then we needed to develop a solution that would enable us to display the data and interact with it in a way that allowed our entire business to respond and evolve.
Creating metrics that matter
In our solution, metrics mattered. In fact, they were the most critical aspect of the entire project. We wanted to ensure that we identified the metrics that were important to both our leadership team and organization. Metrics were at the heart of the solution and it’s where we started our plan.
Our overall process from start to finish involved stakeholders from every part of our organization. We asked for input from the system and process owners who knew what was important within their section of the business. They were our primary means of acquiring the metrics. Once the metrics were established through a back and forth review process, we were able to create a solution that would enable them to examine and use the metrics to effect change. The start-to-finish process looked something like the following high-level steps:
- Obtain direction and goals from our leadership team.
- Initiate reviews and metric discussions with system and process owners across the organization.
- Review collected metrics from system and process owners.
- Review the first list of metrics with the leadership team and refine the primary list of metrics to present on a leadership dashboard.
- Get signoff from the leadership team for the list of required dashboard metrics.
- Create and test the CSEO Scorecard solution.
- Finalize targets for top metrics.
- Run the first CSEO Scorecard review and test metric targets.
- Finalize CSEO Scorecard and implement system wide.
Step 1 through 9 took us approximately three months. The end result was a dashboard and scorecard solution that gave our leadership team an accurate and agile view of the entire organization.
Building criteria for metric selection
One of the most important early steps in the process was defining the criteria for what we wanted to present for our top-level dashboard metric. We were given a target of 35 metrics that would comprise a complete view of our organization. In order to determine which metrics would fit onto that dashboard, we developed a set of criteria:
- The metric must reflect strategic values of the organization. We only wanted metrics that would drive us in the direction we wanted to go. We summarized this as data that was important to us and our customers.
- The data presented by the metric must be actionable. If a metric wasn’t performing as we thought it should, we wanted to present it on the dashboard so as to call attention to it and be able to use the data do something about it.
- The metric must be timely and accurate. If the data supporting a metric wasn’t current or didn’t accurately represent the state of our business, it wouldn’t help us. Our high-level business reviews happen on a monthly basis, so metrics that update less frequently than once a quarter weren’t useful to report.
- The metric must be easy to understand with a clear definition. We wanted metrics that were clear, concise, and didn’t require further extrapolation or a reference card to explain what it was representing.
- The metric data must be valid for at least one year. Brief trends and small aspects of our business come and go. We wanted a picture of what CSEO would look like a year from now. This meant capturing the important business data, the things that would last.
- The metric needs to have a clear target for value and timeline. We needed to identify what our goals were for the metric, and a timeline for achieving those goals. If we couldn’t establish an accurate timeline to improve an underperforming metric, it didn’t belong on our leadership dashboard.
- The metric must have single owner. A single owner encompasses three aspects for the data: a single definition of the metric, one way to obtain the data, and one expert to consult on how we could best leverage the metric on the leadership scorecard.
Establishing metric goals and categories
In addition to the criteria for selection, we also worked with the leadership team to establish the high-level goals for what this dashboard would report. Our leadership team wanted a solution that would truly benefit the business; something we could use to make CSEO better. We came up with three primary goals for the dashboard results:
- Improve employee and vendor satisfaction. We wanted to improve the experience of working for and with CSEO. This meant improving satisfaction for both our employees and our customers.
- Enable growth. Growth speaks for itself. In the IT industry, you can’t stand still. We wanted the dashboard to help us plan for future grow and move forward in the right direction as an organization.
- Drive operational effectiveness. Being better is important to CSEO. Microsoft CEO Satya Nadella has articulated his goal for Microsoft, which is to “empower every person and every organization on the planet to do more and achieve more,” and that certainly includes our own organization. Operational efficiency is a big part of both that vison and the roots of digital transformation.
Another important consideration was the different parts of our organization that contribute to the whole. Breaking our organization down into primary business and functional parts enabled us to create categories into which we could place our set of metrics. The categories were:
- Engineering. The engineering category contains metrics that track
our performance in managing our IT operations. These metrics include:
- Major incidents
- Unrealized revenue
- Data center patching
- Efficiency. The metrics in the efficiency category measure how
we’re doing in areas related to maximizing our CSEO resources. These metrics
- Cost per project launch
- On-time delivery to customers
- Compliance. The compliance metrics show us where we stand
regarding compliance, including important areas such as legal compliance and
internal policy compliance. These metrics include:
- Accessibility compliant applications
- Security development life cycle onboarding
- DevOps toolkit compliance
- Costs and people. In this category, we use metrics that underline
the key human resources and financial aspects of our business. These metrics
- Employee to vendor ratio
- Budget and spending variance (year-to-date)
- Budget forecast and spending variance (quarter-to-date)
- Experience. The Experience metrics tell us how our employees,
partners, and customers are engaging with our products and services. These
- Positive sentiment percentage
- Retail store user engagement
- Search success rate
- Quality. The Quality category has metrics that measure the
critical quality and performance of services that support our business. These
- Microsoft Teams meeting index
- Sales opportunity conversion rate
Creating and implementing CSEO Scorecard
Once we established our goals and metrics, we began creating our dashboard that we named CSEO Scorecard. We created our solution using the following technologies:
- Microsoft Power BI. Power BI provided an immediate solution to almost all of our high-level solution goals. It provides dynamic views on live data and the ability to drill down into said data. It easily integrated with our cloud-based processes and business systems and it was intuitive and easy to begin creating data visualizations. In fact, the ease of Power BI implementation was one of the main reasons we were able to go from idea to solution in three months.
- Azure Analysis Services. Continuing with the cloud-based nature of CSEO Scorecard, we used Azure Analysis Services for data analytics. We pulled our data from a huge dataset, and we needed to be able to manipulate and represent our data using different models to support the variety of drill-down capabilities we built into CSEO Scorecard.
Figure 1. The CSEO Scorecard dashboard.
Maintaining a rhythm of business
CSEO Scorecard transformed the way we perform our regular organizational review meetings. With the dashboard presenting live data and the ability to drill down to specific metrics and results immediately, we are now able to streamline our meeting process. We also improved our decision-making process. If a question came up in the meeting, we no longer needed extra time to run additional reports or consult with metric owners. With CSEO Scorecard we could expose the necessary data, find the answer to improving our processes, and determine a business direction during the meeting.
We’ve begun establishing a new rhythm of business with CSEO Scorecard as the core component. By creating a process and rhythm around CSEO Scorecard, we ensure we have the most relevant and accurate data and give our metric owners time to validate the results and provide commentary prior to moving into our monthly review meetings.
Our CSEO data analysts work to maintain the rhythm of business for CSEO Scorecard. They create and manage the dashboards, perform the data analysis tasks, and work with the metrics owners to establish business and data standards for the metrics on the Scorecard. The monthly rhythm of business looks something like the following high-level steps:
- The first step in the process is a Scorecard refresh. All of the data is refreshed and aggregated, and reports are assembled. The initial results of the refresh are sent to metric owners who have 10 days to review the results, ensure accuracy, observe highlights and lowlights, and provide feedback in the form of commentary.
- Once the metric owners have had time to review the results of the refresh, the data analysts examine the commentary, confirm highlights and lowlights, and make any necessary changes to the dashboard results.
- When the data analysts’ changes are complete, they lock the Scorecard in preparation for the monthly review. The final highlights and lowlights are available to metric owners, so they can view what results will be displayed for their part of the business.
- After the Scorecard is locked, it goes to the leadership team for monthly review. The leadership team uses the dashboard to assess business performance, determine potential courses of action, and create necessary action items.
- After the monthly review is complete and the leadership team has discussed action items with the owners of the specific metric involved, the Scorecard is published to all of CSEO for viewing.
Figure 2. The monthly rhythm of business process for analyzing and publishing scorecards
Several times throughout the process of assembling CSEO Scorecard we encountered challenges such as:
- Too many metrics. The sheer number of metrics and the broad nature of our business meant that it was very difficult to select the most important metrics. We were initially able to reduce the number of metrics that we presented to almost 80, considerably more than the 35 they asked for. We continued to work with leadership and metric owners, examining and comparing metrics until we could settle on the 33 that ended up on the initial CSEO Scorecard. We’ve since been able to reduce that number to 23. We've worked to create the most applicable and actionable dashboard metrics possible.
- Maturity in the metrics. Many of our metrics didn’t use complete or accurate datasets or were not as updated as we wanted. We had a lot of aspirational targets, but we needed them to be realistic. We worked with metric owners to improve data quality and determine realistic targets for the metrics.
- Reluctance to share information and contribute. While we all belong to the same organization, metric owners were sometimes protective of their areas of responsibility. They wanted to know why we were looking at their business and why their metrics needed to be disclosed to the rest of the organization. In many cases we found metric owners were protective because they didn’t think that the metric results would be favorable to their business unit. While the goal wasn’t to subject the team to criticism, it simply revealed to leadership the need for improvement. This then resulted in resources being assigned to correct the issue, ultimately leaving the metric owner with a better, more efficient process or system. We were able to show metric owners how the contribution to CSEO Scorecard could help them rather than impede or inconvenience them.
We’re using CSEO Scorecard to achieve our primary goals for the project: enable growth, drive operational effectiveness, and improve satisfaction. CSEO Scorecard has allowed us to realize benefits within our organization:
- Cost and time savings. We can conduct our review meetings using live data, and establish goals and plans based on those results right in the meeting. Consequently, we come out of the meeting ready to improve. We don’t need to go back and forth between teams and leadership or obtain follow-up data or information; everything is done in real-time. In addition, the time required to prepare for review meetings has been reduced if not eliminated. We’re reporting using our live dashboard, so no reports or PowerPoint decks have to be created before the meeting. The dashboard is used for everything.
- Rapid quality improvements. The metrics that we use on the dashboard give our leadership team visibility into our business, and quickly reveals areas that might need improving. This enables us to enact changes quickly and efficiently. Once we initiate the plan of action, we can monitor the results to ensure that the fix is working.
- Reduced outstanding issues. With the immense level of transparency that the dashboard provides, we can more easily see where processes aren’t working. In six months, we doubled the number of metrics in the dashboard that met or exceeded their threshold.
While the primary goal of CSEO Scorecard is moving our organization forward, we’re anticipating changes and improvements to the dashboard in the near future that will enable it to provide even more value. These changes include:
- Move to level 2 dashboards. We’re working on creating level 2 dashboards for each of the individual components within our organization. With the improved dashboards, we’re hoping to provide the same value we realize in our review meetings to smaller business units within our organization. This will enable them to have the same opportunities for efficiency and improvement and empower them to drive efficiencies within their own business units.
- Fill in data detail. Having a comprehensive dashboard solution quickly reveals missing data. CSEO Scorecard has showed us several of our processes and systems that could benefit from a deeper and more detailed data collection.
CSEO Scorecard has transformed the way that CSEO understands and manages our business. We’ve worked it into our regular processes, and it provides the most definitive source of operational truth for our organization. We’ve taken petabytes of system and process data and refined it into a tool that provides accurate and actionable information to our entire organization. With CSEO Scorecard, we’re improving satisfaction, enabling growth, and driving operational effectiveness on our road to digital transformation.
For more information
Microsoft IT Showcase
© 2018 Microsoft Corporation. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.