Microsoft IT uses a structured review process to monitor the Sales and Marketing organization’s app portfolio. Using modern, agile methodologies, our process facilitates efficient collaboration among key stakeholders, incorporates data analytics to resolve service incidents, and focuses on improving the end-to-end service offering. Along with efficiency in tracking and resolving service issues and effective conversations among stakeholders, we’ve seen our organization gain accountability and engineering efficiencies.


The Sales and Marketing group at Microsoft is responsible for the marketing, sales, and delivery of Microsoft products to our customers across the globe. The Sales and Marketing group is supported by a large app portfolio that provides the business functionality for our team. The Sales and Marketing app portfolio contains 180 applications.

At Microsoft IT, supporting and ensuring the business readiness of this environment is a significant responsibility and is critical to our company’s business success. In aligning with modern engineering practices, we’ve created a process we call LiveSite review to monitor and continuously improve our business app environment.

Enabling modern engineering with LiveSite reviews

We use the term LiveSite to refer to our production services. These services drive our primary business functions and provide the foundation for Sales and Marketing functionality. The way our teams have managed their LiveSite environments in the past has varied greatly. Some teams focus on telemetry data, some on incidents, and others on depth of engagement with technical and executive teams. We wanted to encourage a set of practices and tools that provide a common end-to-end live site experience across all our teams.

Understanding the LiveSite review

The LiveSite review is a critical part of the modern engineering push across Microsoft IT. It provides a structured approach to reviewing the functionality of our service environment, along with data analytics and incidents that impact service health. LiveSite reviews ensure that all stakeholders are involved in the review of a service and that we capture the end-to-end experience for a service. The goal for LiveSite reviews is to ensure that all stakeholders are working as one team—that we’re always reviewing the services we deliver to ensure optimized services and the best user experience now and moving forward.

There are some important concepts in the LiveSite environment that help us to better define LiveSite responsibilities and how different functional components of our service environment are grouped and managed:

  • Service line. A service line is a collection of services that defines a wide scope of business activity. There are multiple service lines that combine to provide the complete set of business functionality within Sales and Marketing.
  • Service. A service provides the automation of a specific business process that delivers value to customers. A service is supported by one or more apps or functional components within our environment.
Diagram that shows Relationship management at the top and connecting to the following below it: Customer planning,  Annuity lifecycle management,  Customer lifecycle management,  Account match service

Figure 1. An example of service line and service relationship

For example, we define Relationship Management as a service line in Sales and Marketing. Within the Relationship Management service line, there are several services that support that service line: customer planning, annuity lifecycle management, customer lifecycle management, and account match service. Each of these services are supported by one or more app environments that are examined and managed through the LiveSite review process, as well as a higher level LiveSite review for the Relationship Management service line.

Our functional capability is controlled at the lowest level by a large group of related apps. However, the modern engineering process is designed to place the focus on service-oriented presentation—putting the user experience and business effectiveness at the center of all discussions, decisions, measures, and practices. As such, our service lines and services are grouped according to business functionality.

Understanding LiveSite review teams and processes

LiveSite reviews are managed by a triad of roles that champion different aspects of the LiveSite review process. The triad that supports the LiveSite review process is critical to driving modern engineering throughout the Sales and Marketing environment.

A pie-shaped diagram with three pieces and arrows pointing clockwise between them. The three pieces are labeled: Service engineering,  Engineering program manager,  Software engineering

Figure 2. The LiveSite review triad

The three primary roles in the LiveSite review triad are Engineering Program Manager, Software Engineer, and Service Engineer. There are leads in these three areas for the overall service line, and then each service within the service line also has a set of three leads.

Service line triad responsibilities

The service line triad is responsible for the service line, which includes at a high level, each service within the service line. However, the service line triad’s focus is the overall business functionality and user experience for the service line as a complete solution.

Table 1. Service line triad roles and responsibilities



Engineering Program Manager

  • Schedule and run the bi-weekly LiveSite reviews.
  • Capture actions, notes, and key output from meetings.

Service Engineer

  • Drive the content and ensure data readiness.
  • Ensure the team has consistent access to tools.
  • Understand the end-to-end service line health.

Software Engineer

  • Ensure software engineering readiness and consistent service team awareness.
  • Understand the end-to-end engineering quality for the service line.

Service triad responsibilities

For each service within a service line, there’s a service triad that maintains responsibility for service-specific functionality and health. Service triads often deal with low-level and functional issues and bug fixes, but they’re still focused primarily on the state of the user experience and overall business functionality.

Table 2. Service triad roles and responsibilities



Engineering Program Manager

  • Ensure customer impact is accurate and action items are added to the appropriate backlog and prioritized.
  • Define what a severity 0 issue is for their service.

Service Engineer

  • Ensure LiveSite incidents are captured accurately and all required data is available for LiveSite review.
  • Ensure cloud costs are understood and be prepared to discuss.

Software Engineer

  • Ensure root causes of LiveSite incidents (LSIs) are understood and be prepared to discuss short-term and long-term solutions.
  • Ensure cloud costs are understood and be prepared to discuss.

Conducting LiveSite reviews

The LiveSite review meeting is designed to get the appropriate stakeholders in the same place discussing the most important aspects of their respective areas. LiveSite reviews are held for each service, service line, and for Sales and Marketing as a business unit.

LiveSite review agenda items

The LiveSite review for each area is different, but all LiveSite reviews use all or some of the following agenda items. These items are based on the team examining dashboard information from the appropriate tools and reviewing that information, according to the agenda item:

  • Discuss LiveSite incidents. Typically, this is the first item in any LiveSite review agenda. It enables our teams to identify important incidents early to raise awareness of events and discuss critical services interruptions. These conversations usually lead to clarifying the root cause and discussing mitigation or fixes. We also provide status updates for existing LSIs, review new LSIs, and establish action items from these. This is one of the most important agenda items. Teams discuss the following for each LSI:
    • The root cause of the incident.
    • The time to detect, which allows us to assess our monitoring capabilities.
    • The time to engage, which allows us to assess our ability to engage the right engineer to fix or mitigate the issue.
    • The time to resolve, which provides a measurement of how quickly incidents are being resolved.
  • Review universal metrics. Universal metrics primarily consist of cloud utilization data, compliance review status, and support incident trends. This agenda item is intended to make our teams aware of support trends and causes for support incidents. By performing cloud utilization reviews, we can assess services for opportunities to improve the effectiveness of our cloud architecture business efficiency and optimize our consumption costs in the cloud. Compliance reviews ensure that the team is aware of the latest compliance status so we can address areas of compliance risks and the impact of potential non-compliance.
  • Review Quality of Service metrics. Our Quality of Service (QoS) metrics provide specific measures that are relevant to the service and its ability to meet business or customer outcomes. Our teams regularly review QoS metrics to observe performance trends, identify deviations, and identify opportunities to modify a service in response to this data.
  • Review upcoming changes. Our teams review any upcoming changes to services, such as new code deployment or feature updates. Reviewing these changes also helps to identify potential areas of risk and ensures a broad awareness of changes that will be taking place.

LiveSite review process overview

The overall LiveSite review process for each audience is listed in the following table.

Table 3. The LiveSite review process by team




Agenda items

Service triads

Service triad


  • Discuss LSIs.
  • Review universal metrics.
  • Review QoS metrics. (Optional)
  • Review upcoming changes. (Optional)

Service line triads

Service line triad
Service triads
Cloud leads


  • Discuss LSIs.
  • Review universal metrics.
  • Review QoS metrics.
  • Review upcoming changes.

Sales and Marketing leadership team

Service line triads
Service triads
Cloud leads
Sales and Marketing leadership team


  • Discuss LSIs.
  • Review a summary of universal metrics.

Looking forward with LiveSite reviews

To better enable modern engineering practices, we’re aggressively embracing, driving, measuring, and rewarding the LiveSite review process within our teams. We’re starting to realize a LiveSite culture in our

  • We’ve improved the quality and consistency of existing conversations around service and service line provisioning.
  • We’ve moved from an application-oriented approach to a service offering–oriented focus. Now we can capture and address the end-to-end user experience.
  • We’ve increased accountability and engineering capacity across our teams.
  • It’s easier to plan, execute, and track service maturity and health.

LiveSite reviews also encourage the adoption of modern engineering practices and a DevOps approach to our teams. By enabling a DevOps approach, we’ve experienced the following benefits:

  • A more agile development environment:
    • Increased deployment frequency
    • Reduced lead time and faster reaction to changing business needs
    • Reduced manual effort for each release or pullback
  • A more reliable development environment:
    • Increased application and quality performance
    • Reduced user impact due to reduced failure rates
    • Reduced time to repair or mitigation

For more information

Microsoft IT


© 2017 Microsoft Corporation. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

You might also be interested in

Automating data protection with Azure Information Protection scanner
October 26, 2018

Automating data protection with Azure Information Protection scanner

Read case study
DevOps is sending engineering practices up in smoke
October 10, 2018

DevOps is sending engineering practices up in smoke

Read blog
Redesigning business applications at Microsoft using PowerApps
October 01, 2018

Redesigning business applications at Microsoft using PowerApps

Read case study
Creating consistent and engaging corporate apps with PowerApps
October 01, 2018

Creating consistent and engaging corporate apps with PowerApps

Read Article