Service Management Functions

Service Monitoring and Control

Updated: March 27, 2004

Executive Summary

The Service Monitoring and Control (SMC) service management function (SMF) is responsible for the real-time observation and alerting of health (identifiable characteristics indicating success or failure) conditions in an IT computing environment and, where appropriate, automatically correcting any service exceptions. SMC also gathers data that can be used by other SMFs to improve IT service delivery.

By adopting SMC processes, IT operations is better able to predict service failures and to increase their responsiveness to actual service incidents as they arise, thus minimizing business impact.

There are several underlying factors why effective service monitoring and control is increasingly important, these include:

•	Business Dependency. Organizations are increasingly reliant on IT infrastructure and IT services, and IT’s role in business delivery continues to expand. With this dependency, IT customers have greater exposure to IT failures, which often have severe impact to critical business functions.
•	Business Investment. Many organizations have realized the competitive advantage that IT provides and have made substantial investments in IT infrastructure. This forces a greater demand for demonstrable immediate return on investment (ROI) and the delivery of continuous long-term benefits.
•	Technology Complexity. As the IT Infrastructure continues to become larger and more distributed, it becomes more difficult to understand all the intricate requirements necessary to keep the IT infrastructure in good condition.
•	Business Change. Business-side changes have the potential to cascade to much larger tactical shifts in IT infrastructure. With business-side imperatives changing directions at a much faster pace, there is an increased demand to shorten IT technology delivery life cycles, increase architecture agility, and make better use of tools.

The key benefits of effective service monitoring and control are:

•	Early identification of actual and potential service breaches.
•	Rapid resolution of actual and potential service breaches through the use of automated corrective actions.
•	Minimized business impact of incidents and potential incidents.
•	Reduction in actual service breaches.
•	Availability of up-to-date infrastructure performance data.
•	Availability of up-to-date service level and operating level performance data.
•	Continued alignment of the monitoring performed and the business requirements.
•	Continued evolution of monitoring to meet business and technological change.
•	Maximized usage of management tools through effectively planned and integrated processes.

SMC provides the above benefits by carrying out the following six core processes, which are described in detail in the following sections:

•	Establish
•	Assess
•	Engage Software Development
•	Implement
•	Monitor
•	Control

Top of page

Introduction

Document Purpose

This guide provides detailed information about the Service Monitoring and Control service management function for organizations that have deployed, or are considering deploying, monitoring tools technologies in a data center or other type of enterprise computing environment.

This is one of the more than 21 SMFs (shown in figure 1) defined and described in Microsoft® Operations Framework (MOF). Every SMF within MOF benefits from some aspect of SMC because these functions are inherent to ongoing process improvement. This is especially true in the Operating Quadrant of the MOF Process Model where the SMFs are closely interrelated.

Figure 1. MOF Process Model and Related SMFs.
See full-sized image

The guide assumes that the reader is familiar with the intent, background, and fundamental concepts of MOF as well as the Microsoft technologies discussed. An overview of MOF and its companion, Microsoft Solutions Framework (MSF), is available in the Overview section of the MOF Service Management Function Library document. This overview also provides abstracts of each of the service management functions defined within MOF. Detailed information about the concepts and principles of each of the frameworks is also available in technical papers available at http://www.microsoft.com/mof.

What's New?

The SMC guidance contained in this document has been completely revised to include updated material based on new Microsoft technologies, MOF version 3.0, and, ITIL version 2.0. The SMC SMF now has more in-depth information for establishing an effective monitoring capability, including upfront preparation such as noise reduction. It also includes more complete information on run-time activities necessary to continuously optimize the monitoring process, its artifacts, and deliverables.

Feedback

Please direct questions and feedback about this SMF guide to msmfeed@microsoft.com.

Top of page

Service Monitoring and Control Overview

Goals and Objectives

The primary goal of service monitoring and control is to observe the health of IT services and initiate remedial actions to minimize the impact of service incidents and system events. The Service Monitoring and Control SMF provides the end-to-end monitoring processes that can used to monitor services or individual components.

Service monitoring and control also provides data for other service management functions so that they can optimize the performance of IT services. To achieve this, service monitoring and control provides core data on component or service trends and performance.

The successful implementation of service monitoring and control achieves the following objectives:

•	Improved overall availability of services.
•	Greater focus on service availability rather than component availability, resulting in a reduction in the number of SLA and OLA breaches.
•	An improved understanding of the components within the infrastructure that are responsible for the delivery of services.
•	A corresponding improvement in user satisfaction with the service received.
•	Quicker and more effective responses to service incidents.
•	A reduction or prevention of service incidents through the use of proactive remedial action.

The service monitoring and control function has both reactive and proactive aspects. The reactive aspects deal with incidents as and when they occur. The proactive aspects deal with potential service outages before they arise.

Scope

The Service Monitoring and Control SMF monitors and controls the entire production environment and works with the business, third parties, and the following SMFs to identify specific service monitoring and control requirements for their areas:

•	Capacity Management
•	Service Level Management
•	Availability Management
•	Directory Services Administration
•	Network Administration
•	Security Administration
•	Job Scheduling
•	Storage Management
•	Problem Management

Once the relevant requirements have been identified and agreed on with the SMC manager (see Chapter 5, “Roles and Responsibilities”), an ongoing program of proactive monitoring and controlling processes is implemented. These processes identify, control, and resolve IT infrastructure incidents and system events that may affect service delivery.

The service monitoring and control process interacts with the incident management process to ensure that data on automatically resolved faults is available to incident management and that any situations which cannot be immediately addressed using the automated control mechanism are directly forwarded to incident management for proper handling. This is of particular importance to the staff performing the incident management and problem management processes since more service incidents are generated using SMC than come directly from affected end users.

Service monitoring and control also deals with the suspension, in a timely and controlled manner, of the monitoring and control process for a particular configuration item or service. It specifically works with the Release Management and Change Management SMFs in order to minimize the impact to the business.

Any infrastructure that is deemed critical to the delivery of the end-to-end service should be monitored, usually to the component level. Some requirements, however, may prove impossible or impractical to meet, and so the initiator and the monitoring manager must agree on what is to be monitored before monitoring begins.

Service monitoring and control is the early warning system for the entire production environment. For this reason, it exerts a major influence over all areas of the IT operations organization and is critical to successful service provisioning.

Core Concepts

Readers should familiarize themselves with the following core concepts, which will be used throughout the SMC guide.

Service

Service Definition

In the context of the Service Monitoring and Control SMF, a service is a function that IT performs for or with the business. A service is defined from the business organization’s point of view. For example, e-mail and printing may each be considered a service, regardless of the number of lower-level components or configuration items (CIs) required to deliver the service to the end user.

In Microsoft Windows® technology terms, a service is a long-running application that executes in the background on the Windows operating system. These services typically perform working functions for other applications. In this SMF, this type of service will be referred to as a Windows service, an application service, or a server process.

Services in use within an organization are recorded in the service catalog. The service catalog is created and managed by the Service Level Management SMF. It includes a decomposition of services to its supporting infrastructure called service components.

Figure 2. Service component decomposition
See full-sized image

Service Components

Service components are configuration items (CIs) listed in the CMDB. These are atomic-level infrastructure elements that form the decomposition of a service. Service components that have instrumentation and can be used to determine health are observed and interrogated in order to assess the overall health of a service.

Microsoft has also developed the System Definition Model (SDM), which businesses can use to create a dynamic blueprint of an entire system. This blueprint can be created and manipulated with various software tools and is used to define system elements and capture data pertinent to development, deployment, and operations so that the data becomes relevant across the entire IT life cycle. For more information on the SDM and the Dynamic Systems Initiative (DSI), please refer to http://www.microsoft.com/DSI.

Instrumentation

Instrumentation is the mechanism that is used to expose the status of a component or application. In most cases, instrumentation is an afterthought for both packaged and custom applications, so it is not exposed properly. For example, events are frequently not actionable and lack context, or performance counters often do not show what users need in order to identity problems. In addition, few components or applications expose management interfaces that can be probed regularly to determine the status of that application.

Health Model

The Health Model defines what it means for a system to be healthy (operating within normal conditions) or unhealthy (failed or degraded) and the transitions in and out of such states. Good information on a system’s health is necessary for the maintenance and diagnosis of running systems. The contents of the Health Model become the basis for system events and instrumentation on which monitoring and automated recovery is built. All too often, system information is supplied in a developer-centric way, which does not help the administrator to know what is going on. Monitoring becomes unusable when this happens and real problems become lost. The Health Model seeks to determine what kinds of information should be provided and how the system or the administrator should respond to the information.

Users want to know at a glance if there is a problem in their systems. Many ask for a simple red/green indicator to identify a problem with an application or service, security, configuration, or resource. From this alert, they can then further investigate the affected machine or application. Users also want to know that when a condition is resolved or no longer true, the state should return to “OK.”

The Health Model has the following goals:

•	Document all management instrumentation exposed by an application or service.
•	Document all service health states and transitions that the application can experience when running.
•	Determine the instrumentation (events, traces, performance counters, and WMI objects/probes) necessary to detect, verify, diagnose, and recover from bad or degraded health states.
•	Document all dependencies, diagnostics steps, and possible recovery actions.
•	Identify which conditions will require intervention from an administrator.
•	Improve the model over time by incorporating feedback from customers, product support, and testing resources.

The Health Model is initially built from the management instrumentation exposed by an application. By analyzing this instrumentation and the system failure-modes, SMC can identify where the application lacks the proper instrumentation.

For more information on topics surrounding the Health Model, please refer to the Design for Operations white paper at http://www.microsoft.com/windowsserver2003/techinfo/overview/designops.mspx.

Health Specification

A Health Model is documented by development teams for internally developed software. It is also documented by application teams for software that has been heavily customized and extended.

A Health Specification is a set of documented information that is identical to the Health Model. However, this material is specifically created by IT operations (such as the SMC staff) and is designed for commercial off-the-shelf (COTS) software and other purchased service components.

Customer Impact

Having a strong understanding of service health allows instrumentation to be aligned with customer needs. Coupled with the monitoring and diagnostic infrastructures, this will allow administrators to quickly obtain the information appropriate to their circumstances. The guidelines contained in this guide on management instrumentation and documentation will ensure that the structured information delivered to the administrator is meaningful and that the appropriate actions are clear. These improvements will support prescriptive guidance, automated monitoring, and troubleshooting, which, in turn, will simplify data center operations, reduce help desk support time, and lower operational costs.

The more complete and accurate an application’s model is, the fewer the support escalations that will be needed. This is simply because the known possible failures and corrective actions have already been described. With more automation, customers can manage a larger number of computers per operator with higher uptime.

In addition, the modeling documents created can be directly used in producing deployment, operations, and prescriptive guidance documents for customers when the product is released. (Please refer to the section on the Health Model for further information.)

Key Definitions

The following terms are used in the Service Monitoring and Control SMF. The definitions given here are used solely within the context of the SMC SMF.

•	Action/Response. A script, program, command, application start, or any other remedial response that is required. Typical actions are automated, operator-initiated, or operator-driven. Actions are generally defined to correct a system event that represents an incident within the IT infrastructure. However, actions can also be used to perform daily tasks, such as starting an application every day on the same node.
•	Alert. A notification that an operational event requiring attention may have occurred. An alert is generated when monitoring tools and procedures detect that something has happened (at the service, service function, or component level).
•	Control. Automated response or collection of responses. The three types of controls are diagnostic, notification, and interoperability.
•	Event. An occurrence within the IT environment (usually an incident) detected by a monitoring tool or an application that is consistent with predefined threshold values (within, exceeding, or falling below) that is deemed to require some sort of response or, at a minimum, is worth recording for future consideration.
•	Reporting. The collection, production, and distribution of an agreed-on level and quality of service information (for example, for use in capacity, availability, and service level management).
•	Resolution completion. The point in the control process where manual/automatic action has been taken and all recording and incident management actions have been successfully completed.
•	Rules. A predetermined policy that describes the provider (the source of data), the criteria (used to identify a matching condition), and the response (the execution of an action).
•	SMC Tool Agent. A component of the SMC tool, which typically resides on the managed node and is responsible for functions such as capturing events and executing responses. In some cases, SMC tools can also have agentless configurations.
•	Threshold/criteria. As used in the system and network management industry, a threshold is a configurable value above which something is true and below which it is not. Thresholds are used to denote predetermined levels. When thresholds are exceeded, actions may occur.

Top of page

Processes and Activities

Implementation of the SMC SMF should follow the Microsoft Solutions Framework (MSF) life cycle for vision/scope or justification, planning, development, test or stabilization, and release. For complete project-focused implementation, organizations should use MSF guidance for SMC. This implementation should include iterative deployment, limited trials and pilot environments, and consistent use of the MSF Risk Management Discipline.

As a result of its monitoring and controlling activities, SMC enables IT service provisioning by monitoring services as documented in agreed-on service level agreements or other agreed-on or predicted business requirements. Monitoring is also performed against the service components of operating level agreements (OLAs) and third-party contracts that underpin agreed-on SLAs, where these are in place.

After SMC gathers, filters, and agrees on overall service requirements with the business, it then works with IT operations peers in service level management to identify the IT services and infrastructure components across each layer of the enterprise that deliver these requirements.

In order to gather the overall service requirements from the business, SLAs will be referenced, as well as composite OLAs and underpinning contracts as needed. The component level technical requirements for other SMFs are also agreed on in parallel. In many instances these will mirror the business requirements, but many technology-specific requirements, data collection, and storage requirements that require monitoring will also be identified. The layers that need monitoring generally include:

•	Application
•	Middleware
•	Operating system
•	Hardware
•	Networking and access
•	Facilities and environmentals

The IT infrastructure that delivers the agreed-on services is identified and decomposed into infrastructure components (that is, configuration items) that deliver each service. If a configuration management database (CMDB) is available, it can be used to identify the configuration items.

The attributes of each configuration item that need monitoring are also identified (for example, disk space on a server or memory usage) and a definition of what constitutes a healthy state is also established for each configuration item. The actions to be taken or the rules to be followed in the event that a criterion is met or a threshold exceeded are also defined.

Performance of the day-to-day monitoring and control process can begin only after these criteria or thresholds and rules have been configured within the monitoring toolset and then deployed and reviewed. These are critical to the successful operation of the process and to the delivery of high-availability services.

Continuous day-to-day monitoring against these set criteria identifies real incidents and system events across the IT infrastructure. When an incident or system event is highlighted, remedial action (that is, automated response) is started to ensure that agreed-on service levels continue to be met.

To fully adopt SMC, an IT operations organization will follow 6 core processes (shown in Figure 3):

•	Establish
•	Assess
•	Engage Software Development
•	Implement
•	Monitor
•	Control

Each of these processes is described in detail in the following sections.

Figure 3. SMC core processes
See full-sized image

Establish

Overview

The Establish process collects, develops, and implements the foundational components of the Service Monitoring and Control SMF. The Establish process focuses on the initial setup of the SMC capabilities and is not part of the run-time workflow.

Figure 4. Main activities of the Establish process

The Establish process is composed of three main activity areas:

•	Prepare SMC Data. The formalization of health information with the collaboration of other SMFs and line organizations.
•	Prepare Run-time Data. The establishment of SMC processes and roles.
•	Prepare SMC Tools. The identification and implementation of critical management technologies for SMC.

It is important for organizations to carefully execute all the steps in the Establish process. Organizations may go through multiple iterations of the Establish workflow throughout the MSF life cycle in order to achieve optimal process functionality and to fully experience the benefits from the investment in monitoring tools and technologies.

This Establish process can be used for companies that currently do not have a service monitoring and control function/process in place, or it can be used to update and improve an existing SMC management function.

Planning Considerations

As shown in the following diagram, the three main activities in the Establish process can be performed both in sequence and in parallel with each other. This increases the efficiency of implementation and also saves time.

Figure 5. The Establish activities and subactivities sequence
See full-sized image

The performance of some subactivities in the Establish process is dependent upon other subactivities being carried out as prerequisites. Examples of these dependencies are described below:

•	Prepare SMC Data: Conduct SMC Enterprise Analysis. This subactivity, in which resources are assigned and identified, should be carried out after the Prepare SMC Run-time Process: Formalize Roles subactivity.
•	Prepare Run-Time Process: Formalize Roles. This subactivity should be executed after preliminary information has been captured by the Prepare SMC Data: Collect SMC Prerequisite Material subactivity. When roles are being formalized and the base staff is being identified, the assessment data from the parallel activity will help to determine the number of personnel required, as well as their overall capabilities.
•	Prepare Run-Time Process: Adopt SMC Process. This subactivity requires that all material from the Prepare SMC Data activity, especially from the Collect SMC Prerequisite Material and Conduct SMC Enterprise Analysis subactivities, be completed prior to starting. This subactivity also requires integration based on the design created during the Prepare SMC Tools activity, especially the Create Management Architecture subactivity.
•	Prepare SMC Tools: Formalize Tool Requirements. This subactivity should be executed after information has been captured by the Prepare SMC Data: Collect SMC Prerequisite Material, Conduct SMC Enterprise Analysis, and the core components of the Develop Health Definition subactivities have been collected. This subactivity should involve any individuals assigned from the Prepare Run-Time Process: Formalize Roles subactivity.
•	Prepare SMC Tools: Create Management Architecture and Initialize SMC Tools. These subactivities should not be conducted until almost all of the core information from the Establish process has been collected.

Establish Process Activities

The following sections provide further details about each of the activities in the Establish process flow.

Prepare SMC Data

The objective of the Prepare SMC Data activity is to collect data used in all aspects of SMC, and to create detailed health specifications and models on the service components that need to be monitored and controlled by the SMC run-time process and tools. To effectively develop this material, a comprehensive review process must take place, as well as collaboration with other IT functions.

Collect SMC Prerequisite Material

Materials that aid with the implementation and optimization of service monitoring and control must be collected, categorized, and made accessible. A good place to start is with the key pieces of information that are generated or managed by other MOF SMFs.

•	Service Level Agreements (SLAs), Operating Level Agreements (OLAs), and Underpinning Contracts (UCs). These documents define the requirements and expected behaviors of IT services. This information typically includes targets on availability, continuity, and capacity; service hours; escalation; service level objectives; and associated metrics. This information is useful for SMC since it becomes the basis for monitoring thresholds. These documents also define the principal parameters to be used when reacting to exception conditions. These documents typically include information about escalation steps, hours of operation, and notification practices and will be used in SMC’s Control process. Services and service conditions not listed in these agreements are typically not monitored by SMC. SLAs, OLAs, and UCs are created by the Service Level Management SMF. Further information about these documents is available at http://www.microsoft.com/mof.
•	Service Catalog. A service catalog hierarchically organizes an IT service (as defined in an SLA) into its requisite service components. Service components can be other services but, at an atomic level, are configuration items (CIs). This is important to SMC because actual monitoring is performed at the service component or CI level. Associating the CI or infrastructure being monitored, such as a server or application, to its parent service/s is the role of this document.
•	Problem Management Information. Knowledge generated by the Problem Management SMF is important to SMC. This body of knowledge, such as the Known Problem Base, is a collection of current and historical problems that have been investigated by problem management and includes a root cause analysis and possible workarounds. This material is useful to SMC especially when developing automated responses in the Control process.
•	Configuration Management Database (CMDB). The CMDB provides a single source of information about the components of the IT environment. The CMDB is created and managed by the Configuration Management SMF. This information is especially useful when developing class categorization and tools-specific rules for SMC infrastructure targets.
•	Incident Management and Service Desk Records. Knowledge generated by the Incident Management and Service Desk SMFs is typically presented in the form of a knowledge base. This information usually contains historical records of past incidents, categorizations, prioritizations, initial diagnostics, possible escalation steps, and eventual closure. This material is especially useful to SMC when developing health standards, defining roles, and developing management tools architecture.
•	Availability, Continuity, and Capacity Management Information. The SMFs in the Optimizing Quadrant—especially Availability Management, Continuity Management, and Capacity Management—generate important material including the methods for analysis and response to specific service level breaches. This material should be collected along with such other diagnostic models as dependency chain mappings, availability plans, and continuity plans. This information is especially useful when developing event rules.
•	Other Data Sources. Information not necessarily associated to specific SMFs can be collected from key individuals responsible for tracking infrastructure information. These individuals include network administrators, security administrators, systems architects, tools engineers, and system integration engineers.

Collaborate with Other SMFs .

The process of collecting material from other SMFs provides a good opportunity to educate other service managers about the Service Monitoring and Control SMF and to explain the needs of the SMC SMF in terms of prerequisite materials. SMF materials that commonly need to be updated or improved for SMC include:

Conduct SMC Enterprise Analysis

After the SMC prerequisite materials have been collected, a detailed survey and analysis should be made of the infrastructure and tools, management processes, and organizational structures and locations. This survey should validate the information that was collected from the other SMFs as well as increase the knowledge about the environment that will be managed by service monitoring and control.

Analyze IT Infrastructure and Service Catalog Decomposition.

The SMC team should have a clear understanding of IT infrastructure’s composition, especially the components that make up business-critical services. During this activity, any additional findings not already documented in the CMDB may be added with the coordination of configuration management. Key information that affects SMC architecture, design, and tools selection includes:

Analyze Infrastructure Management and Tools.

Review the current process used to determine the short-interval (or real-time) health of the environment. An organization may not have a stand-alone process for this determination. Instead, it may be using an extended version of availability management and service level management monitoring. These current processes may provide additional information to help increase the successful adoption of SMC processes.

Analyze Organizational Design – Physical and Logical Distribution.

A complete survey must be made of the organizational design and distribution of supporting IT staff. This information will be used in designing the SMC process adoption and, more importantly, the SMC tool architecture—especially the placement of consoles and servers and the forwarding and routing of events. For example, a centralized organizational model might require that alerts be forwarded to a centralized location where operators will be constantly available for monitoring the console. For more detail on organizational model considerations, please refer to the MSM Management Architecture Guide located at http://www.microsoft.com/technet/solutionaccelerators/techguide/msm/winsrvmg/mgmtarch/20/mgmtarc1.mspx.

Collaborate with Key IT Line Organizations.

During the Conduct SMC Enterprise Analysis activities, the SMC team should begin to establish a partnership with key IT line organizations. It is important to create these relationships to make sure that products from these teams will be addressable for monitoring and control within SMC’s capabilities. The Establish: Prepare Run-Time Process: Formalize External Interactions activity will provide detailed information on furthering this relationship. The two most important groups to collaborate with include:

Develop Taxonomy Standards

Taxonomy standards provide a common means for understanding health levels across all services managed with SMC. These standards may change and improve as additional infrastructure and tools are added under SMC’s scope. For a detailed health model and definitions for the Windows operating system, please refer to the Design for Operations white paper at http://www.microsoft.com/windowsserver2003/techinfo/overview/designops.mspx.

Classification Standards.

Classification standards are health attribute classes that categorize event-related information. Whereas incident management has a process to determine the classification of incidents as they occur, SMC’s classification is predetermined for each event that is exposed by instrumentation. Incident management’s sorting and identification process may help to define SMC’s standard. Classification standards are important to SMC so that events and alerts are handled as effectively as possible on the basis of membership.

Prioritization Standards.

Prioritization standards are health attribute classes and types that define the taxonomy for urgency and impact. Whereas incident management has an evaluation process to determine the priority of incidents as they occur (on-demand), SMC’s prioritization is predetermined for each event that is exposed by instrumentation. Incident management may already have an incident priority coding standard that SMC can adopt with minor tuning. Prioritization standards are important to SMC so that events and alerts are handled as effectively as possible on the basis of its membership to a specific taxonomy. This upfront definition is also critical so that events and alerts are uniformly classified. In other words, a level 1 designation for an event in application A and level 1 designation for an event in application B should both be equal in value or importance.

Define Health Specification and Health Model

All the information collected and analyzed within the Prepare SMC Data activities is used to create a Health Specification for each service component. A Health Specification (also called a Health Model for internally developed software) documents significant information used for monitoring a specific component. This may include all actionable events, event exposure and behavior, and instrumentation protocols and behavior. Ideally, this information is directly codified into a language or configuration dataset that may be used by SMC tools. It is important to define taxonomy standards prior to documenting Health Specifications so that the specific attribute values related to classification and prioritization levels align to a common reference.

There are two types of Health Specifications:

•	Class-level. Creates specifications based on a class of common infrastructure or service components. In a large organization with a significant online presence using similar hardware and applications, an example may be a Health Specification for Web servers.
•	Override-level. Creates specifications based on individual infrastructure or service components that fall outside of a class grouping. In a large organization consisting mostly of databases using Microsoft SQL Server™, an example may be a Health Specification for a specific host running Microsoft Access.

For more information on how to create a Health Specification or Health Model, please refer to the "Steps in Building a Health Model" activity in the Engage Software Development process of this SMF guide.

Prepare Run-Time Data

The Prepare Run-Time Process activity includes key activities for the implementation of SMC’s run-time process.

The successful implementation of the SMC process requires sustained executive commitment, training for SMC staff, and ongoing review, mentoring, and process optimization.

•	Executive Commitment. Sustained executive commitment to SMC must be established as early as possible—for example, during the vision/scope phase of SMC’s project life cycle. Full SMC implementation will vary in length based on the size and diversity of the infrastructure and services being monitored, along with the desired level of automation for the Control process. Executive sponsors are needed to provide high-level advocacy, process authority, and funding; to arbitrate organizational disagreements related to SMC; and to enforce such standards as new release criteria as defined in the Engage Software Development process. For example, new release criteria may state that new applications being accepted by IT operations must include a Health Model as part of the release package.
•	Staff Training. SMC staff and related personnel should be familiar with fundamental MOF concepts and have proficiency with the SMC processes. Effective training will accelerate the adoption of SMC by the organization, and the new knowledge and skills gained by the staff will reduce SMC process issues.
•	On-going Review, Mentoring, and Process Optimization. The initial SMC implementation is based on the point-in-time conditions of a given environment, which will invariably change and evolve. Without a commitment to pursue ongoing improvement, an SMC SMF implementation will eventually break down and become ineffective.

Formalize Roles

In this subactivity of Prepare Run-Time Process, the SMC roles for the organization, including any minor company-specific nuances, are formally defined. Many organizations also use the role name as a job position or title. An example of a company-specific nuance may be the addition of numbering associated with pay or seniority level, such as SMC Operator 1 or SMC Operator 3. For a complete listing of standard SMC roles including their duties, please refer to Chapter 5, "Roles and Responsibilities."

Where available, key individuals should be assigned SMC roles and become immediately involved in the Establish activities. This will help foster organizational learning and maintain continuity.

Initially, individuals may be assigned multiple roles; but as the SMC scope and capabilities expand, the roles may be more narrowly defined and assigned to single individuals.

Formalize External Interactions

Prior to officially starting the SMC capability, the principal external interactions should be formalized, along with the establishment of clear and coordinated lines of communication. It is important to formalize external interactions in order to reduce errors and omissions resulting from miscommunication and misunderstanding. This also helps in controlling cross-SMF request volumes and makes responses more predictable.

Outbound Interactions.

The following outbound interactions summarize the handoffs or requests from SMC to other teams.

Inbound Interactions.

The following inbound interactions summarize the handoffs or requests from other teams to SMC.

Adopt SMC Process

When formally adopting the SMC process for an organization, consider the fact that MOF is a framework as opposed to a strict methodology. This means it is adaptable and can be modeled to accommodate company and even organization-level specific needs. MOF’s integrity as a best practice descriptive guidance is maintained as long as core elements are preserved; terms, their scope, and definitions are unchanged; and pre-established measurement for maturity is used. Any deviation from the base SMC MOF model should enhance the function, not complicate it. Adoption tuning may be used to address geographic distribution and industry-specific legislative requirements.

When initiating the SMC SMF processes, ensure that process controls and the KPIs are established for monitoring the performance of the SMC process itself. See Appendix B, "Key Performance Indicators," for more details.

Prepare SMC Tools

The Prepare SMC Tools process flow activity focuses on key activities that should be executed in order to establish effective SMC technology and automation. Tools and technology are important to the SMC SMF since they enable repeatable, real-time observation, processing of events, and automated response.

Formalize Tool Requirements

There are many factors to take into consideration when selecting the principal tool used for SMC. Information collected and analyzed in the Establish: Prepare SMC Data process flow activity should be incorporated to build specific selection criteria. Other SMF teams should be involved in defining these requirements, along with input from software development and application teams. SMC tool requirements must be concrete and ideally contain measurable objective criteria.

The following list of considerations may be used in developing SMC tool requirements and selection criteria:

•

Performance. SMC tool requirements should address the needs for appropriate levels of performance to ensure low alert latency.

•

High-Availability Options. SMC tool requirements should address the needs for high-availability options such as clustering, failover, and synchronization for failover.

•

Tool Architecture. SMC tool requirements should address the needs for appropriate tools architecture so that the data sources and protocols are supported, the method of collection and threshold calculation as specified in an SLA’s SLO and metrics can be applied, and have robustness for anomalies like a spike in network latency.

•

Event Routing and Forwarding. In organizations that have a geographically distributed SMC capability or have multiple consumers of console data, then the SMC tool requirements should address the needs for effective event routing and forwarding.

•

Autodiscovery. SMC tool requirements should address the needs for automatically discovering new managed nodes, infrastructure change, and monitoring targets.

•

Deployment. SMC tool requirements should address the needs for simple yet effective rules and agent deployment.

•

Network Adaptability. SMC tool requirements should address the needs for network adaptability in order to facilitate complex network topologies, routing protocols, and security segmentation.

•

Lightweight. SMC tool requirements should address the needs for a lightweight monitoring agent in order to minimize the impact of SMC on the infrastructure being monitored.

•

Scalability. SMC tool requirements should address the needs for scalability, such as the number of managed objects per server and the number of simultaneous events it can process at a given time. At a minimum, the tool must be able to address short-term infrastructure growth and conditions.

•

Interoperability. SMC tool requirements should address the needs for interoperability, such as integration with other management tools, and such processes as trouble ticketing

•

Reporting. SMC tool requirements should address the needs for reporting and offline data storage.

•

Data Repository. SMC tool requirements should address the needs for knowledge base and/or SMC data repository facilities.

•

Vendor Background. SMC tool requirements should address the needs for stable vendor support and that a commitment is present to correct tool issues through updates and patches.

•

Security. SMC tool requirements should address the needs for security, such as granular levels of access and role-based authorization, and safe alert transport and storage.

•

Pricing. SMC tool requirements should address the needs for pricing with evaluation of the overall total cost of ownership (TCO).

•

Dependencies. SMC tool requirements should address specific infrastructure and configuration dependencies for the tool itself. This is a very important and often overlooked consideration.

Here are examples of dependencies based on directory services:

•	Most organizations want to lock their directory services schema. A conflict may be caused if the SMC tool needs to extend this schema in order to add its own attributes.
•	If organizations do not have directory services and the SMC tool needs this for authentication or deployment, then the tool will not work correctly.

Design Management and Tools Architecture

Using a combination of all the knowledge that has been compiled through the Establish process flow activities, an initial management architecture should be created. This architecture is manifested typically in large graphical representations with supporting detail in separate documentation.

This architecture should include all core decisions on the following key areas:

•	Physical Infrastructure. Geographic and physical layout, failover, and clustering.
•	Network Topology. Network paths and logical routes.
•	Event Flow. Event format, flow, and forwarding.
•	Storage. Accessible data for reporting.
•	Console and Workflow. User and role interaction.
•	Security. Access control and secure transport and verification.

Initialize SMC Tools

Actual implementation of tools should follow the MSF life cycle. This implementation process should include the initial deployment of the tool in an isolated lab, then the pilot environment where it is iteratively improved, and then the release into production.

A typical implementation will involve the following activities:

•	Install operational database and SMC tool servers and application.
•	Develop monitoring rules for identified targets.
•	Develop monitoring and control scripts for identified targets.
•	Deploy agents.
•	Deploy rules and scripts.
•	Test and validate.
•	Optimize.

Noise Reduction

A process should be adopted to reduce the initial noise levels, which are caused by a barrage of alerts in the SMC tool. Keep in mind that there may be a barrage of legitimate alerts once a more effective monitoring process and toolset is in place. Issues that were previously undiscovered may surface and should be addressed with problem management. Noise reduction is an iterative process that includes the following high-level activities:

•

Initial review of Health Model, Health Specifications, and SMC tool rules. The SMC team as well as relevant subject matter experts review the detailed material and compile potential areas of improvement to be shared with the software development or application teams.

•

Isolated lab testing. After the Health Model and Health Specifications have been translated into a collection of rules, this material, any companion data collectors, and control scripts are checked to make sure that they do not introduce any adverse performance impacts to the SMC tool or managed node. Performance impacts can be caused by issues such as memory leaks and stale processes. During this test pass, the following performance counters are recorded:

•	Process
•	Processor
•	Disk
•	Network

•

Pre-production testing. Once the rules, companion data collectors, and control scripts have been checked in the isolated environment, they should then be promoted into a pre-production test environment where actual daily activities are performed on the infrastructure. An example of a pre-production environment can include a limited deployment to a pilot set or, where possible, carefully coordinated production systems that send events to both the production SMC tool and to a test SMC tool configuration. All the alerts generated in this testing should be forwarded to a common location, such as an e-mail distribution group, and subject matter experts can then subscribe to this alias. The alerts are then triaged and further diagnosis is made to reduce the alert count.

•

Reduction of alert volumes. Reduction of monitored events and alert volumes should be performed through a filtering and evaluation of validity and actionability:

•	Validity. Assessment of an alert to make sure that it indicates the actual problem that was experienced. An alert is valid if it accurately reports the state of the component, its functionality, and/or overall service. Invalid alerts are those that inaccurately report information.
•	Actionability. Assessment of the completeness of the alert’s information in order to perform corrective action. Key attributes of the alert should be clear, unique, and may also be supplemented with a knowledge base article. An alert is actionable if the alert text and related information provide clear steps to resolve the issue.

The effectiveness of this reduction and additional suppression can be best measured using the Alert to Ticket ratio.

•	1 to 1. For every alert that is generated by the processing rule, it is estimated that one ticket will also be created. This is the goal and most ideal situation.
•	2 to 1. For every two alerts generated by the processing rule, it is estimated that one ticket will also be created. A ratio of less than 2 to 1 is often used as a target for highly mature SMC implementations.
•	Multiple to 1. This is usually considered beyond acceptable limits. Alerting should be disabled or better suppression and correlation should be implemented. However, there may be unique instances where this is unavoidable such as an unresolved recurrent critical issue. For these unique situations, the alert should be kept for further analysis.

Assess

Overview

Assess is the second major process in SMC and is responsible for the review and analysis of current conditions in order to make necessary adjustments to any aspect of the SMC function. Assess is similar to the Establish process’ initial analysis because of the front-end holistic review that takes place in both. It differs because the goal of Establish’s analysis is for implementing the foundational components of SMC, while Assess is concerned about the ongoing analysis for change and optimization within the run-time process group.

The approach to executing the Assess process flow is holistic. Although listed as a sequence, it should be seen as a global, or centralized, evaluation.

Figure 6. Main activities of the Assess process

Assess should be performed when a new service component is introduced; when there is a change to the infrastructure, CIs, SLA, or service catalog; after specific Control actions have occurred, and at a predefined interval to review monitoring.

It is important to continuously assess in order to understand the impacts of different variables and to develop the necessary strategies that will be implemented in the Implement process.

Formal tests and validation activities within the run-time process can also be conducted as needed in the Assess process.

The activities in assess should use all available automation—for example, autodiscovery, tools, and scripted procedures.

Assess Process Activities

Review SMC Requests

For the Review SMC Requests activities, all analysis is performed in the Assess process and execution or actions are performed in the Implement process.

Examples of SMC requests include:

•

Suspend Monitoring. This request is typically generated for the temporary suppression of alerts for a given timeframe. The Problem Management, Change Management, and Release Management SMFs typically generate this request, as well as special cases and conditions as defined in the SLA.

Patch management operations may also request a suspension of monitoring during the patching process.

•

Restart Monitoring. This request is typically generated when problems are identified that are related to the SMC agent or are affecting the system. Other situations include patches that have been applied to the system, which requires rebooting, or the monitoring agent must be rebooted or refreshed. Restart monitoring requests are generated from problem management, change and release management, as well as special cases and conditions defined in the SLA.

•

Start Monitoring (New/Change). The start monitoring request is generated from the Change Management and Release Management SMFs. This involves defining a Health Specification or Health Model and implementing the agent, rules, scripts, and configuration. The analysis portion of this request, specifically the Health Specification or Health Model as well as configuration parameters, is performed in the Assess process. All other deployment and implementation specifics are handled in the Implement process. These activities should be managed though the MSF life cycle as part of normal application deployment.

•

Change Monitoring Parameters. The change monitoring parameters request is generated from teams in IT operations and passes through change management for routine changes or through problem management during a break/fix situation. Key parameters involved in monitoring changes include:

•	Providers
•	Responses
•	Thresholds
•	Frequency (Suppression)
•	Rule Attribute (such as Rule Name)

Examples of change monitoring parameters requests include:

•	Threshold Change. Changing a specific threshold that determines when alerts are triggered.
•	Frequency Change. Changing the sampling interval that the SMC tool polls the CI.
•	Rule Change. Changes to individual rule sets that define the processing of an event. This could also include the optimization in changing the processing categories such as consolidate to filter and filter to collection.

•

Removal of Monitoring. The removal of a monitoring request is generated from many teams in IT operations and passes through change management. This request is typically associated with the decommissioning of infrastructure components.

Review Data from Other SMFs

Artifacts from other SMFs may have a direct impact on SMC. Although changes to key documents are promoted through change and release management, internal SMF processes may not be subject to change and release management on the basis of impact and policy. The SMC Assess process should continuously evaluate the following SMF data:

•

SLA and Service Catalog. Changes to the SLA have significant importance to SMC in relation to monitoring scope and inclusion (determining whether a service should be monitored) and service components (determining the infrastructure that should be monitored and at what level).

•

Capacity and Workforce Plans. Changes to these plans may impact SMC’s ability to deliver its services. SMC should have adequate resource capacity, including staffing.

The Assess process should also check the reporting and data volumes, especially if other SMFs are running as-needed reports and affecting the SMC tools. Teams who are customers of SMC data should not perform any reporting function using the SMC tool operational database. These customers should use external data sources provided by SMC so that they do not adversely impact the production systems.

It is important to remember that SMC does not create reports; this is the responsibility of other SMFs. For example, SMC is not responsible for the creation of an availability report. This is explicitly the role of the Availability Management SMF, although SMC may provide the empirical data used for this availability report. The SMC tool may have reporting capability; however, this functionality may be assigned to the respective team that has responsibility for it.

•

Operating Quadrant Conditions. Any changes to the data managed by these SMFs in the Operating Quadrant may directly impact SMC.

•	Security Administration SMF. Changes in security policy, access control, authentication, and authorization may require changes to the architecture of SMC tools. For example, when a Control procedure is executed, it typically runs under predefined user and group contexts. If there are any changes to this user and group, it may cause the procedure to fail; or worse, it may execute in unpredictable ways.
•	Directory Services Administration SMF. Changes in directory services may require changes to the architecture of SMC tools. For example, if the SMC tool relies on the directory to store and deploy configuration data, changes to the directory’s schema and reference model may disable tool capabilities.
•	Network Administration SMF. Changes in the network may require changes to the architecture of SMC tools. For example, if new routes are added to the network that changes the path of SMC messages, saturation of that segment can cause SMC tools to be unable to receive their important alerts.

Review Monitoring and Control

Conditions of SMC-specific components should also be reviewed and assessed. This is important in order to deliver the agreed-upon levels of monitoring and control capability as well as support to the other SMFs that rely heavily on SMC services. The following activities describe the review of various SMC-specific components.

Assess SMC Tool Components

•	Agent Condition. The agent collects service component events and performs preliminary filtering and, if defined within rules, raises an alert that is sent to the SMC tool server. The agent also facilitates the execution of Control procedures on the managed node. Consistent operation of the agent is critical to SMC and should be checked frequently. Make sure that the agent is providing accurate polled checking (also called a heart beat) and that it is operational and functioning normally.
•	Server Condition. The server is a core processor of events and alerts and performs deeper correlation prior to creating notification using e-mail or page, or through the console. The server should be assessed for proper operation to make sure that no serious faults have occurred and that all tool subsystems are functioning normally. Also check to make sure that the server is receiving data from agents. If no alerts are being received, it indicates that either the environment and all the services are in perfect condition (no faults) or, more commonly, that there is a failure in the SMC tool.
•	Database and Reporting Condition. The tool database is the repository of events and alerts and their metadata, such as receipt time, source, and state. The database and its associated SMC tool reporting functions should be checked frequently to make sure that all subsystems are functioning normally, data has not been corrupted, cascading errors have not been transmitted to different areas, and necessary resources are available such as tablespaces.

Review SMC Analysis Schedule

The frequency of scheduled optimization analysis should decrease over time. This schedule for periodically assessing the monitoring of a specific service decreases because SMC will become more stable and increase in its optimization and ability to reuse its process artifacts.

Analyze Monitoring and Response Rules

The rules implemented in the SMC tool should be continuously evaluated for optimization. Ideally, alerts that are presented to operators are a true indication of a service issue and map directly to a specific actionable response. All other alerts have either been suppressed, removed from SMC, or automatically resolved using Control mechanisms.

•

Generate SMC Reports. Reports should be generated on SMC indicators on a regular basis. The frequency for performing this is determined by the analysis schedule.

•

Analyze SMC Statistics. The following statistics should be reviewed to understand the performance of SMC as well as to identify opportunities for improvement. Each value is mapped over predefined timeframes (such as daily/weekly/monthly).

•	Number of Alerts Generated. As the Health Specification or Health Models are refined and rules are optimized, the mean of this count should significantly reduce.
•	Top 10 Alerts by System. This count should be reviewed to determine the alerts and events that should be evaluated for optimization. This statistic should also be analyzed to see if certain problems recur and may be chronic. This information should be given to problem management and if the solution is consistent each time, an automated Control response may be developed.
•	Alert to Ticket Ratio. This is a key statistic that indicates the quality of SMC alerts. The goal is to achieve a 1:1 ratio between alerts and tickets. This indicates that each alert is valid and has a well-defined and well-documented problem set associated with it.
•	Mean Time to Detection (such as Alert Latency). This statistic should dramatically improve with the implementation of effective SMC tools. Alert latency is the measurement of the delay from when a condition occurs to when an alert is raised. Ideally, this value is as low as possible.
•	Number of Tickets with No Alerts. A high count of tickets with no alerts is an indication that monitoring missed critical events. This statistic can be used as a starting point for improving instrumentation and rules.
•	Number of Events per Alert. As rules and correlation improve, this count should increase. Often, multiple events are triggered; however, there is typically only one true source of issue. A high events per alert count may also indicate opportunities for reducing the number of exposed events.
•	Number of Invalid Alerts. Alerts that are generated with incorrect fault determination should be carefully reviewed and corrected. The number of invalid alerts may increase during the initial deployment of new infrastructure components and services; however, it should drastically decrease with better rules and event filtering.
•	Mean Time to Repair. This statistic is typically used in capacity and availability management; however, SMC should analyze problems that were corrected using SMC’s Control. This metric measures the effectiveness of the automated response from this process. This value should decrease as more situations are handled by SMC automation.

Obtain Feedback from Monitoring Consumers

On a weekly or biweekly basis, interview SMC data consumers (console operators, recipients of auto tickets, and other notified parties) for anecdotal information. The objective of this activity is to capture opportunities to improve the quality of SMC work products through observed behaviors that may not necessarily be reviewed through formalized metrics.

Engage Software Development

Overview

The purpose of the Engage Software Development process workflow activities is to give operational guidance to internal software development and application teams for creating applications that are more operations-ready and monitoring-friendly. This guidance will improve the overall availability and reliability of their applications.

Figure 7. Main activities of the Engage Software Development process

Engage Software Development Process Activities

The following sections provide further details about each of the activities in the Engage Software Development process.

Collaborate on Operations Requirements

Infuse SMC Findings for Application Improvement

SMC should provide feedback to internal software development and application teams in order to improve overall manageability, especially with the current version of the application in production so as to influence subsequent versions that are being developed.

This activity includes the following key communications:

•	Validity of Instrumentation. Provide feedback on the validity of events, with the potential to remove those that refer to conditions that do not truly exist.
•	Reliability and Consistency of Instrumentation. Provide feedback on the reliability and consistency of the instrumentation for potential correction and improvement.
•	Actionability of Instrumentation. Provide feedback on the actionability of instrumentation, specifically the use of name and description fields, as well as making sure to retain the unique ID numbering processes, and minimize use of overloaded attribute values.
•	Completeness and Accuracy of Instrumentation. Provide feedback on the completeness of information contained in the alerts and events, as well as the accuracy and compliance to taxonomy standards.
•	Initial Prioritization. Provide feedback on the initial prioritization of instrumentation. For example, the software development team may have considered a specific event to have a priority level of High; however, in production with relative weighting with all other applications, it should actually be Low.
•	Instrumentation Behavior. Provide feedback on the frequency and exposure protocol or method used. The instrumentation may be triggering too often and causing too many events for the same condition. The instrumentation may be using an older protocol specification when a newer and more secure version and API are available.
•	Synthetic Transaction Capability. Software development may be able to improve or expose probes that can be used to perform synthetic transactions, which test internal business logic through a simulated transaction.
•	Preliminary Diagnosis and Self Correction. The goal for software development in relation to IT operations is to develop applications that are aware of their own issues and self correct them. SMC can provide consultative guidance-based operations experience to help applications mature in this direction. For example, strategies used in the Monitor and Control processes may be implemented internally into the application.

For more information on topics concerning management instrumentation for software development projects, please refer to Enterprise Instrumentation Framework for .NET at http://msdn.microsoft.com/vstudio/productinfo/enterprise/eif/

Include SMC Requirements in Release Package

Requirements in release management should be added to address the needs of SMC. This may include:

•

Delivery specifications (Health Model and instrumentation specifications)

•

Probes and interfaces for Control

•	Command line
•	Remotely accessible (accessible using WMI, for example)

Prepare Service Component Health Model

Development and application teams should be required to deliver their software packaged with its associated Health Model. A Health Model (also called a Health Specification for COTS) documents significant information for monitoring a application. This may include all actionable events, event exposure and behavior, and instrumentation protocols and behavior. Ideally, this information is directly codified into a language or configuration dataset that may be used by SMC tools. It is important to define taxonomy standards prior to documenting a Health Model so that the specific attribute values related to classification and prioritization levels align to a common reference.

There are two types of Health Models:

•	Class-level. Creates specifications based on a class of common infrastructure or service components. In a large organization with significant online presence using similar hardware and applications, an example may be a Health Specification for Web servers.
•	Override-level. Creates specifications based on individual infrastructure or service components that fall outside of a class grouping. In a large organization consisting mostly of databases using Microsoft SQL Server, an example may be a Health Specification for a specific host running Microsoft Access.

Reasons Why a Health Model Is Needed

Not knowing the information contained in the Health Model contributes to the following issues:

•	Administrators do not know when things are going wrong until something breaks.
•	When something breaks, it is difficult to determine what is broken and what to do about it.
•	Automatic monitoring tools do not have sufficient knowledge about the system to repair the problem.
•	Product support does not have the information required to troubleshoot the application.

The Health Model addresses the above problems by:

•	Prioritizing an application’s top known support and customer issues.
•	Documenting all management instrumentation that an application contains that can be used to determine health.
•	Documenting all known health states and transitions that the application can potentially go through during its life cycle.
•	Documenting the detection, verification, diagnosis, and recovery steps for all “bad” health states.
•	Identifying instrumentation (events, traces, and performance counters) necessary to detect, verify, diagnose, and recover from bad health states.
•	Refining the model as new states, transitions, and diagnostic steps are identified through customer, support, test, and community inputs.

General Guidelines for Creating a Health Model

The following is a list of best practices that can be used when creating a Health Model.

•	Define events with proper severity, so do not mark an event as an error unless it actually requires someone to take action and fix the condition.
•	Define events with unique ID and source combinations. Do not overload an event ID, which can cause monitoring tools to parse the event description to find the ID.
•	Do not generate events too frequently.
•	Define event descriptions accurately and, as much as possible, make the description actionable.
•	Do not expose performance data through events.
•	When appropriate, expose well-defined interfaces.
•	Measure availability or performance: generate events or alerts when defined criteria exist or thresholds are exceeded.
•	Determine the next steps to be taken: management rule sets can take advantage of scripts and state variables on the managed nodes to diagnose further.
•	Use simple measurements: CPU/memory usage, Windows Events, ability to read or write to a file or API, and service status results, for example.
•	Allow threshold modification: The Health Model must be able to customize to fit customers’ IT policies for infrastructure health.

Steps in Building a Health Model

Building the Health Model requires the following steps:

1.	Obtain a thorough understanding of application behavior and internal condition triggering.
2.	Enumerate all management instrumentation the application exposes. This will help identify additional health states and transitions, align instrumentation with the model, and identify where additional instrumentation is necessary.
3.	Analyze instrumentation and document health states, detection signatures, verification steps, diagnostic steps, and recovery actions.
4.	Analyze the service architecture for potential failure modes not currently exposed by instrumentation.
5.	Add all states that can only be detected by inspecting instrumentation or by exercising instrumentation methods.
6.	Create models that show health states and transitions between them.
7.	As the code evolves, update the model to accurately reflect the code. Add new health states and events to the model, and make sure that required instrumentation is in place.
8.	Use feedback from SMC and other SMFs to discover unknown problem states, and update the model accordingly. Add instrumentation where required to support these new states.

The following example gives a thorough description of the steps used in building a Health Model.

Steps 1 and 2. Obtain a thorough understanding of application specifics and management instrumentation exposure.

This can be accomplished by SMC collaborating with the application and development teams.

Step 3. Analyze instrumentation and document health states.

Using the SMC data repository, identify application events, and populate information for each key event.

Table 4. Examples of Data That Should Be Collected

Item	Description
Event ID	Event ID as reported to log
Symbolic name	Symbolic name for the event.
Facility	[Optional] Facility for the event.
Category	[Optional] Category for the event.
Type	Event type as reported to the event log.
Level	Severity of event. Revise if necessary. These might include: Critical: The application has encountered a critical degradation in its health or capabilities, which prevents it from servicing any subsequent operations. Error: The application has encountered a partial degradation in its capabilities, but it may be able to continue to service further requests. Warning: The application has encountered problems that are not immediately significant but which may indicate conditions that could cause future problems. Also, the application has detected problems in a different application. (However, these problems do not affect the application’s health or capabilities.) Informational: The application has encountered a positive change in its capabilities (that is, recovered from a previous degradation). These often negate previous degradations. Verbose: Diagnostic trace signifying detailed information from intermediate steps taken by the application while executing.
Message description	Event message description as written to log. Review and update as needed. Admin Event messages must have: Explanation: The explanation should provide a text description of what occurred and the change in the capabilities of the service that resulted from it. If the change is negative (that is, a degradation in capabilities), this description should specify the degradation that occurred. If the change is positive, this description should state what the new or restored capabilities are. User Action/Remedy: (not applicable for informational events): The user action/remedy presents steps the user can take to fix the problem, to diagnose it further, or both. It could include running a utility or performing a different task to fix the problem, retrying an operation, or looking into another log for further information about the problem.
Tag	This column should show into which classifications the event falls. Tags for event types that are specific to the service can also be added. Install: The event indicates the installation or un-installation of an application or service within the service raising the event. Settings: The event indicates a settings (configuration) change in the service. Life cycle: The event indicates a run-time life cycle change (for example, start, stop, pause, or maintenance) in the service. Security: The event indicates a change that is security related. Backup: The event indicates a change that is related to backup operations. Restore: The event indicates a change that is related to restore operations. Connectivity: The event indicates a change that is related to network connectivity issues. Low Resource: This event is related or caused by low resource (for example, disk or memory) issues. Archive: This event should be archived for the purpose of availability analysis. (These events must be infrequent—for example, restarting the computer.)
Insert parameters	Enter real property names for each of the insert parameters for this event. Use commas to separate insert parameters.
Blame component	If the blame for this failure falls on one of the dependencies, state the dependency to blame for the failure.
State before	Operational state of the application or service before the event.
State after	Operational state of the application or service after the event.
Desired state	Operational state in which the application or service would have been, had the event not occurred.
Event group	Name of a group of related events, all signifying a transition from one health state to another. Use a separate name for each transition line, but give the same name to all events that indicate that particular transition.
Availability	Current level of service availability in this state. Availability can be: Red: No service/functionality is available. Yellow: Partial service/functionality is available. Green: All service/functionality is available.
Verification	Test, probe, or presence/lack of an informational event that can be used to verify whether the service is in the detected state.
Diagnosis	What should be inspected to determine the root cause of why the application is in this state? Diagnosis typically starts by enumerating the list of "Detection" events and identifying where diagnosis should start for each one. Events, traces, configuration settings, WMI providers, and performance counters can all be sources for diagnostic information.
Recovery	How can the application recover from this state? What actions should be taken? Configuration settings, WMI providers, troubleshooters, and monitoring rules can all be used as potential recovery steps.
Auto-retry	Does the application automatically attempt to recover from this state? If so, how often?
Anti-event	Event that indicates a possible return to a healthy state for this event. If verified, invalidates the original transition to a bad health state.
Comments	General comments around this event, this state, or both.
Source file	Convenience column for listing the source file from which this event is logged. (Note: This is optional but has proven useful for some teams doing their analysis.)
Probability	Probability of occurrence of this event based on knowledge of the code path and experience from previous support issues. This is fairly subjective and is meant to help prioritize which events are most important to work on. This field can have a value of: Rare Low Medium High

Step 4. Analyze the service architecture for potential failure modes.

Map both the internal and external dependencies and how they can fail.

•	Examine the code for locations where failures are encountered, recovery logic has been written, or both.
•	Ensure that each of these locations in the code exposes the proper type of instrumentation based on the instrumentation selection guidelines provided later in this document. The instrumentation must provide the administrator or user with clear information about actions to take, the cause of the problem, the loss in functionality, and further diagnostic direction.
•	Make sure to have instrumentation to signal transitions from bad states to good (anti-alerts).
•	Update the instrumentation and state diagrams with this information.

Step 5. Add states that can be detected only by exercising instrumentation.

Not all health state transitions can be detected, diagnosed, and verified from inside of the service itself. For this reason, it is also important to document which client applications or services rely on the services, how they might be exercised to test the health of the service, and how the management instrumentation that they expose could indicate the failure to supply proper service to them.

An application might, for example, publish the average transaction time over a certain interval as a performance counter. An external service can detect a performance degradation by comparing this to historical data and generate an appropriate event. An application might also be blocked by waiting for an external application that has stopped responding.

Step 6. Create the health state diagrams.

A visual representation helps illustrate how the application or service looks as a whole. A visual health state transition diagram also can pinpoint where instrumentation is missing.

1.	Create a diagram that shows the states and the signals of transitions between those states (event groups)
2.	Look for locations where there are clear transition/recovery paths that no instrumentation will detect.
3.	Add the proper instrumentation to the code to be able to detect these conditions, and update the spreadsheet and diagram accordingly.
4.	Add events or other instrumentation to signal transitions from bad states to good.

Step 7. Incorporate code changes.

The code base is always evolving. New code is introduced, and old code is refactored. As the code evolves, keep the model up-to-date with the new code. These modeling documents need to be treated as living specifications that must be kept in synchronization with the current architecture at all times.

Step 8. Incorporate customer feedback.

Customers, community, product support, and test resources will report problems and solutions over the life cycle of the application.

New health states will be identified, alternate verification and diagnostic steps will be found, and quicker recovery paths will be discovered as services are deployed and used. The Health Model is a living set of documents. It must be improved over time as customers communicate how they manage the services in their environments and identify where management instrumentation needs to be added to future releases.

Implement

Overview

Implement is a major process in SMC that is responsible for the implementation of decisions made from the analysis in the Assess process. Implement is part of the run-time function of SMC.

The Implement set of activities is performed after Assess has qualified and analyzed a particular need and has designed a solution. The Implement activities are executed by SMC’s internal staff in coordination with other SMFs, especially those in the Operating Quadrant. As appropriate, change and release management are largely responsible for controlling the alteration of tools and infrastructure.

The activities in the Implement process flow should take advantage of all available automation, such as autodiscovery, tools, and scripts.

Figure 8. Main activities of the Implement process

Implement Process Activities

The following sections provide further details about each of the activities in the Implement process.

Adjust Monitoring Infrastructure

Implement Monitoring for New Service Components

Implementing monitoring for new systems and applications flows through the Assess: Review SMC Requests activity to analyze the monitoring target’s needs.

It is important to consider the impact of the Domain, Security, and Network models during this implementation. The Security and Domain models will dictate the user context in which the SMC tool performs its work. If the user/group using the SMC tool does not have adequate privileges, then the SMC tool will be unable to probe health conditions on the target. Control scripts may fail or partially execute from lack of adequate permissions. The Network Model dictates the access of monitoring traffic to the SMC tool server. If certain ports are blocked or if specific networks are segmented such as in a perimeter network (also known as a DMZ), then health status cannot be communicated and notification will fail.

Adjust Monitoring Parameters

Adjust Thresholds.

A threshold is the tolerable limit of a metric before an alert is generated. This limit is defined in the SLA, usually by availability, continuity, or capacity management. Any adjustments of thresholds should first be analyzed through the Assess process. Threshold adjustment should also be coordinated by change management as appropriate. When adjusting thresholds, make sure the new values are within the operating parameters of the element. Also make sure that thresholds match definitions from the Health Specification or Health Model.

Adjust Alert Prioritization.

Changes to alert prioritization should be made with caution since certain changes may make an alert too visible (the notification may be inadvertently distributed to higher-level personnel) or hide the alert (the notification may be undetected and unresolved). Changes to alert prioritization should be performed after Assess has reviewed and optimized the alert’s validity and actionability. (See this link for more details: Validity and Actionability.)

Adjust Rules.

Changes to rules should also be made with caution due to the potential for causing a flood of events or even damage through the misapplication of automated Control procedures. Following is a list of general guidelines for identifying the proper rule type to which changes should be applied:

Adjust Event Routing and Forwarding.

Changes to event routing and forwarding should be based on changes to the organizational model of the company. Event routing and forwarding is typically performed in SMC tool implementations with a multitiered topology or with multiple single configurations needing wide alert visibility.

Develop and Implement Automated Response.

Automated corrective response or control scripts can be developed after Assess has analyzed these opportunities for specific alerts. This automation should only be written against high-confidence conditions.

Develop or Update Knowledge Base and Document Event Behaviors.

It is important to keep good documentation on all event and instrumentation behaviors, rules, and responses. Knowledge base articles may be used as a way to keep track of these changes and optimizations.

Adjust Resources

As more infrastructure is monitored by SMC, there may be a need for increased staff to support the Assess and Monitor capabilities. Capacity and workforce management should coordinate any changes to staffing levels and resource allocations.

Monitor

Overview

The process of monitoring is concerned with the real-time observation of health conditions through technology-based notifications triggered by predefined thresholds and conditions. The Monitor process also documents the health state to ensure that adequate management information is available for maintaining agreed-to levels of service performance or, at a minimum, for quickly recovering service levels in the case of failure.

This process can also initiate a regular set of tasks (for example, daily/weekly/monthly) to record historical data for trending purposes. This data is normally used by other SMFs within the MOF Optimizing Quadrant (such as Availability Management and Capacity Management) and also to aid staff investigating underlying problems as part of the problem management function.

Monitor is performed by a monitoring operator role, typically in a Network Operations Center (NOC) or within the service desk.

Figure 9. Main activity of the Monitor process

Monitor Process Activity

Monitoring Mechanism

Monitoring can be performed using multiple views into the SMC tool. The two most commonly used notification media are through a dynamic console or through a notification device using e-mail or short messaging.

•

Console Notification. SMC tools can show the health state of services and service components through a console such as in a centralized organization with 7x24 operations. This is the most common means of achieving SMC visibility over a large infrastructure.

•	Alert-based. For ease of use, consoles can provide an iconic view such as showing a red, yellow, or green flag to indicate alert priority and status.
•	Pattern-based. Consoles can also represent data in graphical format such as a line graph. This facilitates signature-based pattern recognition, which is performed by senior SMC operators or SMC engineering staff.

•

E-mail or Short Messaging Notification. SMC tools can show the health state of services and service components through e-mail and short messaging typically sent to a pager, PDA, or cell phone. This is different from an incident or problem management dispatch in that the objective here is to communicate service and service component health, not necessarily a failure condition that must be acted upon.

Control

Overview

Many of the conditions observed in the Monitor process may represent incidents that can be automatically corrected in order to maintain or recover a service or a service component that may be affecting the business operations.

In order to minimize the impact of such incidents on business operations, the Control process deals with taking appropriate remedial actions to maintain or recover the affected services or their components. Actions referred to here are all performed in response to a message generated by one or more management tools. If an event creating a message represents an incident, most management systems can start actions to control, or correct, it. However, controlling actions are also used to perform daily tasks, such as starting an application every day on the same node.

Figure 10. Main activity of the Control process

Automated Control Response

Automated actions do not require any operator intervention and usually start as soon as a message is received. An operator can manually restart or stop them if necessary.

Where automated actions are used, the start rule should be recorded in the monitoring tool. If the operation of the rule is successful, it should be similarly recorded in the tool and the incident closed.

The unsuccessful operation of an automated response should, however, invoke the incident management process in order to resolve the incident. In this instance, the incident record is required to record the start and unsuccessful operation of the rule. Manual actions then need to be carried out by the appropriate support specialists using the agreed-on incident management process.

When automated actions have been run successfully, the advice should be closed without reference to the incident management process. The data on these successes should be made available to any other SMFs that may require it for trending purposes, or to aid proactive activity within availability management, capacity management, and problem management.

Closure and Recording

When an incident record has been raised following the unsuccessful operation of an automated action, the alert needs to be closed in the monitoring tool and the incident record should also be updated and closed.

During the closure process, the incident record should be updated with any further resolution information that may be useful in the future if the incident recurs.

It may also be helpful to update any local knowledge base that is provided within the service monitoring and control tool itself with any appropriate information relating to the particular advice issued or remedial actions required. This will ensure that the knowledge base grows into a valuable management tool for the future.

Control Process Activity

Control Functions

To initiate Control, service monitoring and control must define a set of rules as a predetermined task or set of tasks that are to be followed when a specific event occurs. These rules can be a script, program, command, application start, or any other response that is required in reaction to the event.

If the rule specifies that remedial action is required, then this should take the form of either manual or automated tasks. The process followed for each option is different. Where manual actions are required, the incident management process should be invoked in order to open an incident record. This invocation can be automatically completed by the monitoring tool or may require the operator to initiate it directly or by using the service desk.

The following are the three types of control functions:

Diagnostic Control

All diagnostics should be performed automatically by the system. Any incidents that require operator-based diagnosis should be forwarded to incident management for proper handling.

Guidelines for Creating Diagnostic Control.

The following best-practice guidelines should be considered when creating automated control capabilities.

Interoperability Control

Rules for alert handoff to incident management should be formalized in the Establish process. Theses rules should include specific incident prequalification data and could possibly include all the information about the specific event and instrumentation, conditions, alert, and knowledge base information. The handoff should be seamless and controlled and should update traceable states either within the SMC tool or through logged notification.

In general, all alerts that need manual investigation or diagnosis should be handled b

Service Management Functions

Service Monitoring and Control

On This Page

Executive Summary

Introduction

Document Purpose

What's New?

Feedback

Service Monitoring and Control Overview

Goals and Objectives

Scope

Core Concepts

Service

Service Definition

Service Components

Instrumentation

Health Model

Health Specification

Customer Impact

Key Definitions

Processes and Activities

Establish

Overview

Planning Considerations

Establish Process Activities

Prepare SMC Data

Collect SMC Prerequisite Material

Conduct SMC Enterprise Analysis

Develop Taxonomy Standards

Define Health Specification and Health Model

Prepare Run-Time Data

Formalize Roles

Formalize External Interactions

Adopt SMC Process

Prepare SMC Tools

Formalize Tool Requirements

Design Management and Tools Architecture

Initialize SMC Tools

Noise Reduction

Assess

Overview

Assess Process Activities

Review SMC Requests

Review Data from Other SMFs

Review Monitoring and Control

Assess SMC Tool Components

Review SMC Analysis Schedule

Analyze Monitoring and Response Rules

Obtain Feedback from Monitoring Consumers

Engage Software Development

Overview

Engage Software Development Process Activities

Collaborate on Operations Requirements

Infuse SMC Findings for Application Improvement

Include SMC Requirements in Release Package

Prepare Service Component Health Model

Reasons Why a Health Model Is Needed

General Guidelines for Creating a Health Model

Steps in Building a Health Model

Implement

Overview

Implement Process Activities

Adjust Monitoring Infrastructure

Implement Monitoring for New Service Components

Adjust Monitoring Parameters

Adjust Resources

Monitor

Overview

Monitor Process Activity

Monitoring Mechanism

Control

Overview

Automated Control Response

Closure and Recording

Control Process Activity

Control Functions

Diagnostic Control

Interoperability Control