Chapter 1 - MS Cluster Server Concepts
Microsoft Cluster Server (MSCS) software provides a clustering technology that keeps server-based applications highly available, regardless of individual component failures. This chapter presents the following terms and concepts involved in MSCS clustering:
MSCS is the Microsoft clustering-solution software used with Windows NT Server, Enterprise Edition. MSCS version 1.0 supports clusters of two, specially linked servers running Windows NT Server, Enterprise Edition. The primary function of MSCS occurs when one server in a cluster fails or is taken offline. With MSCS, the other server in the cluster takes over the failed server's operations. Clients using server resources experience little or no interruption of their work as the resource functions move from one server to the other.
There are two main components of MSCS: clustering software and Cluster Administrator.
The clustering software enables the two servers of a cluster to exchange specific types of messages that trigger the transfer of resource operations at the appropriate times. The clustering software has two chief features: the Cluster Service and the Resource Monitor. The Cluster Service runs on every cluster server and controls cluster activity, communication between cluster servers, and failure operations. The Resource Monitor facilitates communication between the Cluster Service and the application resources.
The Cluster Administrator is a graphical application that you use to manage a cluster. You can install and run Cluster Administrator on any computer running Service Pack 3 with version 4.0 of either Windows NT Workstation or Windows NT Server. You can administer any MSCS cluster from a cluster server, a non-clustered Windows NT Server, or from a computer running Windows NT Workstation.
You can also administer clusters using Cluster.exe, a command line tool, or custom administration tools developed using the MSCS automation interfaces. For more information on Cluster.exe, see "Administering Clusters from the Command Line" in Chapter 4, "Managing MSCS." For more information on the MSCS automation interfaces, see the Microsoft Platform Software Development Kit (SDK).
Advantages of Clustering
Network clients connect to cluster resources the same way they connect to any network server, so using clusters requires no additional training for end users. Clustering also enables static load balancing: the distribution of processes across servers.
MSCS clustering provides three main benefits for networks and network administrators: high availability of resources, scalability, and centralized administration, which are discussed in the following sections.
High Availability of Resources
Client/server applications rely on the availability of network services. These services are provided by resources, which are defined in this context as any entity residing on a server that can be managed by clustering software and that provides a service to clients in a client/server environment. For example, a public file share, a Web server, and a database application can all be managed as resources.
If these resources are unavailable because of downtime for a software or hardware upgrade, application failure, or hardware failure, this interrupts client work. Even if you schedule software and hardware maintenance during non-business hours, a server can unexpectedly fail, disrupting client service.
MSCS improves the availability of client/server applications by increasing the availability of server resources. Using MSCS, you can set up applications on two servers (nodes) in a cluster. MSCS provides a single, virtual image of the cluster to clients. If one node fails, the applications on the failed node are available on the other node. Throughout this process, client communications with applications usually continue with little or no interruption. In most cases, the interruption in service is detected in 5 seconds, and services can be available again in as few as 30 seconds (depending on how long it takes to restart the application).
Clustering provides high availability with static load balancing, unlike fault tolerance. Fault-tolerant solutions offer error-free, non-stop availability, usually by keeping a backup of the primary system. This backup system remains idle and unused until a failure occurs, which makes this an expensive solution. In a cluster, redundant systems are active, not passive. For example, one node in a cluster can service Web clients while the other node provides access to a SQL Server database. If either node fails, the resource (either the Web server or SQL server) fails over to the other node. The node that is still functioning responds to both Web and SQL requests from clients.
Scalable, but Affordable
Clusters are highly scalable: CPU, I/O, storage, and application resources can be added incrementally to efficiently expand capacity. This creates reliable access to system resources and data, as well as investment protection of both hardware and software resources. MSCS clusters are affordable because they can be built with commodity hardware (high-volume components that are relatively inexpensive).
By clustering existing hardware with new computers, you protect your investment in both hardware and software: Instead of replacing an existing computer with a new one of twice the capacity, you can simply add another computer of equal capacity. For example, if performance degrades because of an increase in the number of clients using an application on a server, you can add a second server to a cluster, which improves performance and also increases availability.
In an ordinary server environment, you use various administrative tools to identify the servers on your network and monitor their contents and activities. In the MSCS cluster environment, the administration of applications and services is centralized through the use of the single tool, Cluster Administrator. For example, you can use Cluster Administrator for the following tasks and activities:
For more information about using Cluster Administrator, see Chapter 4, "Managing MSCS."
MSCS Clustering Components
In MSCS, a cluster is a configuration of two nodes, each of which is an independent computer system. The cluster appears to network clients as a single server. For MSCS, both nodes must be running Windows NT Server, Enterprise Edition.
The network applications, data files, and other tools you install on the nodes are the cluster resources, which provide services to network clients. A resource can be hosted on only one node at any time, but you can configure the resource to fail over, in which the operation of the resource moves from one node to the other in response to a failure. You can also use MSCS to bring resources online and take them offline.
Related or dependent resources are organized into logical resource groups. Like the individual resources that make up a group, the group itself must run on only one node at a time. Figure 1.1 shows the relationship between nodes, groups, and resources.
Figure 1.1 Groups and the resources within the groups
Groups that contain an IP Address resource and a Network Name resource (along with other resources) are published to clients on the network under a unique server name. Because these groups appear as individual servers to clients, they are called virtual servers. Because users access applications or services on a virtual server in the same way they would if the application or service were on a physical server, end-users do not need any special training to use clustered resources. They do not even need to know that the system is using clusters. Figure 1.2 shows the client view of a network that includes both non-clustered servers and MSCS nodes. The client uses services provided on the MSCS node, but rather than connecting directly to the node, the client connects to the virtual server that provides the necessary service.
Figure 1.2 Example of servers and virtual servers on a network
Basic Cluster Configuration
The nodes in an MSCS cluster are connected using one or more shared storage buses and one or more physically independent networks (sometimes referred to as interconnects). If one network does not connect clients to the cluster, but serves only to provide a redundant path for intracluster communication, it is referred to as a private network. In this case, the network that supports clients is referred to as the public network.
Note Although MSCS clusters can function with only one interconnect, two interconnects are strongly recommended to eliminate a single point of failure, and are required for the verification of MSCS original-equipment-manufacturer (OEM) systems that include both hardware and MSCS software.
There are one or more local disks for each node. Each shared storage bus attaches one or more disks, which hold data used either by server applications running on the cluster or by applications for managing the cluster. MSCS currently supports only SCSI shared-storage buses. Each disk on the shared SCSI bus is owned by only one node of the cluster. The ownership of disks moves from one node to another when the disk group fails over or moves to the other node.
Figure 1.3 shows a typical configuration for a two-node cluster that is accessible by multiple clients on a local area network (LAN). The two cluster nodes are physically connected to the network. The cluster appears as a single network entity. The administrator tool can be MSCS Cluster Administrator (running locally or remotely) or an alternate administrative application (written using the Microsoft Platform SDK).
Figure 1.3 Configuration of two-node cluster
For information on choosing and configuring SCSI hardware, see Chapter 3, "Setting Up an MSCS Cluster."
Failover is the process of having cluster resources migrate from an unavailable node to an available node. A related process, failback, occurs when resources are restored to the node that has been offline after it is back online. Failover is automatically initiated by the Cluster Service when it detects a failure on one of the cluster nodes. Because each cluster node monitors both its own processes and the other node, the need for failover is detected without delay. In most cases, MSCS can detect a node has failed and begin failover in less than 10 seconds.
Figure 1.4 shows the failover process. A node on the right has failed, causing FilePrint group2 to fail over to the node on the left.
Figure 1.4 Failover of single group
When a single resource fails, MSCS fails over the group to which the failing resource belongs. Using Cluster Administrator, you can:
For groups, failover policy controls the conditions under which MSCS declares a group to be offline and no longer tries to bring it online. You can set the failover threshold and the failover period. The failover threshold is the number of times the group can fail over within the number of hours specified by the failover period. For example, if a group failover threshold is set to "5" and its failover period to "3," the clustering software stops attempting to bring the group online and leaves the resources within the group in their current state. For example, if the IP Address resource is brought online but the Network Name resource fails, the group is left offline, but the IP Address resource is left online.
For resources, the restart policy affects whether or not a resource is brought back online after a failure and the order in which this happens. As with groups, resources have a failover period and a failover threshold. The failover threshold specifies the number of times that the resource can fail within the number of minutes specified by the failover period. However, you can also choose whether or not the resource affects the other resources in its group when it is restarted. Typically you want the restarting of a resource to affect its group, but in certain cases you may not. For example, if you want to avoid failing over the whole group when a resource fails because the resource is less important from an availability standpoint than the other resources in the group, you can leave the group unaffected.
You can specify two polling intervals and a timeout value for resources. The polling intervals affect how often the MSCS Resource Monitor checks that the resource is available and operating. There are two levels of polling; they are known in Cluster Administrator as "Looks Alive" and "Is Alive." These values are named for the calls that the Resource Monitor makes to the resource to perform the polling. In "Looks Alive" polling, MSCS performs a cursory check to determine if the resource is available and running. In "Is Alive" polling, MSCS performs a more thorough check to determine if the resource is fully operational. The timeout value specifies how many seconds MSCS waits before it considers the resource failed.
For more information on how MSCS handles groups and resources during failover and failback, see the following section, "MSCS Resources." For more information about setting group failover policy and resource restart policy, see Chapter 4, "Managing MSCS."
A resource is any entity that can provide a service to a client and can be taken offline and brought online by the MSCS clustering software. MSCS organizes resources by type. MSCS supports several of the most common types of resources by providing resource dynamic-link library (DLL) files that communicate with the Resource Monitors and Cluster Service. Support for other resource types can be added by third-party vendors using the application programming interface provided in the Microsoft Platform SDK. Table 1.1 describes the default resource types.
Table 1.1 Resources supported by MSCS
All resources can have the following states:
When a resource is offline, it is unavailable for use by a client or another resource. When a resource is online, it is available for use. The initial state of any resource is offline. When a resource is in one of the pending states, it is in the process of either being brought online or taken offline. If the resource cannot be brought online or taken offline after a specified amount of time, and the resource is set to the failed state, you can specify the amount of time that MSCS waits before failing the resource by setting its Pending timeout value in Cluster Administrator.
Resource state changes can occur either manually (when you use Cluster Administrator to make a state transition) or automatically (during the failover process). When a group is failed over, the states of each resource are altered according to their dependencies on the other resources in the group.
For more information on using Cluster Administrator, see Chapter 4, "Managing MSCS," and Cluster Administrator Help.
Resource Groups and Dependencies
Typically, a resource group is an association of dependent or related resources. Dependent resources require other resources in order to operate successfully.
Because all resources in a group move between nodes as a unit, dependent or related resources never span a single group boundary (resources cannot be dependent on resources in other groups).
Figure 1.5 shows how dependent resources are joined to form a group. The node on the right contains the Web server group, which is made up of three resources upon which Internet Information Server 3.0 (or later) depends: a Network Name, an IP Address, and an IIS Virtual Root.
Figure 1.5 A group of dependent resources
The transfer order of the resources is dictated by the dependency relationships within the group. A resource upon which another resource depends must be brought online before the dependent resource and taken offline after it.
The disk resource is an example of a resource upon which other resources frequently depend. If the data within an application cannot be accessed from the disk media, the application is of little value. The same logic applies to IP addresses, which are necessary for network names and sometimes for services or applications. The cascading levels of interdependency define how groups and the resources within groups are configured.
Resources that are associated with a single department or task can also be grouped for administrative convenience. For example, one of the groups in the cluster can provide file and print services to the finance department and another group can provide similar services to the manufacturing department.
Typical clusters include one group for each independent application or service that runs on the cluster. Typical cluster groups contain the following types of resources:
MSCS uses the dependencies list when bringing resources online and offline. For example, if a group in which a Physical Disk resource and a File Share resource are located is brought online, it is futile to try bringing the File Share online before the Physical Disk (on which the file share is located). It is important to remember this when you configure resources because the groups function properly only when these dependencies are correctly configured.
Most groups have a Physical Disk resource, a Network Name resource, and an IP Address resource. In these groups, the application or service depends on the Physical Disk, which depends on the Network Name, which depends on the IP Address. Given these dependencies, when a group is taken offline, the Physical Disk resource is brought offline first, followed by the Network Name resource, and then the IP Address resource. Similarly, when the group is brought online, the IP Address resource is brought online first, followed by the Network Name resource and the Physical Disk resource.
You set resource dependencies when you add a resource to a group that already contains other resources. You can change dependencies any time using Cluster Administrator, in the Resource Properties dialog box on the Dependencies tab.
For more information on determining resource dependencies for your MSCS implementation, see "Capacity Planning" in Chapter 2, "Planning Your Cluster Environment." For more information on setting resource dependencies, see "Setting Resource Properties" in Chapter 4, "Managing MSCS."
The Quorum Resource
Each cluster has a special resource known as the quorum resource. A quorum resource can be any resource that does the following:
In MSCS version 1.0, of the default resource types, only the Physical Disk resource can be a quorum resource. However, your hardware manufacturer may supply other resource types capable of storing the quorum resource.
The quorum resource stores log data in the cluster log. This data is maintained by the clustering software. If the nodes in the cluster cannot communicate with each other, only one of them is allowed to continue operating. In this situation, the quorum resource, which can be owned by only one node, is used to determine which node can continue operating.
When a cluster node is started, it attempts to take ownership of the quorum resource. If the quorum resource does not already have an owner, the node takes ownership of the quorum resource. In this case, the node has formed a cluster. If, when a node starts, it finds that another node owns the quorum resource, it joins the cluster. If the two nodes are unable to communicate through one of the interconnects, they use the quorum resource to determine which node is allowed to continue running.
You specify an initial quorum resource when you install the first node of a cluster. You can use Cluster Administrator to change the quorum resource to a different storage resource.
If the quorum resource fails or becomes corrupted, or if the quorum log becomes corrupted, the Cluster Service cannot start. To correct this situation, you must modify the Cluster Service to start with a different quorum resource. For complete instructions on recovering from a corrupted quorum resource, see "Quorum Resource Fails" in Chapter 5, "Troubleshooting." For more information on changing the quorum-resource location, see Cluster Administrator Help.
MSCS and Windows NT Networking
MSCS takes advantage of much of the networking inherent in Windows NT Server, Enterprise Edition. It is crucial that certain specific features are configured correctly if your deployment of MSCS is to be successful. In an MSCS cluster, Windows NT networking is used for both node-to-node communication, and client-to-cluster communication.
MSCS and TCP/IP
MSCS requires the Transmission Control Protocol/Internet Protocol (TCP/IP) suite for its internal and client communication. MSCS works only with the TCP/IP protocol suite.
TCP/IP is an industry-standard suite of protocols, providing communications in a heterogeneous environment. While TCP/IP is best known for its access to the Internet and Internet resources, TCP/IP is also well-suited as an enterprise networking protocol because it provides interoperability between different types of computer systems.
Important MSCS does not support the use of IP addresses assigned from a Dynamic Host Configuration Protocol (DHCP) server for the cluster administration address (which is associated with the cluster name) or any IP Address resources. However, you can use either static IP addresses or static IP addresses assigned from a DHCP server for the Windows NT network configuration on each node.
For more information on the Windows NT TCP/IP protocol suite, see Part 1 of the Windows NT Server Version 4.0 Networking Supplement. To access this book online on a computer running Windows NT Server, click Start, point to Programs, and click Books Online.
MSCS and Name Resolution
In Windows NT networking, clients use name resolution services to find network resources. Clients use these same services to find resources in a cluster. MSCS also uses these services when a node joins another node to form a cluster.
In both cases, name resolution is the process of translating computer names to IP addresses. You can use the Windows Internet Name Service (WINS), the Domain Name System (DNS), or both. You can also use IP broadcast name resolution. However, because IP broadcast name resolution increases network traffic, and is ineffective in routed networks, this book assumes you are using WINS or DNS.
MSCS does not require any special configuration of either WINS or DNS. If you use WINS, all cluster resource names that have associated IP addresses are registered with WINS. If you use DNS, you can specify pertinent DNS record entries for the cluster resources. In general, you should treat each cluster group as a separate entity and manage names and addresses accordingly.
Understanding WINS and DNS
Windows Internet Name Service (WINS) is a NetBIOS name service used to translate computer names to IP addresses. WINS dramatically reduces the number of IP broadcasts that are used by Microsoft network clients. If a client needs to resolve a name to an IP address, the client sends a packet to the WINS server, requesting the corresponding IP address. WINS keeps a database of all the IP addresses on your network, each of which is mapped to a unique NetBIOS name.
Client computers that use WINS are configured with one or more IP addresses of WINS servers. When a client starts up, it communicates directly with a WINS server and registers its computer name and corresponding IP address. When a WINS client needs to resolve a computer name to an IP address, the WINS client sends a request to the WINS server for the IP address of the computer name being used.
Domain Name System (DNS) works similarly to WINS and is another option for maintaining IP address databases on your network. Unlike WINS, DNS resolves host names to IP addresses. These names are organized hierarchically rather than on equal levels, as in WINS. DNS name registration is static and must be done manually, unlike WINS registration, which is done dynamically and automatically.
In Windows NT 4.0, the Microsoft implementation of DNS is tightly integrated with WINS. This allows non-WINS clients to resolve NetBIOS names by querying a DNS server. Administrators can remove any static entries for Microsoft-based clients in legacy DNS server-zone files in favor of the dynamic WINS-DNS integration. However, while WINS supports the DHCP, DNS does not.
For more information on name resolution, see Chapter 3 of the Windows NT Server Version 4.0 Networking Supplement. To access the book online on a computer running Windows NT Server, click Start, point to Programs, and click Books Online.
MSCS Network Communication between Nodes
MSCS relies on the Windows NT network services for handling communication between nodes. In general, little network overhead is required during normal operations. MSCS also relies on Windows NT security services to secure all node-to-node communication, including the forming of a cluster.
Communication between the nodes of a cluster enables MSCS to detect node failures, status changes, and to manage the cluster as a single entity. Communication sessions can be multiplexed over multiple networks to eliminate a single point of failure.
Nodes in a cluster communicate using their Cluster Services. The Cluster Service keeps track of the current state of the nodes within a cluster and determines when a group and its resources should fail over to an alternate node. This communication takes the form of messages that are sent regularly between the two nodes' Cluster Services. These messages are called heartbeats.
In addition to heartbeat traffic, some network traffic occurs during startup, failover, and whenever changes are made to the cluster. During startup, node-to-node communication occurs for the purpose of synchronizing the cluster databases that reside on each node. The cluster database stores configuration information about the cluster resources.
MSCS Network Communication between Clients and the Cluster
Clients typically access network applications and resources through network names and IP addresses. When these network applications and resources are hosted within an MSCS cluster, clients can continue to find and access the resources, even though they may move between nodes. MSCS enables this by failing over both the IP address and network name for a given resource.
Figure 1.6 illustrates the Cluster Service involvement in client communications.
Figure 1.6 Communication between client and an MSCS cluster
Cluster Resource Communication
The MSCS Cluster Service communicates with:
Communication between the Cluster Service and Resources
The Cluster Service communicates with resources through the clustering software component known as the Resource Monitor. The Resource Monitor controls and monitors cluster resources, and reports any changes in the resources to the Cluster Service. Because there is a separate Cluster Service and one or more Resource Monitors on each node, this communication takes place in the form of interprocess communications (IPC) and has no direct effect on the network.
The Resource Monitor accepts requests from the Cluster Service and forwards those requests to the cluster-resource DLLs. When a resource detects a change in its state, it will report that change when polled by the Resource Monitor, which then forwards the report to the Cluster Service.
You can control how often the Resource Monitor communicates with resources by setting the "Is Alive" and "Looks Alive" polling intervals using Cluster Administrator. For more information on setting polling intervals using Cluster Administrator, see "Setting Advanced Resource Properties" in Chapter 4, "Managing MSCS," or see Cluster Administrator Help.
By default, MSCS uses the Resource Monitor for all resources on a node. If a problematic resource causes a Resource Monitor to stop responding, you can run that resource in a separate Resource Monitor. Because each Resource Monitor runs as a separate process, this isolates the problematic resource.
Communication between Applications and the Cluster Service
Applications that communicate with the Cluster Service on a node are typically cluster-management applications, such as Cluster Administrator. Cluster-management applications access the Cluster Service through a standard applications programming interface (API), using RPC communication over the TCP/IP protocol. (These applications are defined as MSCS-aware.) The Cluster Service also uses an API set to communicate with cluster resources.
To connect to a cluster from an application, you can specify any network name in the cluster for which there is an associated IP Address resource (assuming the Network Name and IP Address resources are online).
Note Although clients can access cluster resources through node names or the cluster name, this should be discouraged because the resource becomes unavailable if the node becomes unavailable, or the resource moves to the other node.
Figure 1.7 illustrates how applications (whether MSCS-aware or MSCS-unaware) and the Cluster Administrator communicate with a cluster.
Figure 1.7 Communication between applications and a cluster node
For an explanation of what constitutes an MSCS-aware application, see "Determining Applications to use with MSCS" in Chapter 2, "Planning Your Cluster Environment."
MSCS and Security
MSCS relies on Windows NT security at all levels for securing communication between nodes, the joining of nodes, and controlling authentication and access, including client access to the cluster and shared resources. You can use Cluster Administrator to specify which users or groups can:
To audit the access of files on File Share resources, use My Computer or Windows NT Explorer to set auditing. (Use User Manager for Domains to set the audit policy for file and object access.)
Table 1.2 lists the Windows NT security features you can use to control client access to the various types of resources supported by MSCS.
Table 1.2 Securing MSCS resources
In general, set permissions on the individual applications and services as you would normally.
For more information on auditing events, see Chapter 9, "Monitoring Events," in Windows NT Server Version 4.0 Concepts and Planning.
Domain Accounts Required to Install, Run, and Administer MSCS
Prior to installing MSCS, you must log on to both nodes under a domain account that has Administrator permissions on both nodes. Both nodes should have computer accounts on the domain and be members of the same domain (instead of a workgroup). When you install MSCS, you specify the domain user account under which the Cluster Service runs. This account requires no special account privileges, but password restrictions, such as requiring password changes and Change password on next logon, should be turned off.
When you install MSCS on a member server, MSCS Setup gives cluster administrative permissions to the local Administrators group. When you install MSCS on a PDC or BDC, MSCS Setup gives cluster-administrative permissions to the Domain Administrators group.
When you administer a cluster from a remote location, such as from another server or a computer running Windows NT Workstation, you must also have administrative permissions on both nodes of the cluster, or you must have specific permissions to administer the cluster. By default, the local Administrators group on both nodes has permission to administer the cluster. To give permission to administer the cluster from domain accounts that do not belong to the Administrators group on both nodes, in the cluster Properties dialog box, click Permissions on the General tab.
If you have administrative permissions on both nodes of the cluster, you can fully administer the cluster. However, if you have only specific permissions to administer the cluster, you cannot change the cluster description or change cluster security.
For information on using Cluster Administrator to set cluster access permissions, see Cluster Administrator Help.
For information on changing the account used to run the Cluster Service, see "Managing Security" in Chapter 4, "Managing MSCS."
Disk Resource Security
Two disk subsystems exist in a clustered environment: those relevant to the shared SCSI bus and those that support Windows NT. These two systems should be kept separate: Do not put Windows NT system files on the shared SCSI bus. The Windows NT system files can reside on either FAT or NTFS drives, but the shared SCSI bus must contain only NTFS-formatted drives.
All standard Windows NT file-security options apply to the disk resources on the shared SCSI bus. You can control access to folders and files of File Share resources on a user or group basis. You can set these access permissions through My Computer, Windows NT Explorer, or Cluster Administrator.
Important Do not modify the access permissions on the disk that contains the quorum resource. MSCS must have full access to the quorum log. (The quorum log file is used to write all cluster state and configuration changes that cannot be committed to the other node.) For this reason, you should never restrict either node's access to the folder on the quorum disk containing the quorum log (\MSCS by default).
Security Considerations on Non-Domain Controllers
Local users and local groups have security context only on the local computer. If these groups were allowed to have cluster-administration permissions, the security context would be meaningless when failed from one node to another. For this reason, you cannot use the MSCS Permissions dialog box to give local users or local groups permissions to administer the cluster. The single exception to this rule is the local Administrator group.
Similarly, on non-domain controllers, you should not specify local accounts and groups when setting access permissions for files and folders that reside on any disk on the shared-SCSI bus. This is not a problem on domain controllers (DCs) because the local accounts and groups have security context on all DCs in the domain.