Technical White Paper

Published: August 1, 2003

Executive Summary

The Operations and Technology Group (OTG) at Microsoft recently deployed Microsoft® Exchange Server 2003, the latest edition of the company's industry-leading enterprise messaging application. OTG not only serves the company by running the IT utility for its myriad employees and locations, but also serves as the first and best customer for the various enterprise product development groups at Microsoft, deploying Microsoft software within the company before it is available to outside customers.

The migration from Microsoft Exchange 2000 Server to Microsoft Exchange Server 2003 led to some significant changes in the messaging architecture at Microsoft. OTG has moved toward a fully clustered, mailbox server environment. Each of these server clusters are connected to one or more Storage Area Network (SAN) enclosures for its data storage. The use of clustering technology has improved reliability, increased availability, and improved the process of performing rolling upgrades.

The benefits of deploying Exchange 2003, especially when combined with the benefits derived from the deployments of both Microsoft Windows Server™ 2003 and Microsoft Office 2003, have enabled Microsoft to consolidate its messaging infrastructure. OTG has begun implementing its plans to consolidate 113 mailbox servers in 75 locations worldwide to just 38 mailbox servers in seven locations. Exchange 2003 also supports all mobility messaging services, such as Outlook Web Access (OWA), Outlook Mobile Access (OMA), and Exchange ActiveSync® (EAS), on the same server, enabling OTG to additionally consolidate its worldwide front-end server infrastructure.

The messaging data storage infrastructure has also been updated. Data storage, once a combination of direct attached Small Computer System Interface (SCSI) storage arrays at remote locations and SAN solutions in the Redmond, Washington headquarters data center have been replaced by SANs at all locations. These changes have enabled OTG to increase the number of mailboxes per server and thoroughly enhanced the performance and capability of backup and recovery solutions as well.

As of this writing, OTG has significantly reduced administrative overhead for Exchange, improved system performance and service availability, and improved its own ability to meet its Service Level Agreement (SLA) obligations. Those benefits should become even more dramatic as the company moves closer to its consolidation goal.

Note   For security reasons, the sample names of forests, domains, internal resources, and organizations used in this paper do not represent real resource names used within Microsoft and are for illustration purposes only.

Introduction

Microsoft Exchange Server 2003 represents an important, continuing investment in enterprise technology for Microsoft. Exchange 2003 offers improvements required by enterprise messaging and collaboration customers. Many of the largest companies in the world run their messaging systems on Microsoft Exchange, including Microsoft.

The purpose of this document is to provide an overview of the architecture and design decisions made during the upgrade of Exchange Server 2003 at Microsoft. The paper focuses on the hardware selection and configuration aspects of the project. It also includes discussions on the key technology wins and best practices that emerged from the upgrade. Since OTG is a leading edge implementer of Microsoft technologies and products, the organization brings a unique set of requirements as well as innovative approaches to meeting the needs of its customers. This paper describes these requirements and approaches, as well as the way they affected design decisions for the deployment. The intended audience for this white paper includes technical decision makers, system architects, IT implementers, and messaging system managers.

OTG based its mission for migrating from Exchange 2000 to Exchange 2003 on achieving several objectives:

• To test and improve the product before Microsoft offered it to its customers.
• To consolidate Exchange server sites worldwide to reduce server maintenance and administration costs and workload.
• To simplify the messaging infrastructure based on standardized server and storage hardware for all deployment locations.
• To improve the ability of OTG to meet its SLA obligations for data backup and restore.
• To significantly improve the end-user experience with messaging services at Microsoft.

OTG met all these objectives when it deployed Exchange 2003.

Overview of Current Network Infrastructure

With all of the beta-level and test version software used in its production environment, the Microsoft corporate network is the world's largest experimental computer network. The network is a confederation of functional backbones, spanning the globe. Each backbone is defined on regional boundaries with connectivity focused on the Main corporate campus located in the Puget Sound Metropolitan Area.

The network is architected following a multi-domain routing model. It is divided into four regional networks, with each network functioning as a single Open Shortest Path First (OSPF) routing and addressing domain. The four regions cover the following areas: 1. the Puget Sound metropolitan area in western Washington State; 2. Europe, Africa, and the Middle East; 3. Japan, the Pacific Rim, and the South Pacific, and 4. the remainder of North America and South America.

Each regional network consists of a backbone area (Area 0) and multiple areas to ensure scalability of each regional network. External Border Gateway Protocol (EBGP) is used to exchange routes between the regional networks to ensure the scalability of the network as a whole.

The Puget Sound Metropolitan Area Network (MAN) supports the bulk of data traffic on the global enterprise network providing gigabit rate connectivity between buildings and the main datacenters located in the area. The current campus is comprised of 70 separate buildings and two datacenters with a network infrastructure providing access to corporate resources, developer lab networks, and Internet connectivity to any location within the campus.

This network relies on Gigabit Ethernet and Packet over Synchronous Optical Network (SONET), using privately owned or leased Dark Fiber as the transport medium. In the metro area, efficient use of limited fiber resources is realized by leveraging Wave Division Multiplexing (WDM) technologies to provision multiple circuits across a single physical link.

The available network bandwidth is significant for applications like Exchange Server 2003 and site-to-site connectivity. As of June 2003, the network had grown to encompass:

• Three enterprise data centers, nineteen regional data centers worldwide
• 310 sites in approximately 230 cities in 77 countries
• The largest wireless LAN (802.1x EAP-TLS) in the world
• More than 24,000 wireless devices
• More than 4,000 wireless access points
• More than 250 wide area network (WAN) circuits
• More than 200 WAN sites in more than 70 countries
• More than 3,300 IP subnets
• More than 2,000 routers
• More than 2,600 network layer 2 switches
• More than 275 ATM switches
• More than 10,000 world wide servers
• More than 350,000 LAN ports

Overview of Current Messaging Infrastructure

Managing the complex messaging infrastructure at Microsoft is a team effort that involves many different groups within OTG. Organizationally, OTG is comprised of more than 2,500 staff members that are responsible for operations spanning more than 400 IT locations worldwide. In addition to providing the IT utility for the company, OTG plays a key role in helping Microsoft meet its main business objective of software development and marketing. As the first and best customer of Microsoft, OTG serves as an early adopter of new Microsoft software, such as Windows Server 2003, Microsoft Office 2003, and Exchange Server 2003. The result of this process is known in the industry as "eating your own dog food."

In the "dog food" messaging environment of OTG, servers regularly receive software patches, operating system test releases and upgrades, Exchange server test releases and upgrades, and more. Each Exchange server is "touched" by OTG for these software upgrades on an average of two times each month. The changes to software are implemented to test new scenarios, meet specific requirements, and continually run the latest application concepts through real world, enterprise-level testing. The rate of change is very high in OTG.

Microsoft employees place a significant load on the messaging infrastructure. The average employee at Microsoft possesses three computers, typically all of which are used to synchronize with Exchange. In addition, a significant portion of that population also carries Pocket PC and Smartphone devices that also synchronize with Exchange. The average Remote Procedure Call (RPC) operations per second (a measurement of work) at Microsoft is significantly higher than at any other company known to OTG. Microsoft often works with customers and partners to benchmark their messaging infrastructure. The workload managed by the Exchange servers at Microsoft is typically more than double than the load measured at these companies.

At the time of this writing, the messaging environment at Microsoft consists of more than 200 servers, including 190 Exchange 2003 servers (113 of which are mailbox servers) in 75 locations worldwide, including servers in additional cross-forest test environments. This environment supports:

• Global mail flow of 6,000,000 messages per day, with 2,500,000 average Internet e-mail messages per day, 70 percent of which is filtered out as either unwanted spam e-mail, virus-infected, or to invalid e-mail addresses. Comparing bytes over the wire, the size ratio of blocked message content versus accepted message content received at Microsoft is 40:1. The average size of a typical e-mail message is 44 KB.
• Approximately 85,000 mailboxes, each being increased from a 100 MB to 200 MB limit. Average 100 MB mailbox was only 44 MB in size.
• More than 85,500 distribution groups.
• More than 230,000 unique public folders managed on public folder servers.

The OTG server infrastructure includes:

• Corporate standard client configuration comprised of Windows® XP Professional and Microsoft Office Outlook® 2003.
• Legacy, stand-alone mailbox server configurations of 500, 1,000, or 1,500 mailboxes on stand-alone servers. Stand-alone servers are being replaced by clustered SAN solutions worldwide and have been scaled per server to support 2,700 user mailboxes in regional locations and 4,000 user mailboxes in the headquarters data center.
• One centrally located support organization in headquarters supports all Exchange servers worldwide.
• In addition to the Main corporate Exchange Active Directory® forest, three additional forests are used to host Exchange mailbox servers at Microsoft:
  • A Level A Test forest dedicated that runs development and test code for Exchange, operating in a frequently changing server software environment.
  • A specialized Level B Test forest, serving as a limited-use production environment used by one product division that hosts a limited number of user mailboxes. Specialized hardware configurations and test scenarios can be run in this environment. Level B Test uses a two-node server cluster connected to a SAN scaled to support 5,000 user mailboxes.
  • A legacy test environment forest that is used for testing Windows server operating system versions one version back from the currently released version (specifically Windows 2000 Service Pack-specific testing) with Exchange.

Note   OTG uses both Level A Test and Level B Test forests to test cross-forest behavior and support with the Main Microsoft corporate production forest.

The OTG service levels include:

• The global service availability Service Level Agreement (SLA) goal in the Main corporate forest, calculated as the availability of mailbox databases per minute (including both planned and unplanned outages), was 99.9 percent for stand-alone server designs. This was increased to 99.99 percent for the new clustered server designs used with Exchange 2003.
• Worldwide e-mail delivery in less than 90 seconds, 95 percent of the time.
• Backup and restore operation SLA of less than one hour per database.

Note   For security reasons, the sample names of forests, domains, internal resources, and organizations used in this paper are fictitious. They do not represent real resource names used within Microsoft and they are in this document for illustration purposes only.

Sites and Locations

Following the lead of the Exchange 2000 deployment, OTG continued the strategy of deploying Exchange servers in dedicated roles. Table 1 shows the distribution of Exchange 2003 servers by server role. OTG grouped the Exchange 2003 servers into 37 Exchange routing groups that were interconnected with 79 site connectors.

Table 1   Exchange 2003 Server Distribution by Server Role at Microsoft

* The mailbox server consolidation project is slated to be completed as of the end of the calendar year 2003.

** OTG will set up seven messaging hubs and four additional dual-purpose servers that will provide messaging hub services.

*** Exchange Instant Messaging servers will be eliminated as the messaging service is migrated to Windows Real Time Communications (WinRTC) servers.

**** All of the Free/Busy server services will be provided by existing Public Folder servers. OTG will not set up any dedicated Free/Busy servers at Microsoft.

***** Front-End servers were consolidated with the deployment of Exchange 2003 since the technology formerly included in Mobile Information Server (MIS) 2002 product was added into Exchange 2003. To increase system availability, each Exchange 2003 front-end server deployment site was configured with a pair of load-balanced servers.

Routing Group and Administrative Group Structure

In all Exchange deployments prior to Exchange 2000 (including versions 4.0, 5.0, and 5.5), OTG grouped Exchange servers into sites based on the network topology. For Exchange 5.5, OTG designed the environment to strike a balance between the need for large sites and the limitations of network bandwidth within those sites because of directory and public folder replication and message routing traffic.

Since the release of Exchange 2000 on Windows 2000, the limits and boundaries imposed by the Exchange 5.5 model were no longer a concern. The ability to place servers in routing groups independent of their administration group membership allowed OTG to optimize the routing topology without losing the advantages of large administrative groups.

Directory replication is now a function of Active Directory and is an operating system-level issue that is no longer a key concern of the Exchange deployment. Since routing groups and administrative groups need not be the same (as was the case in Exchange 5.5 and earlier versions), the OTG Messaging operations staff is free to place Exchange 2003 servers into groups that match their administrative and operational structure, and into routing groups that match the WAN topology. This leaves directory replication concerns to another OTG team specifically focused in that area. As of this writing OTG maintains 31 Exchange Server 2003 routing groups and 11 administration groups.

Exchange 2000 Legacy Architecture

OTG began its deployment of Exchange 2003 when the product was still in an early beta version. To fully grasp the scope of this project, let us review the previous messaging infrastructure under Exchange 2000, the compelling reasons why OTG had to upgrade to Exchange 2003, and what OTG did to make the upgrade a success. Various challenges and discoveries made by OTG during this experience are included to provide some guidance and considerations as you plan your Exchange 2003 deployment.

Overview of Exchange 2000 Infrastructure

The Microsoft Exchange Server platform is the fastest selling Microsoft server product in history. Since 1996, when Exchange 4.0 was released, Exchange Server has sold more than 50 million seats. Table 2 provides an overview of the evolution of the internal deployment of Exchange Server at Microsoft since 1996 when Microsoft first released Exchange Server.

Table 2   The Evolution of Exchange Server Deployment at Microsoft

* It takes 25 minutes to restore a database from backup disks.

Legacy Server and Storage Design

OTG used stand-alone servers in both the headquarters data center and in all regional deployments. The servers were categorized into four basic mailbox server configurations as shown in Table 3.

Table 3   Microsoft OTG Exchange 2000 Server Configurations

The storage design varied depending upon the requirements of each server configuration. All Exchange 2000 mailbox servers supported 100 MB mailboxes. The regional server configurations used direct attached SCSI storage disk arrays that were backed up over the 100 Mbps LAN. The data center configuration servers used three SAN arrays, each one comprising one SG. They were backed up over the Gigabit LAN.

OTG used best practice guidelines when designing their original Exchange servers with consideration towards maximizing system performance and availability with both the server and storage hardware. To optimize the disk input/output (I/O), each volume of an SG was designated as a Logical Unit Number (LUN). Since each LUN was assigned a drive letter, each server, hosting three SGs comprised of three LUNs each, used nine drive letters.

OTG configured each SG to maintain three separate LUNS. The mailbox data LUN using 24 18-GB disks and the Log LUN using six 18-GB disks were both configured using a striped mirror configuration, known as Redundant Array of Independent Disks (RAID)-10. The SAN also maintained a dedicated backup LUN utilizing 12 36-GB disks in a RAID-5 configuration. This LUN was used to support two days of online, disk-to-disk backup retention.

Each SG supported five databases, and each database supported 200 mailboxes, meaning that they could support up to 1,000 mailboxes per SG and 3,000 mailboxes per server.

Performance, Scalability, and Supportability Challenges

Exchange 2000 was a major upgrade from previous versions of Exchange. However, as powerful as Exchange 2000 was, OTG still had to work around some limitations.

Number of Servers to Manage Too High

Due to an inability to consolidate servers and sites effectively, the number of sites with servers drove support costs significantly higher and added complexity into the messaging environment. Some of the more common cost factors associated with the distributed environment included:

• More systems to backup
• Additional maintenance of backup systems at larger number of sites
• More personnel added to administer backup processes
• Greater power and cooling resources required at additional sites
• More onsite support staff added for hardware maintenance at multiple sites

From a complexity perspective, the larger number of systems meant more moving parts in a complex machine; i.e. the more backup jobs required, even with the same success rate, means a higher number of failures to troubleshoot and resolve. The planned 90 percent reduction in the number of sites with servers dramatically reduces the number of moving parts in the messaging machine, thereby reducing the exposure to failure on a number of fronts.

Recoverability of Databases within Service Level Agreement (SLA) Time Difficult

Even small efforts to consolidate resulted in higher scaling on servers in a number of sites. As the number of mailboxes on a server continued to increase with scalability improvements in the product, database sizes grew as well. More significantly, the initiative to increase the maximum mailbox size from 100 MB mailboxes to 200 MB mailboxes promised an immediate doubling in the size of databases.

Since Exchange 2000 does not offer support for new recovery options such as Recovery Storage Group (RSG) functionality or Volume Shadow Copy Service (VSS), a database outage due to corruption on an Exchange 2000 Server meant that the process of database restoration would result in an extended outage. In many sites, backups were managed across multiple computers in a datacenter, which resulted in backups and restores occurring over the 100 MB LAN, for which restore times averaged, at best, 16 GB per hour. The original restore SLA was full database restore in one hour, a goal that was quickly becoming unattainable.

Cluster Scalability Limitations

Windows 2000 Advanced Server supported two-node clusters and Windows 2000 Datacenter Server supported four-node clusters. With Exchange 2000 running on Windows 2000 Advanced Server, for an optimized configuration, OTG needed to have multiple drive letter volumes associated with each SG. There were also additional drive letters used in the server configuration, such as the Simple Mail Transfer Protocol (SMTP) drive (a dedicated inbound/outbound queue device). As a result, each virtual Exchange server within the cluster, after accounting for the collective SGs and the SMTP drive, used ten extended drive letters. This does not account for the required, reserved drive letters used by the server node itself, such as for the floppy disk, operating system volumes, and a CD drive. OTG could only use two servers in a cluster before it exhausted the supply of available letters assignable to disk volumes. The lack of available drive letters prevented OTG from adding additional instances of Exchange servers into a clustered environment.

Backup Infrastructure Inflexible

OTG processed a single-stage backup for regional servers. The regional servers used the 100 Mbps LAN to perform a direct, disk-to-tape backup. In Redmond, servers performed a two-stage backup process: first disk-to-disk within the SAN, and then disk-to-tape. To ensure that the backup process completed during non-business hours, OTG needed to deploy Gigabit Ethernet network adapters in each Exchange server to ensure that they could get the throughput necessary to push the data across the LAN and onto tape.

Data restoration required the creation of a temporary restoration server to serve as a staging server for retrieving data from tape. OTG learned that it in addition to the time it took to restore the data, before that process could start, a tape drive had to read and seek the starting point of that particular database on a tape. This process often entailed a wait of 90 minutes or more before any data actually transferred to disk. The typical throughput for data restoration (once data began to flow) on the OTG 100 Mbps network was approximately 300-350 MB per minute. With a selective restoration of a sample 15 GB database, the total time needed to complete the job was often more than two hours – far in excess of the SLA.

In the end, OTG based its entire architecture of Exchange 2000 on the technical requirements for meeting backup and restore efforts within the allotted SLA time window.

Reasons for OTG to Upgrade

OTG had many compelling reasons to upgrade to Exchange 2003. Of course, in its special role as a group running Microsoft product group dog food software, OTG was committed to deploying Exchange 2003. This deployment was an effort to improve the product with real world, enterprise experience and feedback, long before any customers would receive the product.

In addition, Exchange 2003 resolved the Exchange 2000 challenges for OTG as described earlier. The deployment of Exchange 2003 enabled OTG to improve service to its customers and to reduce operations requirements. Microsoft realized the following business benefits:

• Reduced number of servers
• Improved server availability, reliability, and manageability
• Improved clustering support
• Improved security
• Improved data backup and recovery
• Improved support for mobile users
• Improved integration with Office 2003

Site and Server Consolidation

As of this writing, with the deployment of Exchange 2003 completed, OTG is in the process of implementing a long-planned consolidation of regional mailbox servers and locations. OTG had 113 mailbox servers in 75 locations around the world. The end goal of the consolidation plan is to reduce the number of locations by 90 percent, down to seven worldwide, using 38 clustered Exchange virtual mailbox servers. This level of server reduction will significantly reduce the administrative workload required of the messaging infrastructure in OTG.

Normally an increased number of mailboxes per server and a greater amount of data per SG would present an increased risk in the event of failure. Indeed, OTG measures database service availability as a factor of downtime multiplied by the number of databases affected. For example, a one-minute outage affecting a single SG of five databases on a server containing three SGs (containing 15 databases) is measured as five minutes of downtime. In addition, OTG studied its downtime incidents and learned that its planned downtime exceeded its unplanned downtime by a factor of 6:1.

Despite the fact that the number of mailboxes per server is growing, and that mailboxes are doubling in size, the site and server consolidation project is expected to improve OTG's overall availability as well as its backup and restore performance SLAs. It is also expected to reduce the OTG server management workload significantly, thereby reducing costs.

For more information about OTG's Exchange Server 2003 site consolidation plan, see the iT Showcase technical white paper titled, "Exchange 2003 Site Consolidation" at http://www.microsoft.com/technet/itsolutions/msit/default.mspx.

Availability/Reliability/Manageability Enhancements

Exchange 2003 offers a variety of enhancements that make it a compelling upgrade.

Virtual Memory Management

The virtual memory improvements to Exchange 2003 reduce memory fragmentation and increase server availability. Specifically, Exchange is much more efficient in the way it reuses blocks of virtual memory. These design improvements reduce fragmentation and increase availability for higher-end servers that have a large number of mailboxes.

Virtual memory management for clustered Exchange servers is also improved. In Exchange 2003, when an Exchange virtual server is either moved manually or failed over to another node, the MSExchangeIS service on that node is stopped. Then, when an Exchange virtual server is moved or failed back to that node, a new MSExchangeIS service is started and, consequently, a fresh block of virtual memory is allocated to the service.

Exchange System Manager (ESM)

Administrator functionality using ESM has been enhanced in Exchange 2003 with these key updates:

• Improved method for moving mailboxes. The Exchange Task Wizard now allows you to select as many mailboxes as you want and then, using the task scheduler, to schedule the move to occur at some point in the future. You can also use the scheduler to cancel any unfinished moves at a selected time. Using the wizard's multi-threading capabilities, you can move up to four mailboxes simultaneously.
• Improved Public Folder interfaces. To make public folders easier to manage, Exchange 2003 includes several new public folder interfaces in the form of tabs.
  • The Content tab displays the contents of a public folder in Exchange System Manager.
  • The Find tab enables searches for public folders within the selected public folder or public folder hierarchy. A variety of search criteria can be specified, such as the folder name or age. This tab is available at the top-level hierarchy level as well as the folder level.
  • The Status tab displays the status of a public folder, including information about servers that have a replica of the folder and the number of items in the folder.
  • The Replication tab displays replication information about the folder.
• New Mailbox Recovery Center. Using the new Mailbox Recovery Center, you can simultaneously perform recovery or export operations on multiple disconnected mailboxes.
• Enhanced Queue Viewer. The Queue Viewer improves the monitoring of message queues. Enhancements include:
  • The X.400 and STMP queues are displayed in Queue Viewer, rather than from their respective protocol nodes.
  • The Disable Outbound Mail option allows you to disable outbound mail from all SMTP queues.
  • The refresh rate of the queues can be set using the Settings option.
  • Messages are searchable based on the sender, recipient, and message state using Find Messages.
  • Queues are clickable for displaying additional information about that queue.
  • Previously hidden queues, DSN messages pending submission, Failed message retry queue, and Messages queued for deferred delivery, have been exposed.
• Enhanced control of message tracking log files. When using Exchange System Manager, you have greater control over your message tracking log files. Exchange 2003 automatically creates a shared directory to the message tracking logs and allows you to change the location of the message tracking logs.
• Improved error reporting. Error reporting allows server administrators to easily report errors to Microsoft. Although error reporting was included in Exchange 2000 SP2 and SP3, its implementation is improved in Exchange 2003. For example, if users do not want to view the standard error reporting dialog box, they can configure Exchange to send service-related error reports to Microsoft automatically.

Improved Cluster Support

Clustering in Windows Server 2003 provides a number of improvements that allows OTG to take full advantage of this technology to provide a solid clustered server standard to support its global Exchange mailbox server consolidation initiative. The new standard provides for a better level of scalability and availability over any previous deployment methodologies used for Microsoft's corporate Exchange deployment.

Support for Up to Eight Nodes

Exchange has added support for up to 8-node active/passive clusters when using Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition. This enabled OTG to boost the number of servers in their Exchange Server 2003 clusters, thereby substantially improving server availability and reliability while reducing the number of Exchange deployments necessary to manage the Microsoft corporate messaging environment.

Support for Volume Mount Points

Exchange now supports the use of volume mount points when using Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition.

A volume mount point is a feature of the NTFS file system that allows linking of multiple disk volumes into a single tree, similar to the way the Distributed File System (DFS) of a server links remote network shares. Administrators can link many disk volumes together with only a single drive letter pointing to the root volume. The combination of an NTFS junction and a volume mount point can be used to graft multiple volumes into the namespace of a host NTFS volume.

Improved Failover Performance

Exchange has improved clustering performance by reducing the amount of time it takes a server to failover to a new node. Exchange specifically optimized the process of shutting down services on the running active node, expediting the failover and the startup of services on an alternative node, thereby improving overall system performance.

Improved Security

When Microsoft prioritized security as its first order of business, Exchange 2003 realized many benefits:

Kerberos

Exchange 2003 uses Kerberos delegation when sending user credentials between an Exchange front-end server and Exchange back-end servers. In previous versions of Exchange, when users opened applications such as Outlook Web Access (OWA), Exchange used Basic authentication to send the user's credentials between an Exchange front-end server and Exchange back-end servers. As a result, companies had to use a security mechanism such as IPSec to encrypt the information.

Exchange 2003 also uses Kerberos when authenticating users of Microsoft Office Outlook 2003.

Forms-Based Authentication in OWA

Exchange 2003 enables a new logon page for OWA that will store the user's name and password in a cookie instead of in the browser. When a user closes the browser, the cookie is cleared. Additionally, after a predefined period of inactivity, the cookie is cleared automatically. The new logon page requires users to enter their domain and network user names and passwords or their full user principal names (UPN), e-mail addresses, and passwords to access their e-mail. This feature is also known as cookie authentication.

User Selectable Security Options in OWA

The OWA logon page allows users to select the security options that best fits their needs. Based on the cookie authentication technology, the Public or shared computer option (selected by default) provides a short default timeout option of 15 minutes. Alternatively, OWA users who are using computers in their offices or homes where they are the sole operators, can select the Private computer option. When selected, the Private computer option allows a much longer period of inactivity before automatically ending the session. Its internal default value is 24 hours. To match enterprise security needs, an Exchange 2003 administrator can customize the inactivity timeout values for both option settings.

Blocking Attachments in OWA

Similar to existing functionality found in Microsoft Outlook 2002 and later, the OWA feature of Exchange 2003 can be configured to block users from accessing certain file type attachments. This feature is useful in stopping untrustworthy attachments from potentially compromising corporate security.

Secure/Multipurpose Internet Mail Extensions (S/MIME) Support in OWA

S/MIME increases the security of Internet e-mail by enabling digital signing of messages as well as message encryption. Digital signatures provide authentication, non-repudiation, and data integrity. Message encryption provides confidentiality and data integrity. Within OTG's configuration, when configured to use S/MIME, private keys are stored in a roaming profile, which is made available when the user logs onto a computer connected to the corporate network. All S/MIME encryption, decryption, and messaging signing operations are performed on the local computer using the private key. All public keys, necessary for non-repudiation and decryption, are stored in the Active Directory. User private keys are never passed, in any form, between the user's computer and the Exchange server.

Restricted Distribution Lists

In Exchange 2003, you can place restrictions on those who can send e-mail messages to an individual user or a distribution list. Submissions can be restricted to specific users, groups, or all authenticated users. Restricting submissions on a distribution list prevents non-trusted senders, such as unauthorized Internet users, from sending mail to an internal-only distribution list.

Improved Security with Clustering

Exchange 2003 clustering, when run on Windows Server 2003, includes the following security features:

Permission improvements mean the Windows Cluster Service no longer requires Exchange Full Administrator rights to create, delete, or modify an Exchange virtual server.

• The Kerberos authentication protocol is enabled by default
• Internet Protocol security (IPSec) support for front-end and back-end servers
• Internet Message Access Protocol 4 (IMAP4) and Post Office Protocol 3 (POP3) services are no longer included by default when creating virtual servers

Improved Recoverability Technologies to Better Meet SLA Requirements

Backing up and restoring large databases or SGs take a long time even over the fastest network connections. However, the coupling of Exchange 2003 with Windows Server 2003 offers an alternative solution that takes a small fraction of the time needed by tape media methodologies for backup and restore.

Volume Shadow Copy Service (VSS) Integration Framework

VSS, a feature of Windows Server 2003, provided OTG with the ability to perform online snap and clone functions on the databases. This allowed OTG to have a mirror copy of the data in a single point-in-time. VSS enables OTG to get either a mirror copy or a snap copy of the production data. Depending upon the type of failure, be it a mailbox store, an SG, or multiple SGs affected by corrupted data, or a massive spindle failure where the entire data structure is lost, OTG can recover upwards of 800 GB of data in minutes, as opposed to standard restoration methodologies that would take many hours to recover that amount of data.

Recovery Storage Group (RSG)

The new RSG is a specialized, offline SG that can be created alongside the standard SGs on the production server in Exchange. RSG provides added flexibility in quickly restoring mailboxes and databases. With this new feature, a damaged Exchange database can be quickly restored in an offline mode to a production server in an offline status. Once the database has been restored to the RSG, the Exchange tool ExMerge can be used to export the contents from one or more mailboxes back into production. RSG eliminates the need for dedicated restore servers for single mailbox restore operations, thereby reducing server downtime.

Mailbox Recovery Center

The new Mailbox Recovery Center makes it easy to perform simultaneous recovery or export operations on multiple disconnected mailboxes. This is a significant improvement over Exchange 2000, where such operations had to be performed individually on each disconnected mailbox. With this new feature, you can quickly restore Exchange mailboxes, and thereby reduce downtime.

Mobility Features/Enhancements

Significant enhancements were made in Exchange 2003 for the mobile, client-side experience. All of the mobility features previously found in Mobile Information Server 2002 (MIS), a separate, adjunct solution to Exchange 2000, were incorporated into Exchange 2003.

Outlook Web Access (OWA)

The new version of OWA in Exchange Server 2003 represents a significant upgrade from OWA in Exchange 2000. The new version is a full-featured e-mail client, with support for rules, spelling checker, signed and encrypted e-mail, and many other improvements. A redesigned interface provides an enhanced user experience similar to that of Outlook 2003, including a new Reading Pane (previously called the Preview Pane in Outlook) and an improved navigation pane.

For OWA users connecting by means of either dial-up, low bandwidth wireless networks, or by using Secure Sockets Layer (SSL), the new use by Exchange 2003 of data compression technology provides substantial overall performance improvements compared to those realized from using previous versions of Exchange Server. Additional performance improvements were attained by the elimination of all ActiveX controls required to use OWA on client computers connecting to Exchange 2003. When using earlier versions of Exchange Server, these controls, when not available in the client computer's Internet Explorer cache, had to be downloaded each time OWA was run.

Outlook Mobile Access (OMA)

Exchange 2003 now includes the OMA application previously offered in MIS. OMA allows users with browser-based mobile devices to use mobile devices to access their e-mail, Contacts, Calendars, Tasks, and to search the global address list. Users can use OMA with a mobile device that has a mobile browser.

MIS had to be installed in every network domain where these services were needed. Since Exchange 2003 comes with built-in mobile services, installation on network domains is no longer necessary.

Furthermore, Exchange 2000 users were limited to using only the MIS servers located in their home domains. Users from a domain within the Microsoft corporate network in which the MIS server was off-line could not use the MIS servers from other sub-domains to access these services.

Exchange 2003 has eliminated the domain boundary limitations for OMA. Any user enabled for OMA use can use mobile services on any of the front-end servers, regardless of their network domain. As an added benefit for OTG, if one region's Exchange front-end servers had to be taken offline for service, the user could still access those services from the remaining servers on the network, thereby all but eliminating downtime for this service.

Exchange ActiveSync (EAS)

The Exchange ActiveSync feature previously offered in MIS server, which enabled users to securely and remotely synchronize their mobile devices directly with the Exchange server, has also been incorporated into Exchange 2003 and enabled by default. By synchronizing a mobile device to an Exchange server, users can access their Exchange information without having to be constantly connected to a mobile network. In addition, users are no longer subject to the same EAS domain boundary limitations that affected OMA in MIS.

Up-To-Date Notifications

Exchange 2003 introduces a new feature within EAS called up-to-date notifications. In the past, the push notifications featured in MIS used the Short Messaging Service (SMS) of a wireless carrier for sending text messages consisting of the first 160 characters of a redirected e-mail. Since SMS used non-encrypted text to transmit its messages, the security of message content was a major concern. Instead of transmitting the first 160 characters of the actual message, up-to-date notifications transmits only a binary command to the mobile device that causes it to start securely synchronizing e-mail over the SSL-protected EAS link. This way, the binary command never contains any portion of the message body yet the user still receives the latest e-mail.

To reduce the amount of traffic a device might receive for a user who regularly receives large quantities of e-mail, Windows Mobile 2003 devices offer the user the option to either specify time ranges during the day called Peak Time in which the synchronization only occurs at specified intervals or synchronize continuously at all times. During Off Peak Time, however, the mobile device is synchronized by up-to-date notifications every time a message arrives. Support for up-to-date notifications requires the use of Windows Mobile 2003 devices such as Pocket PC Phone Edition devices or Smartphones.

Office 2003 Integration

Exchange 2003 is more tightly integrated than ever with its primary client application, Outlook 2003. The combination of the two offers users many enhancements.

Exchange Cached Mode

The use of Exchange cached mode, a feature of Microsoft Office Outlook 2003, enables the user to work in a messaging environment with a perceived connection between the Outlook 2003 client and the Exchange Server. Exchange cached mode isolates the client from most network and server latencies that, in the past, have caused Outlook to appear as if it had stopped responding. Outlook, using Exchange cached mode, connects to the Exchange Server and automatically downloads all incoming content, such as e-mail, meeting requests, and tasks to a dedicated .OST file, which serves as a local cache on the client computer. Once the download has completed, the user can read, reply to, create new, and delete e-mail as well as sending tasks and meeting requests. Outlook, working continuously in the background, connects the local cache file to the Exchange Server to upload the new outgoing content and download any additional new incoming content. Users typically do not notice any difference in messaging performance when using Exchange cached mode, other than the clear benefit of being free of slow network connections or poor server performance.

Exchange cached mode, a feature of Outlook 2003, is supported under both Exchange 2000 and Exchange 2003, but several performance improvements have been implemented specifically to enhance the performance of Outlook 2003 clients when used in conjunction with Exchange 2003.

Exchange cached mode is considered a key requirement toward the Exchange Server consolidation effort. Exchange cached mode will prevent regionally located users from suffering from the effects of system latency when working with Outlook over WAN links connected to remote mailbox servers.

Data Compression

To reduce the amount of information sent between the Outlook 2003 client and Exchange 2003 servers, both Exchange 2003 and Outlook 2003, when working in tandem, perform data compression that significantly reduce network traffic. OTG found that it reduced the total Exchange 2003-Outlook 2003-related network traffic by an average of 40 percent. Exchange 2003 also reduces the total requests for information between the client and server, thereby optimizing the communication between the client and the server.

This significant level of data compression between client and server helped OTG mitigate the effect of additional WAN usage generated when local servers were consolidated onto regional servers. What was formerly all SMTP network traffic locally has now become all Messaging Application Programming Interface (MAPI) Remote Procedure Call (RPC) network traffic across the WAN, but the quantity of that traffic was significantly reduced when compared to traffic generated by previous versions of Exchange and Outlook.

Remote Procedure Call (RPC) over Hypertext Transfer Protocol (HTTP)

Exchange 2003 and Outlook 2003, combined with Windows Server 2003, support the use of RPC over HTTP to access Exchange. Using the Microsoft Windows RPC over HTTP feature enables the secure use of Outlook 2003 over the Internet without setting up a virtual private network (VPN) tunnel with remote access or using OWA. Outlook always communicates with the Exchange server using RPC. When Outlook is configured to use this new feature, it will, by default, first attempt to connect to its corporate Exchange mailbox server by means of RPC over Transmission Control Protocol/Internet Protocol (TCP/IP) as it would in a corporate network setting. If the server cannot be located this way, then Outlook attempts to connect to its corporate Exchange mailbox server by means of RPC over a secure HTTP link on the Internet using SSL. RPC over HTTP comes through the same Exchange front-end servers that serve users of OWA, OMA, and EAS. Effectively this service is identical to OWA to the Exchange back-end servers, but instead of using Internet Explorer as the e-mail client, the e-mail client is Outlook 2003. Similar to OWA, if the RPC connection is made through the Internet, users are prompted to enter their network logon credentials before access to the Exchange Server data is granted.

Note   The feature named RPC over HTTP actually uses Secure Hypertext Transfer Protocol (S-HTTP) over an SSL connection.

Users who use notebooks as their primary Outlook computer will find this feature to be especially useful. Users who travel to customer sites and often end up waiting for the opportunity to make presentations can use RPC over HTTP to keep in touch with their corporate Exchange server without the need for a VPN connection. RPC over HTTP enables a user to make a connection through firewalls at customer sites (which typically block VPN connections) to the corporate Exchange Server, thereby improving their accessibility and productivity.

Unlike OWA, the contents of locally stored personal folder files are available in Outlook on a remote connection in exactly the same way they would be while connected to the corporate network in the office.

Note   Unlike OWA, RPC over HTTP downloads e-mail information when the user connects to the Exchange Server (assuming the use of Outlook cache mode). Therefore, RPC over HTTP should only be used on computers the user controls, such as corporate notebooks, instead of on shared computers or public kiosks.

OTG is optimistic that the use of RPC over HTTP will reduce the number of VPN servers required to meet the needs of the company. Most employees use VPN to connect to the corporate network primarily to use Outlook. To quantify the level of VPN usage, OTG is analyzing the matter to better understand employee needs in an effort to reduce the number of VPN servers deployed without reducing needed connectivity services.

Exchange 2003 Architecture Design Decisions

The successful OTG deployment of Exchange 2003 required the integration of many disparate elements. Not only was the Exchange server software new, but it also required the addition of other new technologies, such as server and storage hardware from third-party sources and Microsoft Windows Server 2003 and Microsoft Office 2003 software, for OTG to gain the maximum benefit from the deployment. Design considerations for the network, including bandwidth requirements and SLA agreements for backup and restore, were also considered. Because of the design decisions made, the resulting changes also led to operational changes in OTG.

Topology

OTG used the topology from the Exchange 2000 on Windows 2000 Server as its basis for designing the topology in the Exchange 2003 deployment. Active Directory was a key element in the organizational structure and administrative requirements for Exchange 2000. OTG was able to use the existing Active Directory structure for the Exchange 2003 deployment.

OTG was already deeply involved in the deployment of Windows Server 2003 in its worldwide network infrastructure when the initial deployments of Exchange 2003 began. This development was critical, for while Exchange 2003 can run on Windows 2000 Server, Exchange 2000 cannot run on Windows Server 2003. Running Exchange 2003 on Windows Server 2003 presents many additional benefits to Exchange, which are discussed in detail later in this paper. Those benefits enabled OTG to begin implementing plans for consolidating the number of servers in the messaging infrastructure worldwide, which drove the design for the Exchange 2003 topology.

For more information about OTG's Exchange Server 2003 topology, see the iT Showcase technical white paper titled, "Exchange 2003 Site Consolidation" at http://www.microsoft.com/technet/itsolutions/msit/default.mspx.

Mobility Design and Configuration

The definition of mobility at Microsoft has grown to include systems not typically associated with mobile technologies. Devices using OTG's mobile infrastructure include more than just Pocket PCs and Smartphones. Microsoft employees using notebook computers or Tablet PCs running Outlook 2003 can use RPC over HTTP to access the Microsoft corporate Exchange servers with just an Internet connection. Any remote, Internet-accessible computer can serve as an OWA client for Microsoft employees. All of these technologies go through the same mobile infrastructure to access Exchange 2003.

The mobility enhancements in Exchange 2003 enabled OTG to modify the design of its mobile messaging infrastructure with additional server consolidations and improved security. The mobility infrastructure in OTG includes such services as OWA, OMA, EAS, RPC over HTTP, and up-to-date notifications.

Consolidation of Front-End Servers

In addition to the mailbox server site and server consolidation project, Exchange 2003 has also enabled OTG to consolidate its mobility server infrastructure (also known as Exchange front-end servers). OTG no longer has to deploy a multiple-server infrastructure within each domain to provide mobility services. Deploying OWA and MIS with Exchange 2000, on the other hand, required an Exchange front-end server dedicated to OWA and separate servers for MIS. By using Exchange 2003, all the mobile messaging features reside on one physical front-end server, enabling OTG to consolidate the number of front-end servers dedicated to hosting mobility features.

OTG reduced its server population from seven OWA servers and seven MIS servers (one set for each domain in the Microsoft corporate network) to seven Exchange front-end sites hosting OWA, OMA, EAS, and RPC over HTTP services. Each Exchange front-end site worldwide hosts a pair of non-clustered, network load balanced Exchange front-end servers. While OTG theoretically could have consolidated to a single set of Exchange front-end servers, the project team decided to retain the larger number due to the network latency that is caused by the great geographic distances between Exchange front-end servers and regional Exchange mailbox servers. If OTG had consolidated to a single set, user performance would have suffered. Network latency would have been particularly evident among those users with slow Internet connections or mobile devices.

Mobile Security Enhancements

OTG also used the enhanced security features for OWA offered in Exchange 2003 for its front-end server deployment, such as time-based logoff and forms-based authentication. Unlike OWA under Exchange 2000, a secure, HTML forms-based, authentication screen appears when a user navigates to a front-end server instead of an NTLM-based dialog box. In addition to logon credentials, the form asks two additional questions:

1. Is the user logging on from a public kiosk/shared computer or from a private home computer?
2. Does the user want to use basic or premium OWA user interface (UI) feature sets? (The answer typically depends on whether the connection is a fast or a slow data link.)

All of the UI elements displayed in the OWA logon page are customizable, enabling the inclusion of company logos, specific URLs to regional front-end servers, custom usage instruction text, and more. OTG created its customized OWA page using these features.

Once the form has been filled out and the user clicks Log On, the data is encapsulated and sent by means of an SSL connection to the front-end server specified by the user when they navigated to the specific server to bring up the authentication form. Once the logon credentials have been sent over the Web, a special time-out cookie is created on the local client computer. Depending upon whether the user indicated the client is a public or private computer, the time-out cookie starts counting up to a threshold of inactivity. Once that threshold is met with no activity having taken place for that duration, the session connection is automatically closed, and requires reauthentication if the user wants to regain access to the Exchange mailbox. OTG configured the time-out cookie to close out inactive sessions on public or shared computers after 15 minutes, whereas inactive sessions on a user's private home computer were configured to last for two hours of inactivity before closing. The session time-out periods are enterprise customizable to meet any security requirements.

In order to provide an additional level of security, OTG chose to deploy Internet Security and Acceleration (ISA) servers to act as the reverse proxy for all Exchange front-end servers. This allowed the front-end servers for Exchange 2003 to be placed behind the firewall, safely within the corporate network, no longer directly connected to the Internet.

Server Design and Configuration

In designing the server platform for its Exchange 2003 deployment, OTG considered a variety of factors. Aside from the normal hardware issues of system reliability and vendor support, the key technical issues considered included new processor technology, cluster implementations, server designs, and mobility issues. As a result, OTG has moved all its Exchange mailbox servers to running in a clustered environment.

Processors

Processor technology continues to advance, improving performance in processing speeds, increasing the number and enlarging the size of on-board caches, and increasing the number of tasks that can be processed in parallel. Most of the servers of the Exchange 2000 infrastructure were based on Intel Pentium II and Pentium III processors running in the 500 to 700 MHz range, with a 100 or 133 MHz front-side bus (FSB).

Given the advances in processor technologies since OTG's deployment of Exchange 2000, OTG chose to deploy Exchange 2003 on new systems based on the Intel Xeon Processor MP Hyper-Threading processors employing a 400 MHz FSB.

Hyper-Threading enables a single processor to process information as if it were two separate processors sharing the same memory bus and cache. In effect, the four-processor, Hyper-Threading servers implemented by OTG functionally serve as virtual eight-processor servers. However, a processor equipped with Hyper-Threading technology does not offer the same performance benefits as a genuine dual-processor system. Because Hyper-Threading processing shares the same on-chip memory cache and main memory bus, OTG has measured an actual Exchange performance increase benefit of approximately 25 percent higher than that of a normal, non-Hyper-Threading processor of the same clock speed.

Clustered Server Design

All the new servers OTG purchased to host Exchange 2003 mailbox servers were set up as clusters and equipped with Xeon Processor MP microprocessors.

Through a combination of Exchange Server 2003, Windows Server 2003, third party SAN technology, and faster servers, OTG decided to create a clustered server design that offers greater operational reliability and a reduction in administrative overhead. Their design choice allowed them to achieve the following specific benefits:

• Reduced service outages by having active node mailbox servers automatically failover to passive node servers.
• Clustered Exchange Virtual Server (EVS) failover performance of just two minutes was achieved, regardless of the amount of the mailbox data contained within the SAN attached to the failed node.
• Increased the number of EVSs as well as the number of supported SGs per EVS within the cluster. Each SG was configured to use three LUNs. Volume Mount Points were used with these LUNs to minimize the number of drive letters used.
• Enabled server consolidation by hosting many more mailboxes per server.
• Reduction in administration and maintenance overhead by consolidating more than 113 mailbox servers in 75 locations into 38 servers in seven locations.
• Reduced potential server outage impact to users (previously six hours or more per user) from a database restoration.
• Improved backup and restore times to less than one hour.
• Achieved server availability of 99.9 percent with a fiscal year 2004 SLA goal of achieving 99.99 percent.
• Enabled the implementation of rolling upgrades to minimize the impact of service outages while speeding up server operating system and application upgrades and patching.
• Doubled the user mailbox limit (to 200 MB)

OTG's design goal was to support 8,000 mailboxes per SAN, with 200 MB mailbox limits, 99.99 percent cluster server availability, and less than one hour per database backup and restore time. The scaling of the data center EVSs in the Main corporate forest was designed to reach 4,000 mailboxes.

Multi-Node Cluster Design

OTG chose to use a multi-node cluster design using multiple active (online) and passive (offline) nodes. This design enables a failed active node to be immediately replaced by an identically configured passive node and for the resources of the failed active node, such as storage, to be immediately transferred to the passive node, thereby insuring that the end user experience is minimized by the failover.

OTG implemented two separate types of passive nodes: primary passive nodes and alternative passive nodes. A primary passive node is a server using equivalently equipped hardware to the active node servers. This allows for full functionality upon an active node failover. The alternative passive node is a server equipped with lower-scaled hardware that is used primarily for tasks such as streaming backup data from disk to tape. It also serves as a reduced performance failover server. Both types of passive nodes are leveraged for rolling software upgrades.

OTG's multi-node cluster design employs both primary and alternate passive nodes. Unlike primary passive nodes, alternative passive nodes are smaller servers primarily designed to carry out disk-to-tape backup tasks. OTG uses all of the passive nodes in the cluster when rolling upgrades of the operating system and/or Exchange are required. Instead of failing an active node to the primary passive node, upgrading the offline active node, then restoring the upgraded node to active status again and rolling through this cycle for every active node in the cluster, OTG's deployment of alternative passive nodes in conjunction with primary passive nodes speeds up the process. OTG first patches all the offline passive nodes, then fails over the number of active nodes equivalent to the number of available passive nodes. These offline nodes are then upgraded in parallel and restored to service when ready. This process is repeated once to upgrade the one remaining active node server.

OTG Cluster Designs

OTG implemented two primary cluster designs for the Exchange 2003 deployment in the Main corporate forest: a regional design and a headquarters data center design. A separate, scaled validation design was also deployed in the Level B Test limited-use production forest. All used the multi-node, Active/Passive cluster design. Table 4 shows the OTG cluster configurations.

Table 4   Cluster design specifications per deployment

• Regional Design. The server specification for the regional cluster implementation consists of one SAN enclosure per cluster, with three active nodes, one primary passive node, and one alternate passive node (designated as AAAPp).
• Headquarters Design. The headquarters clustered implementation is similar in design. It consists of two SAN enclosures, four active nodes, one primary passive node, and two alternate passive nodes (designated as AAAAPpp).
• Level B Test Forest Design. The Level B Test server specification is similar to the regional cluster in design but with greater mailbox capacity. It consists of one SAN enclosure, one active node, and one primary passive node (designated as AP).

To get the best performance at the best price point, OTG standardized on the four-processor, 1.9 GHz Intel Xeon Processor MP server for its active and primary passive cluster nodes for both regional and headquarters data center deployments. For alternative passive cluster nodes, OTG uses two-processor 2.4 GHz Intel Xeon Processor MP servers. Because of this new processing platform, OTG has seen substantial performance improvements in its Exchange 2003 infrastructure.

OTG's cluster design supports a significant increase in both the number and size of mailboxes per Exchange server. It helps eliminate performance impact to users during the second stage backup process because it offloads that stage of the backup process to non-active servers within the cluster, thereby maintaining the SLA.

Storage Design and Configuration

The entire design of OTG's storage configuration was based on effectively managing peak time disk I/O. OTG studied the usage trends of its Exchange 2000 messaging storage infrastructure and learned that the peak period of usage is typically Monday mornings. OTG took that usage data and made it a baseline for designing the Exchange 2003 SAN solution. OTG calculated the average amount of peak time disk I/O per second attributed to each mailbox. They calculated the total I/O rate for a server as the product of the number of mailboxes multiplied by the I/O rate.

For example, on a server supporting 4,000 mailboxes with a peak time I/O rate of 1.2 per mailbox per second, the total I/O rate for that server equates to 4,800 I/Os per second. The amount of data in each I/O transfer in Exchange is four KB, which at that I/O rate, equates to nearly 20 MB of I/O per second. Add to that the fact that each SAN enclosure serves two hosts in the headquarters data center configuration, the I/O rate doubles to nearly 10,000 I/Os per second.

In OTG's design for meeting this demand, each SAN enclosure selected by OTG can support up to 12,000 I/Os per second, affording a margin of headroom for unusual spikes in activity but expected to perform adequately in normal peak periods of I/O activity. Any significant load beyond this would likely result in disk read and write latencies, which would adversely affect the performance of all the mailboxes attached to that SAN. OTG system architects deemed this an acceptable risk, given anticipated conditions, the cost of additional hardware, and monitoring and alerting improvements in Microsoft Operations Manager.

To determine the messaging storage requirements for any enterprise, one must measure average peak time I/O per mailbox user per second, the maximum size of mailboxes, the length of time items are retained in deleted item retention, and the typical usage e-mail patterns turnover rate in an organization. These are the factors OTG considered when designing their Exchange 2003 SAN solution.

OTG allocated additional capacity to each LUN supporting mailbox stores in an attempt to mitigate any requirement for future resizing based on unexpected growth. The LUN was sized to support six and half production databases with a "fluff factor" of 1.4.

Fluff factor is what OTG refers to as the average capacity allocation to support a given mailbox on disk based on deleted item retention, database overhead, non-limited mailboxes etc.  For example, creating 100 MB mailboxes for users on Exchange 2000 actually required them to reserve 140 MB of space per user. The value of 1.4 was trended over the years on production Exchange servers supporting 100 MB mailboxes and was maintained as a basis for designing the new solution with support for 200 MB mailboxes.

OTG's 100 MB mailbox size limit was a hard and fast disk quota limitation set and enforced at the Exchange level by means of policy, but if the user consumed the entire 100 MB of available space, it was often because they had exceeded the amount on the back end. This usually happened when a user deleted e-mail from a mailbox. The e-mail was actually not immediately deleted from the mailbox database on the server. Rather, it was temporarily retained in the database, held in a space known as deleted item retention. Only after three days was the deleted e-mail actually purged from a mailbox database. OTG needed to account for that level of usage overhead when planning its storage needs for Exchange 2003.

Additionally, OTG sized each data LUN to support six and a half databases even though they would only support five in production. This allowed them to duplicate a single corrupted database on the same LUN and then run an integrity check on it. This ability to use the same LUN enabled OTG to provide the fastest possible response to database corruption.

Selecting a SAN

Like many organizations, OTG decided to make a clean break from the paradigm of local (host-based) direct attached SCSI storage to SAN-connected storage. In the past, server storage was treated as a key server component that was closely married to the server hardware. SAN technology has made storage become more like a utility service; it is no longer as closely tied to the server. While this arrangement has both pros and cons, OTG opted for SAN storage because it meets OTG's requirements for future performance, scalability, and capacity. Those requirements could not be satisfied by locally attached storage arrays.

The deployment of Exchange 2003 gave OTG the opportunity to assess how SAN technology had matured since it had last been studied. OTG embarked on a project to qualify and test technology and products from SAN vendors. OTG required that any new SAN technology standard implemented at Microsoft needed to be easily supported in remote locations. OTG required that a storage solution be easy to deploy, modular in design, and remotely manageable.

Within each HP StorageWorks Enterprise Virtual Array 5000 (eva5000) SAN used by OTG are 168 disks. Each SAN enclosure supports approximately 8,000 200 MB mailboxes. Each SAN enclosure has the ability to process about 12,000 I/Os per second before disk latency becomes evident. Each mailbox server in the headquarters data center will support 4,000 mailboxes and is expected to process a peak-time load of between 5,000 and 6,000 I/Os. As a result, one SAN enclosure supports two mailbox servers in the headquarters data center. Regional mailbox servers will support just under 2,700 mailboxes, so the resultant peak-time load of three regional servers is supported by one SAN enclosure.

Storage Allocation Using Volume Mount Points

OTG used the new cluster support for volume mount points in Windows Server 2003 to eliminate the drive letter as a scalability blocker towards bringing on multiple Exchange instances within a single cluster. The chosen design used a drive letter assignment per data LUN (one per SG) with four data LUNs per cluster node (OTG configures each node to support one Exchange Virtual Server). The corresponding log LUNs were configured as volume mount point clustered resources, each of which was dependant upon its parent data LUN. Included in the design is a dedicated Queue LUN that is also maintained as a volume mount point clustered resource, which is dependant on the data LUN assigned to SG1.

The use of volume mount points allowed OTG configure an optimized disk layout using four drive letters to maintain nine physical LUNs. This design allowed for the creation of four Exchange instances that mapped across thirty-six physical LUNs utilizing only sixteen drive letters.

Subsequent LUNs were maintained to support online backup-to-disk with a single disk allocated per SG per node. The disk assigned for SG1 per node support three additional volume mount point LUNs as backup targets for SG2, SG3, and SG4. The backup resources were configured across sixteen physical LUNs addressable by four drive letters.

A representation of the drive letter allocation for the first node and corresponding allocation to support the online backup devices is given in Figure 1.

Figure 1   Drive letter allocation per node.

Note   In the context of Figure 1, VMP represents a volume mount point.

In all, a total of 53 physical LUNs are addressable using 21 drive letters within the clustered design. This allows for easy disk subsystem optimization with LUNs distributed across controllers and Fibre Channel Adapters (FCAs) to ensure peak disk transfer requirements are met as required within the Microsoft production environment.

Redundant Storage System Paths Using Secure Path

OTG's deployment of SAN technology includes an I/O design that not only provides redundancy but also uses that redundancy for optimal data flow.

OTG uses HP StorageWorks Secure Path for Windows to provide many benefits within its SAN infrastructure. Secure Path provides three key benefits:

1. Eliminates the risk of a single point of failure supporting the server and SAN interconnect.
2. Allows for LUN distribution to maintain optimized I/O required on a busy Exchange host, reducing peak read/write disk latency and substantially improving online backup throughput to disk.
3. Insures only single LUN presentation independent of the number of paths to the host.

OTG's implementation of Secure Path uses two FCAs per host, two fibre channel data switches, and two storage controllers. Each FCA, switch, and controller group makes up what is known as a fabric. Secure Path allows the use of two separate fabrics per SAN, and each element of the fabric is interconnected with subordinate elements from both fabrics. More precisely, each active node host in a cluster connects to each switch by means of the two FCAs installed in each host (one FCA per switch). Each switch takes inbound data from each host and has two outbound data connections, one to each controller. Each controller has two inbound data connections, one from each switch, and has one outbound data connection to the SAN enclosure. Secure Path enables OTG to be operationally tolerant to a single component failure in an FCA, a connecting cable, a switch, or a controller. Service performance would be affected in the event of a component failure, but it would be able to continue to operate seamlessly.

Secure Path also assists with eliminating many single points of failure between the nodes and the connected SAN storage. OTG can maintain service in the event of a component failure affecting a single FCA per host, multiple fiber cables, fiber channel switches, or a single storage controller that makes up the SAN fabric. The component failure is detected by Secure Path, which ensures that I/O is maintained by moving LUNs from the failed path to an available path. This process, called failover, requires no resource downtime while maintaining LUN availability. Failed-over LUNs can be failed-back using HP's Secure Path Manager to restore optimized I/O, once failed components have been replaced.

The headquarters data center cluster implementation using Secure Path to connect to a 16,000 mailbox SAN is shown in Figure 2.

If your browser does not support inline frames, click here to view on a separate page.

Figure 2   Secure Path Connecting a Data Center Cluster to a Pair of SANs

Backup and Recovery

With the implementation of Exchange 2003 in a clustered server environment, OTG designed a two-stage backup process (disk-to-disk and disk-to-tape) to meet its SLAs better. This process prevents the tape backup process from affecting the production server performance, and provides greater flexibility in managing the data restoration process. The solution is based on a combination of:

• Exchange Server 2003
• Microsoft Windows Server 2003, Enterprise Edition
• Windows NT® Backup for disk-to-disk backup
• Veritas Storage Management solution for disk-to-tape backup

In the past, it was challenging to maintain the one-hour backup restore SLA on direct attached SCSI storage server implementations. These server designs used a one-step backup process (disk-to-tape), where backups were performed to tape libraries over the Gigabit LAN. OTG's experience showed that they could move data at a rate of approximately 36-37 MB per second, or about 33+ GB per hour. Backups were limited to non-business hours to minimize any impact to clients with mailboxes hosted on these servers. However, if a backup failed to complete by 7 A.M., it had to be canceled. Otherwise, the continuing backup process would have a significantly negative impact on the system performance of the messaging infrastructure for clients.

Recovering a mailbox store affected by corruption in Exchange 2000 meant that 1,000 mailboxes were out of service for six or more hours during the restore operation. This represented a cost in lost productivity of $60-$80 per hour per user. Single mailbox restore operations required dedicated restore servers. This configuration is shown in Figure 3.

If your browser does not support inline frames, click here to view on a separate page.

Figure 3   Previous Regional Messaging Backup Environment

Two-Stage Backup Solution

To solve these problems and support server consolidation, OTG designed a flexible, two-stage process to backup data within a multimode clustered configuration—disk-to-disk (stage 1) and disk-to-tape (stage 2).

OTG leveraged the fact that resources within a cluster resource group can move within that resource group independent of other resource groups. For example, an active node of a clustered Exchange server is attached to a separate cluster resource group of dedicated backup LUNs in addition to the resource groups used for storing production data,.

In the first stage, backup runs on all active nodes within the cluster to complete an online, disk-to-disk backup from the LUNs in the production data resource groups to the LUNs in the backup resource group over a direct attached fibre channel. The backup resource group has the capacity to support two-day online retention. Once that process has completed, the control of the LUNs in the backup resource group is transferred to an alternative passive node. At this point, passive node initiates the second stage, disk-to-tape backup from the backup resource group to the tape library over a direct attached fibre channel. This process frees up the active nodes from the time consuming disk-to-tape data transfer, thereby minimizing the amount of time required of the active nodes for processing data backup operations. This process is shown in Figure 4.

If your browser does not support inline frames, click here to view on a separate page.

Figure 4   Two-stage Backup Process

OTG elected to use this two-stage process rather than using a single stage, disk-to-tape backup over a direct fibre attachment to a tape library. While the single-stage process would eliminate the need for backup LUNs in the SAN, which would free up additional storage capacity in the SAN for more mailboxes, OTG realized that it could not take the risk of losing valuable production time in the event that the node in the cluster might become disconnected from the tape library. If that happened, the node server would be required to reboot to reattach the server to the library. If the active node were the server performing this work, OTG would be required to failover the node so it could reboot and reconnect to the library. OTG considered that an unacceptable risk to system availability. Instead, by placing the burden of backing up to tape on a passive node that does not support users, no loss of production service occurs when the passive node needs to be rebooted to restore the server-to-library connection.

Per-database online backups are scheduled at regular intervals that let OTG back up each entire server between 8:00 P.M. and 1:30 A.M. The databases are backed up concurrently per SG. An important feature here is that Exchange 2003 allows parallel backup and restore operations on a per-SG basis. Therefore, backup operations for each database can be interleaved.

Recovery Solution

With OTG's new clustering solution, a server hardware failure is simply a matter of an automatic cluster node failover; service is negligibly affected. If there is a disk failure, different recovery scenarios are implemented, depending upon the scope of the failure and the time of day at which it occurs.

Methodology is No Longer Scenario-Dependent

The method of recovery employed used to b