Strategies for MS Exchange Service Packs and Version Upgrades
Maureen Tracy Venti, Microsoft Consulting Services, Boston, and
Matthew Finger, Microsoft Consulting Services, Minnesota
AT A GLANCE
Key Point: Thorough testing and pilot phases can improve deployment efficiency and reduce support costs.
Detail: Medium | Task: Development, Deployment |
| Article Section | What's There |
The Three Types of Enhancement | Describes service packs, hot fixes, and new versions or releases. |
Strategies for Service Pack Deployments and Version Upgrades | How to balance the need to keep the production environment up to date against the allocation of testing and operations resources |
Testing Service Packs, New Releases | Guidelines for testing and evaluating risks. |
Using SMS to Deploy | Discusses the Package Command Manager and how to install and deploy using SMS. |
This article offers tips on deploying hot fixes and service packs, and on how to decide when to upgrade to a newer version of software. The discussion explains how to enhance, correct, or extend your system functionality while minimizing adverse impact to the Windows NT Server and Exchange Server messaging environment.
The Three Types of Enhancement
Service Packs
Service packs correct known problems and provide tools, drivers, and updates that extend product functionality, including enhancements developed after the product released. Exchange and Windows NT Server service packs can be released several times per year, and Exchange service packs can contain server and client updates. Service packs go through broad regression testing on Microsoft's internal Exchange Network, which handles about 3 million messages a day.
Hot Fixes
Hot fixes solve problems reported by customers. Before being released, hot fixes are extensively reviewed and tested by Microsoft and verified by the users who reported the problem. They are not regression tested. Hot fixes are very specific: you should apply one only if you experience the exact problem they address and are using the current software version with the latest service packs.
If Microsoft finds a significant problem affecting a wide range of customers, the hot fix is widely publicized with the suggestion that everyone upgrade. All hot fixes released after a service pack has been released are incorporated into the next service pack.
New Version/Release
New releases of Windows NT and Exchange Server are available about every 12 to 18 months, with service packs and .5 releases in between. These contain many new features as well as all fixes and features delivered in previous service packs.
Exchange Server and client versions interoperate completely, although older versions may require a service pack before an upgrade. For example, Exchange 4.0 requires SP2 to interoperate with Exchange 5.0 and 5.5, but Exchange 5.0 does not require a service pack to interoperate with Exchange 5.5. Exchange 5.5 servers will support Outlook 8.01 and 8.02 clients. Outlook version 8.03 is optimized for use with Exchange Server 5.5, but will operate with Exchange versions 4.0 and 5.0. Upgrade requirements and information on interoperability are always published in the product documentation and release notes. A new version of Exchange may require that you apply a Windows NT Server service pack before installing the new version of Exchange.
Strategies for Service Pack Deployments and Version Upgrades
Given the release schedule for service packs and version upgrades for both Windows NT Server and Exchange, you may decide it is unrealistic to perform constant testing and enterprise-wide updating of production server. This section outlines a strategy for balancing the need to keep the production environment up to date against the allocation of testing and operations resources.
General Guidelines
Strategies for software and hardware upgrades are central to managing and maintaining your production environment. Set aside time to test and evaluate service packs and new releases. Build a deployment schedule using the projected dates of new software version releases and service packs, which you can get from the Microsoft Web site (http://www.microsoft.com/exchange/) or from your Microsoft account team, including local sales representatives, system engineers, and MCS. Remember when you plan to be flexible, because release dates are subject to change. The point is to start assessing the impact of new releases and service packs as soon as possible so you can stay up to date.
If there are strong business requirements for deploying the latest release into your environment, then you can proceed to testing. The process below assumes you have a test lab fully equipped to model your production infrastructure.
Evaluating Service Packs
If you report a problem to Microsoft Technical Support (MTS) that has already been fixed in a service pack, the solution is pretty obvious. You should review and evaluate every service pack in a lab as soon as it is released, even if you don't plan to deploy it immediately. Source of information on service packs include: TechNet, MTS e-mail, the support Web site http://support.microsoft.com/support/, and the Premier NewsFlash—if you have Microsoft Premier support—an newsletter that previews hot fixes, new releases, and service packs.
If you have neither MTS support service contract nor TechNet, you can get up-to-date information from http://support.microsoft.com/support/, and the latest service packs from http://www.microsoft.com/downloads/search.asp?.
Evaluating New Releases
When new releases are announced, you should assess the effort of upgrading against the effort of deploying the latest service pack. Microsoft previews new software via Beta and Release Candidate programs. Although beta versions do not always have fully implemented functionality, Beta program subscribers can still use them to evaluate features and assess the degree of change the new release represents. The Release Candidate (RC) is very close to final code, and you can use it for integration testing in the lab and in-depth evaluation of the new features and functions.
The goal is to understand what the latest release of a product offers and determine if there are business reasons for deploying it. A test lab is a great advantage in this process. You can use a test lab to assess a new release for your production installation and to see how much effort is required to implement it. Premier Support customers with critical business needs can arrange through their Technical Account Manager (TAM) to install new releases early.
Refer to the article, " Setting Up a Test Environment for MS Exchange ," for more information on test lab procedures.
When to Deploy Service Packs and When to Wait
The basic question is simple: should you install a service pack for Windows NT or Exchange or wait for the next new version, investing your time and resources in deployment planning?
| • | Each service pack has a README.TXT file that describes its contents, reprints the Knowledge Base (KB) articles for all the fixes, describes new features, provides installation instructions, and explains any known issues. You can conduct further research into issues in the Knowledge Base, which is supplied on TechNet and Microsoft's Web site http://support.microsoft.com/support/. Study the README carefully to determine if the service pack will improve your environment or correct a critical problem, and if its benefits outweigh the risks and effort of deployment. These considerations can help you decide: |
| • | If you are not experiencing any of the problems a service pack fixes or if the problems are low-impact, you may decide not to deploy. If you can, however, you should still install it on a couple of servers in your lab to get first-hand installation experience. |
| • | If you are not experiencing problems and a new Exchange or Windows NT release is expected within six months, you might as well wait for the new version. New releases include service packs to date. |
| • | If you are 2 or 3 service packs behind, you should update your environment. Schedule periodic service pack upgrades as part of your operations maintenance and try never to be more than two service packs behind. |
| • | If you get in the habit of always installing service packs in the lab, you are better prepared to deploy them throughout your organization. For instance, if you encounter a serious problem that is not addressed in a current or coming service pack, there probably is a hot fix available. But you can't deploy a hot fix until you have all current service packs installed. Certain recovery operations require that new servers have the same service packs installed as the server they are replacing. If you have installed and tested service packs in the lab, you are ready for all contingencies. |
Testing Service Packs, New Releases
Testing Service Packs
Service packs in general are cumulative. For example, Service Pack (SP) 3 contains all the fixes in SPs 1 and 2. Always use the README file to check the service pack's contents, its installation prerequisites, and the KB articles detailing the issues it addresses. Use this information to develop test plans.
Coordinate the testing and installation schedules for Windows NT and Exchange service packs as much as possible. Any Exchange dependencies on NT service packs will be documented in the README and in the Release Notes and Exchange documentation. Introduce each service pack separately in the lab to assess its individual impact, and if there are no problems test for any issues between the Windows NT and Exchange service packs by installing them at the same time in the lab. A simultaneous installation reduces server downtime. During testing, watch for risks to your environment. Other good ideas:
| • | Install the service pack on at least one lab server to test coexistence with other servers even if the fixes and features are not needed immediately. Day-to-day use in the lab sometimes reveals problems better than specific tests. Lab experience makes it easier to install the service pack in the production environment, and this can be a great advantage if a critical problem requires a rapid SP installation. |
| • | Coordinate service pack testing with other IS groups that may be affected by it. An NT Server service pack may contain Exchange fixes, for example, and these can affect other areas in your organization. For example, suppose a service pack that fixes a problem Exchange has with Windows NT Backup also changes DHCP. (Studying the README file can show you this.) You need to test both the NT Backup fix and the DHCP change and to coordinate testing with the group that supports DHCP so you can get their sign-off prior to deployment. |
| • | Exchange servers can be dedicated to specific functions. Test service packs on lab servers running the appropriate services and configured identically to production servers. |
| • | If the test cycle is successful, start deploying to non-critical servers first, if possible, and then move to the primary servers once the service pack has been in production for 10-14 days. Always check the README file for specific instructions and to see if there is a recommended sequence of which servers to update first. |
Note: Setting up "test" servers in your production environment helps ensure a successful deployment. For example, set up an Exchange server with test mailboxes and apply the service pack to this server first. For NT Server, set up a test WINS server as a replication partner with the hub WINS server. This requires extra servers but reduces user impact if there is a problem. See the table in the Evaluating Risks section below for help in determining which servers to upgrade first.
Testing New Releases
New releases include features that may change the way the product operates. If the current release of a product is meeting your needs and you are not experiencing problems, you may not want to bother with a new release. You still, however, should preview new releases and plan for their implementation. If time and resources are limited, skip the Betas and save thorough testing and evaluation for the RC, which has more complete features and bug documentation.
Here are some considerations for testing new versions:
| • | Production Exchange Servers can be dedicated to specific functions, so you may need different upgrade procedures for different server types. For example, it is easier and entails lower risk to upgrade an Internet Mail server than a dedicated messaging server with 1000 users. Thoroughly test the upgrade procedure in the lab first on every type of server and client. Use the results to develop an upgrade plan for clients and servers that minimizes downtime and interrupts user productivity the least. |
| • | Develop a version interoperability plan for servers and clients so older versions can coexist with newer ones as you upgrade. It can take week or even months to upgrade all servers and clients. As a general rule, upgrade servers first. Repeat the client-server test plan developed for the initial deployment, this time using the client version currently installed running against the new version servers. New features and version interoperability testing require additional test plans. You need to identify version interoperability issues and inform operations groups and business units. |
| • | A new version must be tested against all other production operating systems, legacy mail systems and gateways, applications, and all production client types. If you have integration test plans from the original deployment, repeat them. Add tests for new features and functionality. Your lab should mirror the production environment, especially client and server types. |
| • | Don't forget to test the Exchange Administrator program and any server monitoring tools that you will use across versions. Coordinate upgrades for servers and administrator/monitoring workstations. |
| • | Simulate system loading with tools such as autorun scripts, LoadSim, and other utilities. |
| • | Configure and test all new features, even if there is no immediate need or interest, so that you can understand how the feature works and what its implementation requires. |
Evaluating Risks
Every production upgrade requires that you evaluate possible risks. Basically, you need to determine what can happen if you don't install the SP, and what can happen if the installation itself causes problems. A failed installation is rare but you should develop a backout plan just in case. Each installation is different, users have different business requirements and critical needs. Do your best to determine what risks an upgrade pose for you.
The table below can help you assess the risk associated with upgrading a particular server in a production environment and determine the best server upgrade sequence (if the SP has no requirements). Always check the README file for requirements or recommendations on server deployment.
The analysis process can be summarized as follows: First, review the SP's fixes and note the ones that will benefit you. Weigh the significance of the problem the service pack will fix for you against the impact on the server and your environment if the service pack installation fails. Consider server roles. Generally, servers that host mailboxes are riskier to upgrade because a failure can lose user data. Servers that run only connectors are safer because they automatically load balance, so if one goes offline another takes over and mail continues to flow.
Note: Always do a full backup of the server before applying a service pack or version upgrade. For other precautions see Knowledge base article 165418, Title: Before Installing a Windows NT Service Pack.
| Server (number of servers) | Probability of server failure without SP | Effect if SP install fails | Upgrade Priority [Scale of 1-3 with 1 being as soon as possible] |
News server (2) | Low | Low: not a critical server | 2—Install this server first after hours as a test of the SP. Run for several days before installing the next server. |
Internet Mail Server (IMS) (3) | High: known problem is addressed in SP | Low: no users on server; there are other IMS Servers that will take up the load; connector can be moved to another server or rebuilt with minimal down time | 1—Install on one server after hours and run for several days before installing the other IMS servers. |
Public folder server (6) | Medium: folder replication performance improvements in SP could be beneficial | Medium: users without PF access; applications use public folders | 2--Install if performance begins to degrade below acceptable levels; install after hours |
Dedicated mailbox server with 1000 users each (15) | Low: no connectors running on server; server performance is good. | High: users without mail | 3—Wait until other servers are upgraded and the process has been validated; check on dates for new release before starting. Install over the weekend to limit user impact. |
Microsoft Mail Connector (2) | High: Microsoft Mail Connector problem fixed in SP | High: all message transfer to MS Mail stops | 2—Install over the weekend to allow testing/checkout. |
Bridgehead (2) | Medium: X.400 performance improvements in SP could be beneficial | Low: not much traffic between sites; downtime would have minimal impact | 1—Install after IMS; install after hours. |
The first servers to upgrade are those with a low impact on the environment but a high probability of experiencing problems fixed in the service pack. Monitor high-risk servers for 10 to 14 days before upgrading. Test-install the SP on high impact server types in the lab first to get a feel for the process and to uncover any issues.
New version releases may require that service packs be installed, so develop an upgrade plan for all server types. For example, you decide to deploy a service pack, but not to update the messaging servers until after all the other Exchange servers because you want to build a new production messaging server. Install the new messaging server with the latest lab-tested Windows NT and Exchange versions and service packs that you are deploying, then move mail- or test-team members to this server for a week or so for final checkout before adding users.
Using SMS to Deploy
Microsoft Systems Management Server (SMS) uses the Package Command Manager (PCM) to deploy software to workstations and servers: when a user logs on, the interactive version of PCM checks the SMS logon servers for software packages at intervals specified by the SMS administrator. The interactive version does, however, create some issues for Windows NT Servers, which are not typically used for interactive logins. Unless the target server is part of the SMS hierarchy, the SMS client with PCM won't get installed. As a result, the server will not appear as a software distribution target in the SMS administrator interface. The interactive version also must remain logged onto the network to receive the deployed software packages even with the SMS client software installed.
SMS 1.2 Service Pack 2 includes a new feature called PCM as a Service that performs the same functions as the interactive version of PCM, but performs installations even if no user is logged into the server. This service was delivered with SMS prior to Service Pack 2 but was installed only on servers that were directly involved in the SMS hierarchy. The service pack update allows the PCM service to function correctly on both Windows NT Servers and Windows NT Workstations.
Another advantage of PCM as a Service is that it installs the software using the security context on the service, unlike the interactive version, which uses the logged-on user's security information. (It should install software in the security context of an account with administrative privileges on the server.) Follow the special procedures listed in the product documentation to install this feature on each machine.
Rolling Out Service Packs Using SMS
It takes only a few minutes to copy service pack files to the server and, depending on the nature of the upgrade, either reboot the server or stop and restart Exchange services. But it can take time to update each server, especially if your organization has remote locations with distributed servers.
Automating service pack deployments with SMS takes careful planning, but it can speed up production server updates and help you stay current with the latest fixes. Start by testing service pack installations in the test lab and ensuring that all servers have the necessary SMS software installed. Be sure to include all server configuration types in your testing. Check that your SMS packages and batch files are functioning correctly, and verify that the target server is updated.
After testing and running the service pack on lab servers for several days, finalize the SMS install procedures. Update local, low-impact production servers and run for 10 to 14 days, then install on some local high impact servers before you attempt to install to a remote server. Start the SMS remote deployment with a low impact remote server. Validate that the remote server was installed correctly and then allow it to run for 10-14 days before attempting widespread deployment. You may want to extend the checkout periods for each step, depending on your hardware and software environment, business cycles, and the nature of the servers you are upgrading.
You can verify a successful installation by looking at the Help menu under About or by typing winver in a command line prompt window.
Install the SMS client
If the SMS infrastructure is configured correctly, the client software is installed automatically when an interactive user logs into the machine. Installation places the machine in the SMS inventory database, which enables the machine to be targeted for software distribution. If the client software is installed on the machine but is not in the SMS database, a configuration problem might affect inventory collection. Knowledge Base article, 126642, Title: Troubleshooting Inventory Collection Problems, addresses the most common causes of inventory problems.
If the server cannot be easily accessed to perform an interactive login, you can remotely install the SMS client software with the Scheduler service. It normally runs using the SYSTEM security context, which is a problem because the SYSTEM account cannot easily connect to network resources. If that's the case, change the security context of the service or use a different user account when connecting to network resources.
The following batch file installs the SMS client on a remote server without changing the security context of the Schedule service. Before using this in production, test it and implement error checking.
SCHEDSMS.BAT @ECHO OFF ECHO Copying command batch file. COPY .\INSTSMS.BAT \\EXCHSVR\C$\TEMP\INSTSMS.BAT ECHO Remotely scheduling SMS client installation. SOON.EXE \\EXCHSVR 5 /INTERACTIVE "C:\TEMP\INSTSMS.BAT" ECHO Completed scheduling. INSTSMS.BAT @ECHO OFF ECHO Connecting to the SMS site server. NET USE X: \\SMSSVR\\SMS_SHR PASSWORD /USER:CORP\USER1 ECHO Installing the SMS client software X:\X86.BIN\RUNSMS.BAT ECHO Removing the connection from the SMS site server. NET USE X: /DELETE ECHO Complete
After install the SMS client software, add PCM as a Service by creating a user account with administrative privileges for the PCM service. For example, if the server is in the CORP domain, you must put the service account named PCMSVC into the CORP\Domain Admins security group.
The software for the PCM service is located on the Systems Management Server Service Pack 2 or higher CD in the \support\pcmsvc32 directory. Install the service using the RSERVICE.EXE utility, which remotely installs, configures, and starts most types of Windows NT services. RSERVICE is configured via a *.INI file. The installation documentation located at \SUPPORT\PCMSVC32\INSTALL.DOC has specific usage instructions.
Here is an example RSERVICE.INI file and the command line to execute the RSERVICE utility. This example targets only one server for the installation.
[domain name] CORP=LISTED [machine list] EXCHSVR=INCLUDE [service account] *=CORP\PCMSVC [service name] *=SMS_PACKAGE_COMMAND_MANAGER_NT [executable file] *=PCMSVC32.EXE [installation directory] *=c:\pcmsvc\x86.bin [startup parameters] [source directory] *=\\SMSSVR\d$\SMS\SITE.SRV\X86.BIN [other files] [access permissions] *=Administrators:FULL smsuser:READ [automatic start] *=yes [registry settings] *=key:HKLM\Software\Microsoft\SMS\Identification SZ:"Installation directory=C:\PCMSVC" key:HKLM\Software\Microsoft\SMS\TRACING DWORD:Enabled=1 key:HKLM\Software\Microsoft\SMS\TRACING\SMS_PACKAGE_COMMAND_MANAGER_NT DWORD:Enabled=1 key:HKLM\Software\Microsoft\SMS\TRACING\SMS_PACKAGE_COMMAND_MANAGER_NT SZ:"TraceFilename=\pcmsvc\LOGS\pacman.log" [logfile path] *=C:\PCMINST.LOG
The command line for RSERVICE using the above example file would be:
RSERVICE /INSTALL .\RSERVICE.INI /C
Once installed and started, the PCM service automatically uses the same parameters as the interactive version of PCM and rechecks the SMS logon server for new software packages on the same interval. As soon as you verify that the PCM service is functioning correctly, you can use the service to perform software installations.
Create the Service Pack SMS Package
Now, create the SMS software package by setting up the source files on a network share and defining the package in SMS. The following procedure illustrates how to create the SMS software package for an Exchange service pack. See the SMS product documentation for specific instructions.
1. | On the network share used as the source for your SMS packages, create a directory for the service pack files, such as \EXCH.SP. |
2. | Copy all of service pack files to the newly created directory. |
3. | Using the SMS Administrator tool, open the Packages window and select New from the File menu. |
4. | Assign the package a name and useful description. |
5. | Even though this package will be applied to an Windows NT Server, select the Workstation button to define it as a workstation type package. |
6. | In the Source Location field on the Package Properties dialog, enter the UNC path to the Service Pack source files. For example, \\SMSSOURCE\EXCH.SP. |
7. | In the Command Lines section, click the New button to create a command line for the installation. If different types of installation are required, multiple command lines can be configured and selected at the time of deployment. |
8. | Enter a descriptive name for the installation. For example, Unattended Installation. |
9. | In the Command Line field, enter the command and command line parameters required to start the installation. For example, to install the Exchange service pack the command line would be UPDATE.EXE /U. |
10. | Select the Background Execution check box. This determines which version of PCM will perform the installation. Checking it causes the PCM service to perform the installation. If it's not checked, the interactive version performs the installation. |
11. | Click OK to create the command line. Repeat as many times as necessary to create all of the required command lines. |
12. | Click OK to complete the creation of the SMS package. |
The Service Pack SMS software package is now ready for test deployment. After you verify the package on a test server, you can use it in production without any further changes.
Deploying Hot Fixes
Hot fixes are usually collected and rolled into the next service pack. In between service pack releases, you should track which hot fixes have been applied. When a new server is built or an existing server needs the current service pack reinstalled, you must reapply all of the hot fixes that were previously installed.
To help manage these hot fixes, the Service Pack SMS software package can be enhanced to include the reapplication of each hot fix. Instead of directly calling the UPDATE.EXE program to install the service pack, write a batch file to include both the UPDATE.EXE and all appropriate hot fixes. As additional hot fixes are required, you can simply modify the batch file. This technique helps track hot fixes and makes it easier to apply the necessary hot fixes when a new server is installed or the service pack must be reinstalled on an existing server. Here is an example:
SP_HF.BAT @ECHO OFF ECHO Applying the Service Pack %0\..\UPDATE.EXE /U ECHO Applying Hot Fix #1 %0\..\HOTFIX1\HOTFIX.EXE ECHO Applying Hot Fix #2 %0\..\HOTFIX2\HOTFIX.EXE ECHO Copying extra files for manual Hot Fix. (example only) COPY %0\..\XFILES\CONFIG.INI D:\EXCHANGE\CONFIG.INI ECHO Updating Registry for manual Hot Fix. (example only) REGEDIT.EXE /s UPDATE.REG
After creating a batch file, you can modify the SMS software package for the service pack to include an additional command line that applies both the service pack and the hot fixes. For example, the command line title could be "Unattended Installation plus Hot Fixes" and the command line would read: SP_HF.BAT. Since this command line also require the PCM service to perform the installation, remember to select the Background Execution check box.
Microsoft TechNet
July 1998
Volume 6, Issue 7