In this series of articles, we have been looking at some of the ways that the threat landscape has evolved over the past decade. In this final article in the series I discuss software servicing, or the art and science of effectively and efficiently keeping software up to date.
What File Versions are Running on the System?
Ten years ago I was a Technical Lead on the Premier Networking support team at Microsoft. Our team helped Microsoft’s enterprise customers with TCP/IP and all the protocols and services that enabled Windows operating systems to communicate across a network. Windows NT 4.0 and Windows 2000 based systems were being leveraged by enterprises around the world.
I recall a tough case that I resolved where a name resolution service (WINS) was having issues trading lists of names and IP addresses (replicating) with other WINS servers on a customer’s very large network. After extensive troubleshooting it turned out the issue was the WINS service on one of the servers had files from a different service pack installed versus all the other services on that server and all of the other servers in the environment. To get the system in this strange state, the administrators had installed Service Pack 2 on the server, and then at some later time installed the WINS service on that same server. They didn’t realize that they needed to re-install Service Pack 2 so that the WINS service was also up to date (an issue that was addressed in newer versions of Windows). I started seeing other similar issues with other networking services.
Over the years I developed many troubleshooting tools that shipped in resource kits and support tools included with various Windows operating systems that became popular with Windows administrators; one such tool was called SPCheck. SPCheck helped determine the service pack level of each file for each networking service that shipped with Windows NT 4.0 and Windows 2000 and could help diagnose this type of “binary mismatch” problem. It became fairly popular among Windows administrators at the time, so we shipped it with the Windows XP Resource Kit.
The point of this story is to illustrate how hard it used to be to understand exactly what file versions were running on a system, even in professionally managed environments – prior to the Automatic Update Client and Windows Update technologies that we take for granted today. Understanding whether a system required a specific security update or already had it installed was challenging especially in large enterprise environments.
Figure: An example of an SPCheck report file showing an analysis of a Windows XP system with no service packs installed
An Abundance of Installer Technologies
Another servicing challenge for systems administrators ten years ago was managing a number of different installer technologies; ten years ago, product teams at Microsoft used nine different installer technologies. This meant that if you were deploying software updates in your environment such as a service pack, bug fix, or security update, one of the first things you would need to determine is which installer each update was packaged with. Each installer worked differently and supported different functionality and switches. The abundance of installers made deployments more complicated than they needed to be. At this time there were a relatively small number of deployment tools that helped administrators determine which systems needed which updates and allowed them to deploy the required updates to large numbers of systems across a network.
Weekly Security Update Releases
Until October 2003, Microsoft released security updates weekly. Any number of security updates were released each week. For example, between June and August 1998, twelve separate security updates were released. Systems administrators needed to pay close attention to security update releases as they were not broadly communicated the way they are today.
A Confluence of Factors
Ten years ago systems administrators had a lot of manual work to do to compensate for a lack of granular deployment tools, a relatively large number of installers and few network deployment tools, and an unpredictable security update release schedule that wasn’t broadly communicated. In January 2003 the SQL Slammer worm took advantage of this confluence of factors. Although the security update that addressed the vulnerability was released on July 24th, 2002, the SQL Slammer worm successfully exploited the vulnerability on numerous systems around the world starting six months later (January 25th, 2003). The vulnerability in Microsoft SQL Server 2000 addressed in Microsoft Security Bulletin MS02-039 was originally released on the Microsoft Download Center (it pre-dated the availability of the Windows Update Service and Automatic Update Client). At this time, many administrators were unaware that they needed to install the update. Compounding the challenge, the steps required to install the update manually were hard to follow, and administrators had no tools to find systems running affected components in their environments.
The Long Road Out of Eden
Over the last ten years Microsoft reduced the number of installer technologies that administrators were required to use from nine to two, which helped reduce the complexity associated with deploying security updates.
Beginning in October 2003, Microsoft started releasing security bulletins to address vulnerabilities discovered in Microsoft software on a predictable, monthly schedule. Since then, security bulletins have been released on the second Tuesday of each month. The bulletins provide a standard list of details to help customers understand and assess the risk to their computing environment posed by the vulnerabilities documented in each bulletin and understand how these vulnerabilities can be addressed. I used an example earlier, where between June and August 1998 twelve separate security updates were released – today only three updates would be released over the same period of time. This predictable cadence allows organizations to plan and resource deployments in a more predictable way. The Microsoft Security Bulletin Advance Notification Service (ANS) also helps enterprise customers plan the appropriate resources for an impending security update release by providing some details ahead of the release of security updates. These changes, in aggregate, along with several others, have been positive for many customers and industry partners.
Microsoft and other vendors worked hard to offer customers scalable deployment technologies and tools that allow customers to deploy security updates quickly and accurately to vast numbers of systems in their environments. Microsoft provides several tools and services that enable systems or their users to download and install updates directly from Microsoft or, for business customers, from update servers managed by their system administrators. The update client software (called Automatic Updates in Windows XP and Windows Server 2003, and simply Windows Update in Windows 7, Windows Vista, and Windows Server 2008) connects to an update service for the list of available updates. After the update client determines which updates are applicable to each unique system, it installs the updates or notifies the user that they are available, depending on the way the client is configured and the nature of each update.
For users, Microsoft provides two update services that the update clients can use:
- Windows Update provides updates for Windows components and for device drivers provided by Microsoft and other hardware vendors. Windows Update also distributes signature updates for Microsoft antimalware products and the monthly release of the MSRT. By default, when a user enables automatic updating, the update client connects to the Windows Update service for updates.
- Microsoft Update provides all of the updates offered through Windows Update as well as updates for other Microsoft software, such as the Microsoft Office system, Microsoft SQL Server, and Microsoft Exchange Server. Users can opt in to the service when installing software that is serviced through Microsoft Update or at the Microsoft Update website.
Enterprise customers can also use Windows Server Update Services (WSUS) or the Microsoft System Center 2012 family of management products to provide update services for their managed computers.
Windows Update and Microsoft Update revolutionized how software gets serviced and help to protect hundreds of millions of users around the world. These services also make tools like SPCheck obsolete because of their sophistication and scale. Usage of Windows Update and Microsoft Update has increased significantly since 2003. Since its introduction in 2005, usage of Microsoft Update has increased dramatically.
Figure: Usage of Windows Update and Microsoft Update, from the second half of 2006 to the second half of 2011, indexed to total usage in the second half of 2006, as published in the Special Edition Security Intelligence Report
Security is a journey not a destination, and we have learned a lot from our customers and partners on this journey. As I mentioned in part 1 of this series, the 10 year anniversary of Trustworthy Computing marks an important milestone and is a natural point to stop and reflect on the past and what we have learned. It also marks the start of the next decade of challenges during which Microsoft will be as committed to helping protect our customers as we have ever been.
Thank you for reading our six part series.