With shrinking IT budgets and staff cuts, Snohomish County was struggling to update and ensure reliability of aging servers. It was behind schedule for replacing servers and spending too much time provisioning and restoring them. By deploying the Windows Server 2008 R2 operating system with the Hyper-V virtualization technology and Microsoft System Center data center solutions, Snohomish County consolidated physical servers by 35 percent and cut server racks in half, reducing power consumption and hardware costs. It expects to save nearly U.S.$1 million during its seven-year computer refresh cycle. The county eliminated downtime from outages and maintenance, boosting availability. Provisioning new servers takes 30 minutes instead of more than six weeks, and IT staff easily back up countywide data and recover failed systems 75 percent faster, while creating a more advanced infrastructure.Situation
Snohomish County, located on Puget Sound in Washington, has more than 711,000 residents. Like most local governments in the United States, Snohomish County has been facing decreasing revenues for several years. As a result, it implemented significant budget cuts. Since 2007, the county has reduced its IT budget by 23 percent and IT workforce by 27 percent. Just nine IT workers manage the data center, which was comprised of about 300 physical servers that are accessed by approximately 2,670 employees across 28 business groups.
“Not only was it time-consuming to maintain all these computers, but also many were aging and needed to be replaced. We had a U.S.$300,000 annual hardware refresh budget, but, because of funding cuts, we didn’t have the staff needed to keep up with the replacement schedule. The backlog of replacement candidates was growing, and we were in danger of pushing refresh dates out so far that the computers would begin to break down,” says Randy Manley, Systems Manager for Snohomish County. “Our IT team was already finding it challenging to ensure continuity, and, with an increasingly aging computer fleet, we knew that our ability to provide reliable service would suffer even more.”Availability and Downtime Issues
Only about five percent of the applications used by Snohomish County employees were on high-availability clusters. The rest were subject to downtime because of hardware failures and scheduled maintenance. “We didn’t have enough budget to put every application on a hardware cluster. Anytime there were hardware failures or maintenance that required powering down physical devices, employees couldn’t access those applications. We were usually able to get applications back up and running within four hours, but that was a long time for employees whose work was interrupted,” says Manley.
||We used Microsoft virtualization technologies to create a more advanced, reliable infrastructure that county employees can depend on, while minimizing costs and optimizing IT staffing resources.
Executive, Snohomish County
Snohomish County IT staff spent a fair amount of the unplanned downtime on troubleshooting. “Before we could begin fixing a problem, we had to figure out what broke down and why,” explains Eric Henry, Systems Engineer for Snohomish County. If a server needed to be restored completely, it could easily take 8 to 12 hours for IT staff to identify the issue and get the server running again from its backup system. Lengthy Provisioning of Underutilized Servers
Provisioning new server computers was time-intensive. First, Snohomish County had to procure the hardware, which could take six weeks to arrive. Then a dedicated IT staff person had to spend about two days configuring and installing the server in a rack.
“We provisioned about three new physical servers every month. It was a constant process of building and racking servers and adding new hardware, and it consumed a lot of resources,” says Henry.
The servers were drastically underutilized. “Our business units have ownership over their applications and servers. When a group received approval for a new application, it would also buy a new server to run it on, even if just a dozen employees were using it. The servers generally ran at just two to three percent of CPU utilization. This would mean a large quantity of dollars would be left on the table, unused inside our server farm,” says Michael Cruz, Enterprise Technology Architect for Snohomish County.
With so many new servers coming on board, Snohomish County was reaching the limits of its data center space. “We ended up building out a new data center, largely to accommodate known and anticipated server growth. We allotted room for 60 server racks—which was three times more than the 20 racks we needed at the time—so that we wouldn’t run out of space several years down the road,” says Manley.
All these servers and the equipment needed to support them consumed a lot of power, which was contrary to the county’s emphasis on environmental sustainability. “Snohomish County is very serious about reducing our environmental footprint. The IT department wanted to lower the amount of energy that our data center required, but that was hard to do when our need for servers kept growing,” says Manley.
By late 2008, its server replacement backlog had reached a critical point and Snohomish County had to figure out how to procure new servers without exceeding its budget or its staffing limitations. It also wanted to make it easier for the IT department to manage and ensure reliability of its server infrastructure.Solution
Snohomish County considered two options for resolving its server backlog issues. “We could spend time identifying and replacing the oldest computers and slowly building out our physical server farm, or we could look at new technologies such as virtualization, which we had heard could help speed server deployments and save money while also making the data center easier to manage. Virtualization seemed much more prudent since it could deliver much greater returns over the long term,” says Cruz.
Virtualization Vendor Selection
Snohomish County evaluated server virtualization solutions from VMware and Microsoft—specifically the Hyper-V virtualization technology that is a feature of the Windows Server 2008 operating system. “VMware was cost-prohibitive and about three times more expensive than Microsoft. Although at the time, VMware ESX had more features, Hyper-V had everything we needed and we knew that the product would evolve. And, after we realized how using Microsoft System Center Virtual Machine Manager could help us simplify virtual machine management, the Microsoft solution became even more attractive,” says Manley. The IT staff was particularly interested in the System Center Virtual Machine Manager physical to virtual (P2V) machine capability, which it ultimately used to convert about 30 physical servers to virtual servers running Hyper-V.
||Microsoft always provided very responsive service and that was really important to us as we considered moving to a whole new type of technology and a new way of operating.
Systems Engineer, Snohomish County
Snohomish County was also reassured by the level of support it could receive from Microsoft Services Premier Support. “We already had a great relationship with Microsoft Services Premier Support. Microsoft always provided very responsive service and that was really important to us as we considered moving to a whole new type of technology and a new way of operating,” Henry said.
According to Manley, “Cost is a huge consideration when budgets and staffing levels are shrinking. When we looked at the cost savings that Microsoft had over VMware, plus its track record of superior service and strong virtualization and management products, it was clear that Microsoft was the best solution for us.”Virtualization Deployment
Snohomish County began its transition to virtualization in June 2008 by building test servers running Windows Server 2008 Datacenter with the beta edition of Hyper-V. As the test systems proved successful, the IT team moved them into production along with System Center Virtual Machine Manager 2008. By March 2009, Snohomish County had begun its production deployment with Windows Server 2008 Datacenter with Hyper-V and System Center Virtual Machine Manager 2008. It upgraded to Windows Server 2008 R2 Datacenter with Hyper-V and System Center Virtual Machine Manager 2008 R2 in September 2009 to take advantage of features such as Live Migration, which enables the county to move virtual machines between Hyper-V hosts with no downtime and reduce the planned downtime associated with routine system maintenance, as well as quick storage migration and rapid provisioning.
“We chose the Datacenter edition of Windows Server 2008 R2 for the county’s main data center because it allowed an unlimited number of licenses for virtual guests on a single physical host. We were able to put 64 virtualization servers on one physical server, which made it very cost-effective,” says Cruz. Snohomish County also uses Windows Server 2008 R2 Enterprise in remote offices, such as its Medical Examiner and Parks departments and the Snohomish County Paine Field Airport. The Enterprise version allows for four virtual guests, which is sufficient for those environments.
As of March 2011, Snohomish County has deployed 170 servers running Hyper-V on 12 physical servers. This includes 69 production virtual machines that run on a four-node hardware cluster and that use the Windows Server 2008 R2 failover clustering feature for high availability. Another 63 virtual machines run in a test environment, and approximately 21 virtual machines reside within the Snohomish County perimeter network on a three-node cluster. The virtual machines run on HP ProLiant DL785, HP ProLiant BL460c, HP ProLiant BL480c, and HP ProLiant DL380 computers.
Snohomish County runs a variety of workloads using the Hyper-V virtualization technology, ranging from web, print, file, and network management services to Microsoft SQL Server 2008 database management software. The county is planning to upgrade from Microsoft Exchange Server 2007 and Microsoft Office SharePoint Server 2007—which run on physical servers—to Microsoft Exchange Server 2010 and Microsoft SharePoint Server 2010, and it will run these programs on virtualization servers.
“Our plan is to virtualize most of our existing environment and use Hyper-V for new servers. So far we’ve virtualized about 50 percent of our infrastructure. We hope to reach 75 percent virtualization by December 2012,” says Manley.
The Snohomish County virtualization solution also includes Microsoft System Center Configuration Manager 2007 R3, which the IT staff uses to update virtual machines and physical hosts; Microsoft System Center Operations Manager 2007 R2, which it uses to monitor the health and performance of its infrastructure; and Microsoft System Center Data Protection Manager 2010, which it uses to back up approximately 18 terabytes of data and restore failed systems. Snohomish County had been using Legato NetWorker (now EMC NetWorker) for backup. “We’re halfway through the process of transitioning to System Center Data Protection Manager. We like that the Microsoft product does disk-to-disk-to-tape backup because that helps us speed backup and recovery; and the way it works so seamlessly with other Microsoft products, such as Hyper-V and Microsoft SQL Server, makes operations easier,” says Henry. Benefits
By using Microsoft virtualization and management technologies, Snohomish County is able to realize substantial cost savings, consolidate physical servers, and boost availability of its server infrastructure. The county has accelerated IT operations, from provisioning to backup and recovery, improved its environmental sustainability efforts, and freed up resources to focus on strategic IT projects.
||Our plan is to virtualize most of our existing environment and use Hyper-V for new servers. So far we’ve virtualized about 50 percent of our infrastructure.
Systems Manager, Snohomish County
“In today’s difficult economy, we must look for continued cost savings. We used Microsoft virtualization technologies to create a more advanced, reliable infrastructure that county employees can depend on, while minimizing costs and optimizing IT staffing resources,” says Aaron Reardon, Snohomish County Executive. Saved 50 Percent in Hardware Costs, Consolidated Servers by 35 Percent
By moving to servers running Hyper-V, Snohomish County has reduced its hardware costs. “Because we don’t have to replace all our aging physical servers, we’re saving about $150,000 per year. That’s close to a 50 percent savings and will amount to around $1 million over our normal refresh cycle,” says Manley.
Snohomish County estimates that it also saves another $10,000 annually on licensing for System Center data center solutions. “For example, instead of licensing System Center Configuration Manager agents on every physical server, we now have one license for each host on the virtual infrastructure. It’s definitely a better deal,” notes Cruz.
The IT department no longer has difficulties keeping up with hardware replacements. “In the past, our staff didn’t have time to replace all the computers that we had budgeted for refresh. But now that provisioning virtual servers is so simple and fast, we can easily handle hardware replacements. We’ve almost totally eliminated the server refresh backlog and the risk of older machines breaking down and interrupting employees’ work due to hardware failures,” adds Manley.
The county has significantly reduced its number of physical servers. “We went from about 300 servers to about 195, a 35 percent decrease. And we cut our server racks in half. Instead of 20, we only need 10,” says Manley. Now that the county can run virtual machines for different business groups on the same physical server, it regularly achieves server utilization rates of 40 percent, which is more than a 100 percent increase.
“Virtualization was essential to achieving maximum value for our server hardware investment,” adds Cruz.
Snohomish County doesn’t need the extra data center space that it had allocated for server growth. “We’re looking at repurposing the data center. For instance, we could lease some of the space to agencies in other counties to help offset revenue deficiencies,” says Manley.Eliminated Downtime, Boosted Availability
By using Hyper-V and failover clustering in Windows Server 2008, Snohomish County has improved availability of its infrastructure. “Before, we could only afford to put about 5 percent of our applications on high-availability clusters. Now that we created the virtualization server farm on a high-availability cluster, all our servers and applications are protected from downtime from hardware failures,” says Cruz.
This helps reassure the IT staff. “We feel much more confident knowing that we’ve built redundancy into our environment. I don’t get any more urgent, middle-of-the-night calls because a system failed. System Center Operations Manager sends us alerts about failures, but the services keep running thanks to failover clustering. We haven’t had any server outages in the virtual environment since moving to Windows Server 2008 R2 with Hyper-V,” says Manley.
County employees no longer experience downtime during scheduled maintenance. “With virtual machines and server clustering, we can move workloads to a different physical server, update drivers, storage adapters, and other components without taking the virtual machine down, and then just move the workloads back when we’re done,” says Henry. “We don’t have to take applications offline. Employees don’t even know that we’re doing maintenance. Their work goes on as usual.”Sped Provisioning and Recovery
IT operations—from server provisioning to recovery—are much more efficient. “The provisioning savings are huge. Instead of taking about six weeks for procurement and two full days to configure and deploy a computer, we can get a new virtual machine up in just 30 minutes,” says Henry.
||Because we don’t have to replace all our aging physical servers, we’re saving about $150,000 per year. That’s close to a 50 percent savings and will amount to around $1 million over our normal refresh cycle.
Systems Manager, Snohomish County
Backup and recovery are much easier since Snohomish County began using System Center Data Protection Manager. “Because Legato NetWorker was a disk-to-tape solution, there was a lot of back and forth over the network during backups. We were starting to exceed what we could back up—we just didn’t have enough time to do it all. Now, with all the data on disk, we streamlined backups. We can easily back up everything that has changed in a given day on that day and everything in the enterprise over the weekend,” says Manley.
The IT staff no longer has to spend time troubleshooting issues before working on restoring systems. “We don’t have to know why something failed in order to get services back up and running. We can just move the affected virtual machine and figure out the problem at a later time,” says Henry.
Restoring failed servers is quicker. “By using System Center Data Protection Manager virtual machine snapshots with disk recovery, we are able to restore individual files within a virtual hard disk or an entire server, and we can perform these recoveries about 75 percent faster. It recently took us less than two hours to restore a 180 gigabyte SQL Server database compared to the eight hours that it would have taken with Legato NetWorker to track down the tape and do the recovery,” says Henry.
Reardon adds, “Such efficiencies are critical to an IT organization that is dealing with shrinking budgets and fewer staff. By using the Microsoft products, we’ve been able to accomplish more within limited parameters.”Supported Green Initiative, Infrastructure Advancements
Consolidating its servers enables the IT department to contribute to the county’s sustainability efforts. “We’re supporting our green initiative by reducing power consumption. We know that cutting more than a third of our physical servers—and reducing the amount of air conditioning needed to cool the data center—definitely helps us save electricity,” says Reardon.
By using Microsoft virtualization and management solutions to streamline key IT operations, Snohomish County has freed up resources for more strategic projects. “Now that we don’t have to worry about the effect of aging computers or lengthy provisioning and recovery processes, we have more time to focus on moving our infrastructure forward. We wouldn’t have had time to upgrade to Exchange Server 2010 or SharePoint Server 2010 if we hadn’t virtualized our environment,” says Manley.
The success of its virtual server implementation has led Snohomish County to explore virtual desktop infrastructure. “Seeing the great benefits of server virtualization gave us the confidence to expand virtualization to the desktop. Using Hyper-V, we can more flexibly manage targeted desktop scenarios,” says Manley.
Microsoft virtualization is an end-to-end strategy that can profoundly affect nearly every aspect of the IT infrastructure management lifecycle. It can drive greater efficiencies, flexibility, and cost effectiveness throughout your organization. From accelerating application deployments; to ensuring systems, applications, and data are always available; to taking the hassle out of rebuilding and shutting down servers and desktops for testing and development; to reducing risk, slashing costs, and improving the agility of your entire environment—virtualization has the power to transform your infrastructure, from the data center to the desktop.
For more information about Microsoft virtualization solutions, go to: For More Information
For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:
For more information about Snohomish County products and services, call (425) 388-3411 or visit the website at: www.co.snohomish.wa.us