Garanti Technology of Istanbul is the IT arm of the Doğuş Group, one of Turkey’s largest private sector conglomerates. The Doğuş Group owns companies in financial, construction, retail, tourism, automotive, and other sectors. Garanti Technology was eager for a more efficient, accurate, and predictable process for deploying security updates to more than 13,000 Microsoft® Windows®–based client and server computers. The company deployed Microsoft Systems Management Server 2003 and took advantage of the Microsoft monthly security update process. With a consistent, stable, orderly process in place, Garanti is able to update as many as 13,000 computers in just one week and use a fraction of the personnel required before. Garanti can also measure the success of updates. The Doğuş companies have been virtually virus-free for 18 months since implementing the new tools and process.
Garanti Technology (known as Garanti Teknoloji in Turkey) is the IT company of the Doğuş Group, which has interests in a range of business segments, including banking, retail, media, construction, automotive, distribution, among others. The largest Doğuş company is Garanti Bank, one of the top three private banks in Turkey.
Garanti Technology manages approximately 1,100 Microsoft® Windows®–based server computers and 12,000 client computers for nine Doğuş companies. Approximately 10,000 of the client computers and 400 of the servers are located at Garanti Bank branch locations throughout Turkey. With a mission-critical IT infrastructure this large and widely distributed, Garanti Technology needed a more predictable, accurate, and efficient process for applying security updates to its workstations.
From 1995 to 2003, the company would react and respond as best it could when it would receive Microsoft security updates to the Windows XP Professional operating system, Windows 2000 Workstation, and various Microsoft Windows Server System™ integrated server software products. The process for applying the updates was manual and time-intensive, requiring that developers write custom scripts for each update and each of eight specific versions of Microsoft operating systems used across Doğuş companies. It took several people a week to write the scripts and as many as 20 to 30 people a month or longer to apply the updates to the company’s computers. To avoid reapplying an update, the staff would have to write another series of scripts to detect whether the 12,000 client computers already had the update. Because Microsoft released security updates as they became available rather than on a predictable cycle, Garanti would often have to repeat the scripting work several times a month.
“Security updates used to come out all the time, and our staff had no way of knowing which ones were most important,” says Kenan Agyel, Manager of Microsoft Systems for Garanti Technology. “We had no automated deployment tools, so it was a huge ongoing effort to implement the updates manually. We would collect the updates and apply them in bulk, so it could be three or four months before a critical update was applied in remote locations. We were always in reactionary mode because we didn’t know when the next update was being released.”
Interruption of computer operations was another problem. To apply each update, the IT staff would have to restart all 13,000 server and client computers. Taking down systems hobbled employees in Doğuş companies and temporarily locked workers out of critical applications.
With no monitoring or reporting mechanism, Garanti had no way to assess and ensure that all client computers had successfully received and applied the updates. It could take a month of checking before all 12,000 client computers were updated.
The turning point came on August 12, 2003, in the form of the Blaster virus. Although the Doğuş financial companies were not affected, two of the conglomerate’s other companies were. These companies, employing 1,500 people, were essentially out of business until all systems could be cleaned and updated. One of the companies was a retail chain with 218 stores, which lost retail revenue as well as employee productivity. Dozens of Garanti Technology IT staff members dropped everything to manually update hundreds of systems and return these two companies to business within one day.
Garanti Technology knew it needed a better way to deploy updates to keep systems safe from disasters like Blaster. The company also needed a way to automate the process of applying updates to avoid consuming inordinate amounts of IT staff time. And it needed a way to judge which updates were most critical for its systems and monitor update success.
Just when Garanti began to look seriously at overhauling its security update process, Microsoft announced Systems Management Server (SMS) 2003, which provides centralized monitoring and management of client and server computers. In addition to providing automated deployment of software and updates, SMS 2003 helps organizations monitor issues such as efficient bandwidth utilization and software usage metering.
In October 2003, in response to customer feedback that a more predictable cycle would improve update deployment, Microsoft moved to a monthly security update cycle. The Microsoft Security Response Center (MSRC) adopted the monthly release schedule to simplify the security update process for customers. It also focused on other improvements in communications around the security updates to provide better prescriptive guidance, mitigation advice, and deployment assistance. Since 2003, the MSRC has launched additional services such as:
- Advance notification that provides a summary of all updates in the monthly package. This summary tells customers how many updates will be released and the anticipated severity ratings, and it gives an overview of products that may be affected.
- Prescriptive guidance, including workarounds for vulnerabilities and risk assessment for specific threats, which makes it easier for customers to deploy the updates.
Garanti liked what it saw and decided to overhaul its security update process to take advantage of the new processes Microsoft provided. The company implemented a detailed security update process that is set in motion when Microsoft releases new updates on the second Tuesday of each month. On the next day, the Garanti IT staff has an update assessment meeting to decide which updates it will apply and which systems are affected. After the meeting, the staff tests the updates on nonproduction (test) systems to make sure the updates don’t interfere with existing applications or services.
After approving the updates, Garanti uses SMS 2003 to distribute them to 525 secondary distribution points. The next night, the staff does a pilot update at a single Garanti Bank branch, and the site is monitored for any feedback the following day. If all goes smoothly, the staff proceeds to apply the updates to the computer systems in five more branches. If the five-branch pilot succeeds, the staff applies the updates to all remaining Garanti Technology systems over five successive nights. The updates could be applied to all 13,000 systems in one night, but the deployment is staged so that the staff can better monitor progress and catch any possible problems.
“With our new security update process modeled on the Microsoft process, we’re able to assess, test, pilot, and deploy security updates within a week after they’re released,” says Yagiz Caparkaya, Senior Systems Engineer for Windows Systems at Garanti Technology.
Garanti Technology also uses Microsoft Operations Manager (MOM) 2005, to monitor its 1,100 server computers. MOM monitors Windows operating systems and Microsoft products such as Microsoft SQL Server™ 2000 and Microsoft Exchange Server 2003 communication and collaboration server to ensure that specified operating thresholds are observed. If a threshold for a particular Microsoft product is exceeded, MOM issues an alarm. SMS 2003, MOM 2005, SQL Server 2000, and Exchange Server 2003 are all part of Microsoft Windows Server System integrated server software. Applications based on these Microsoft products (and other non-Microsoft applications) run under the Microsoft Windows Server™ 2003 operating system, both Standard Edition and Enterprise Edition, and the Microsoft Windows 2000 Server and Windows 2000 Advanced Server operating systems.
Garanti uses Microsoft Premier Support services to facilitate rapid security update deployment. A dedicated Premier Support Technical Account Manager (TAM) sits in on all the monthly Garanti security update assessment meetings and helps the Garanti staff decide which updates to apply and how they might impact production applications.
By implementing a security update process that is aligned with the monthly update releases from Microsoft, Garanti Technology has simplified its security update process, dramatically decreased the time to apply updates to 13,000 computers, and reduced update staffing needs. The company has also improved the success rate of updates, eliminated the system downtime that previously accompanied updates, and, most important, provided superior protection to its business-critical server and client computers.
Scheduled, Repeatable Security Update Process
With a consistent, stable security update process and toolset, Garanti Technology is able to perform security updates easily each month with regularity and predictability. “The Microsoft Security Response Center provides information, notification, support, and guidance that are critical for the success of our update management process,” Agyel says. “Because everything is scheduled, we know what to do and when to do it. It’s no longer a random process.”
One Week Update of 13,000 Systems
The previous security update process required time-intensive manual script writing, pulled dozens of Garanti staff members off their regular jobs to apply updates to remote systems, and took months. The new process requires only three people and updates 13,000 systems in one week. “Within one night all the appropriate update packages are distributed to 400 locations,” says Fatih Arpas, Systems Engineer for Windows Systems at Garanti Technology. “Within one week, all updates are tested, piloted, and applied to 1,100 servers and 12,000 clients.”
Also, Garanti Technology can now apply multiple updates simultaneously with a single system restart, which increases server computer availability and performance, reduces the risk of servers not being restarted properly, and eliminates the worker downtime that used to accompany system updates. “A properly defined process is very important to us. Without this security update management process, we would not be able to manage our systems effectively and make sure they’re secure,” Agyel says.
65 Percent Time Savings per Update
The new update process has reduced operational costs by reducing the number of update distributions per month and the work involved in each distribution. Previously, it took 20 to 30 people to apply updates manually. Today there are just three people on the security update management team who select updates to be applied. Another three people in the software distribution group monitor the rollout of updates.
“With the aid of Systems Management Server 2003 and the Microsoft Security Response Center, we are able to much more easily organize, schedule, and monitor the update management process and to save a lot of time,” Agyel says. “Today we spend a total of 15 to 20 hours to apply a month’s worth of security updates to our 13,000 systems. It used to take at least twice as long to write scripts and far longer to apply the updates. I’d say that we are distributing updates using 65 percent less effort.”
Measurable Security Update Success
Garanti Technology previously had no way to measure the success rate of an update distribution. “We were neither able to assess the success rate of our update installations nor able to report the status,” Caparkaya says. “With SMS 2003, we can measure our update application success rate and report the status. We have a very high success rate and a repeatable process. We can track in real time how each security update installation is progressing.”
Absence of Viruses for 18 Months
Because Garanti Technology now has an automated, easy process for performing security updates, the staff is more proactive and therefore doesn’t postpone critical updates. With the help of the MSRC, Garanti Technology and its customers have not been affected by any worms or viruses within the last 18 months.
“Before, update management took several months, and we were never sure that all clients had the updates,” Agyel says. “Now we can install many updates in a very short time, and this helps mitigate the virus threat.”
For More Information
For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234 in the United States or (905) 568-9641 in Canada. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to: http://www.microsoft.com/
For more information about Garanti Technology products and services, call 90-212-478-3535 or visit the Web site at: http://www.garantitechnology.com/
Microsoft Services helps customers and partners discover and implement high-value Microsoft solutions that generate rapid, meaningful, and measurable results. As the consulting, technical support, and customer service arm of the world’s leading software company, Microsoft Services enables the successful adoption, deployment, and use of Microsoft solutions and technologies for all customers, from the individual to the enterprise.
For more information about Microsoft Services, go to:
Microsoft Windows Server System
Microsoft Windows Server System is a comprehensive, integrated, and interoperable server infrastructure that helps reduce the complexity and costs of building, deploying, connecting, and operating agile business solutions. Windows Server System helps customers create new value for their business through the strategic use of their IT assets. With the Windows Server operating system as its foundation, Windows Server System delivers dependable infrastructure for data management and analysis; enterprise integration; customer, partner, and employee portals; business process automation; communications and collaboration; and core IT operations including security, deployment, and systems management.
For more information about Windows Server System, go to:
© 2005 Microsoft Corporation. All rights reserved. This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Windows, Windows Server, and Windows Server System are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.