Watson Technologies Team

Mike Marcelais, Matt Ruhlen, Kirk Glerum, Brian T. Hill, Matthew Hendel

2008 Outstanding Technical Achievement
Watson has become a core component of Microsoft development, greatly enhancing the ability of developers to make significant software advancements.

By the end of the 20th century, a communication breakdown had emerged between software developers and consumers. Programmers felt as though they were sending their work out into the void, while users were increasingly frustrated by unexplained crashes and bugs they were incapable of reporting in more than anecdotal form. By 2002, however, that chasm had been bridged by Kirk Glerum and the Watson Technologies Team. Watson revolutionized how Microsoft developers dealt with application faults, kernel faults, and unresponsive applications; raised beta-testing standards significantly; and has become a core component of Microsoft development.

In 1998, Glerum was working in the Desktop Applications Division of Office and thinking about features for the upcoming Office XP. "At the time, we would ship products and forget all about them," he says. "We had no idea what people were actually doing with them or how they were failing in the real world." Glerum recalled that Windows used to contain a component called "Dr. Watson," which used the unhandled exception filter to diagnose application crashes. He wondered if the Internet could be used to send such software "autopsies" back to a server, where they could be collected, analyzed, and fixed—with the mutually beneficial side effect of transforming hundreds of millions of Office users into product testers.

While working on the Windows team, Matthew Hendel designed what he called a "mini-dump" that would sort Watson's error reports into various buckets and link them to a debugger. "The coup de grace comes when an individual developer investigates a Watson report," says former Watson team member Brian T. Hill. Developers can view report details in a debugger, much as they would a live-running program, thus making the reports actionable. (Integration was so smooth that Virtual Studio's debugger team was surprised to learn that Watson had been wired into the debugger somewhat clandestinely.) As testing went from internal to external, a "crash curve" emerged to indicate which crashes were most common. "A bunch of Microsoft teams, driven by the passion of the people working on this technology, really came together on this," Hendel says.

Watson was included in three versions of the Explorer browser even before it shipped with Office XP, and its feedback capabilities expanded exponentially when the Windows team decided to integrate the technology into their product. The dozen Office applications Glerum had first imagined being analyzed had expanded to tens of thousands of applications. "This was rather intimidating to me at the time," says Glerum, "because I was running the server that would be collecting all these things. But the Windows team had the right mindset to say, ‘Look, this is not just some little thing for Microsoft applications. This needs to be for the whole industry.'"

The Watson model was subsequently extended to include generic error reporting capable of quantifying and describing some 200 different event types, a setup client, and corporate error reporting that collects crashes in-house before sending them to Microsoft. Watson's reach came to encompass not only crashes and bugs but also hangs, abnormal conditions known as "ship asserts," and "customer Watson," which identifies machines with an abnormal number of problems.

Watson technologies have become an integral part of Microsoft's customer-satisfaction mandate. Watson begins to collect data within days of beginning a new product-development cycle. It currently collects data from millions of beta testers who no longer need to even take the time to type up reports on new products, thus providing an important link between Microsoft and its customers. "I've always been surprised at how quickly Watson became an essential part of our development cycles," Glerum says. "Watson is always on—and it truly transformed the way we develop software."