Remarks by Bill Gates
17th Annual ACM Conference on Object-Oriented Programming, Systems, Languages and Application
Seattle, Washington
Friday, November 8, 2002

ANNOUNCER: Ladies and gentlemen, please welcome the chairman and chief software architect at Microsoft Bill Gates.

(Applause.)

BILL GATES: Good afternoon. It's a real pleasure to have a chance to talk about some of the things going on with programming languages and programming tools. I think it's fair to say that, in this era of computing where everything is connected together, really a limiting factor is the ability to write rich programs -- rich programs that can connect up to others and where you can design them with an assurance that these programs will not only work together but they'll work in a secure fashion.

Our view at Microsoft is that a key set of standards are emerging that will be central to this. These are the standards around Web services, building up from the XML standards and now with many, many related standards -- SOAP, UDDI, WSDL and a whole generation of a rich layer that goes above those, things like WS Security, WS Reliable Messaging; those will create a framework that will allow us to think of software at many different levels of scale as components that can be fit together even in cases where the developers of the different components haven't had to work together or design the same structures.

Microsoft's success has always come from working closely with developers. The leading indicator of what's going on in computing is what developers are excited about. We saw that with the original IBM PC, we saw that with the move toward graphics interface, we saw that with the move to the Internet, and now, with this idea of taking and using the Internet as more than just an HTML presentation standard but rather as using it as a connection fabric for software components to communicate across, again developers leading the way in defining the standards and thinking through what kind of applications will make sense there.

One of the things that is very tough is to think broadly about security. We use the term "Trustworthy Computing" to encompass a variety of things -- being robust in the face of Denial of Service attacks, being robust in terms of being able to show the user, when they're providing information to the system, how that information will be used. Privacy comes up as a primary concern about why people don't use services out on the Internet as much as they might otherwise be willing to.

There is going to be a lot of innovation we believe in programming languages. Some of that will come in extending existing languages, and some of it will come by creating a runtime fabric that allows new languages to come in, languages that in some cases are built around these XML constructs from the very beginning, and capture some of the semantics of the higher level protocols that's inherently in the language itself.

We think that these advances will come in a really sort of ongoing community type process. A lot of invention is taking place in the universities. A lot of evaluation of what makes sense and refinement is taking place in the broad community of developers. And yet there are some of these elements, some of these tools that we think our very substantial commitment to R&D can play a significant role in creating.

Trustworthy computing is really one thing that stands in the way of a lot of the scenarios that are most interesting. One of the scenarios when people are saying the Internet overnight would change a lot of things, one of the scenarios everyone took for granted was e-commerce. Now, of course, e-commerce is not as simple as just pointing a browser to a single Web site and understanding the display information that comes in that single site. E-commerce requires rich discovery, who might be willing to sell me a product of this type. It requires rich schematization to look at the product description, to look at things like references or availability or pricing terms and being able to make comparisons on those things. It requires very rich protocols for actually engaging in the transaction itself, how that is coordinated. It requires advances in authentication. When you're doing business with a company, how do you verify they are who they say they are, on both sides? And then as you are exchanging information with individuals inside that organization, how do you verify that the trust hierarchy is complete; that is, that the person inside that company is, in fact, authenticated as the person who should play the role that they're putting themselves out as?

The idea of federating this security so that you have independent trust hierarchies, corporations saying who their employees are, banks identifying their customers, governments in some cases playing a very pervasive role, but doing this in such a way that you don't have to have total coordination of putting it all into one namespace and one hierarchy, that kind of federated security is very necessary before the e-commerce dream can come true.

In fact, of all the things that the world was a little bit naïve about in terms of the Internet, the need for a platform advance for e-commerce was one that I think was missed more than any other.

There was a lot of talk about using intermediate marketplace companies to solve that, but they wanted to mark up the transaction, to have visibility of the transaction and in a way until the formats are standardized and you have this rich way of having software communicate over the Internet, just having the middleman wasn't going to solve the tough problems.

Once you get the solution down in the software, then you can say to somebody that very inexpensive software lets them go out and offer up their product without anyone trying to put in a markup or anything of that kind. And so just for that alone we need these advances in Trustworthy Computing.

There's a lot of visibility given now to the vulnerability that systems have out there, whether it's the e-mail type viruses or security exploits running against Internet-facing systems.

And I think it's fair to say that the industry has to make more than an order of magnitude improvement in this in order to play the central infrastructure type role that we expect is necessary to get the full benefits out of the systems that have been put into place.

Now, there are many approaches that are going to lead to these improvements. Some of them have to do with taking areas of systematic difficulty, things like stack overruns, buffer overruns and simply having constructs that make sure that those don't occur again.

A lot of it has to do with the process, the discipline of reviewing the code and making sure the amount of code that's actually a point of vulnerability is very, very small. We spent a few months earlier this year just simply doing those reviews, not creating any new code related to features, but simply going through security reviews and that was very enlightening in terms of looking at the surface area of the system that had grown up and if there were a defect in those lines of code that would create a problem. So the process of review, the process of keeping that surface area small, the human elements that go into that are a very big part of it.

But there are also many aspects of this that can be done by having better tools, and investing literally hundreds of millions in these tools, I think, is a key agenda item for the industry as a whole and certainly one that we have focused a lot of our R&D into those areas.

Of course, the sooner that any defect of this kind can be caught the better; ideally, simply never coded, and having programming languages where things like memory management are done automatically has helped with one class of problems that people run into.

We also need to have higher level description, richer typing of the systems, understanding the constraints of the systems so that you can use model-based approaches that look statically at what's going on with the code and actually flag things at compilation time.

We've created a number of things to do rich static analysis. We actually went out and bought for a little over $30 million a company that was in the business of building those kinds of tools, and we said now we want you to focus on applying these tools to large-scale software systems, the kind of system we have in the source code of Windows or Office, and see how far we can get on this.

We also went back and say, OK, what is the state-of-the-art in terms of being able to prove programs, prove that a program behaves in a certain way? This is the kind of problem that has been out there for many decades. When I dropped out of Harvard, this was an interesting problem that was being worked on, and actually at the time I thought, well, if I go and start a company it will be too bad, I'll miss some of the big breakthroughs in proving software programs because I'll be off writing payroll checks. And it turns out I didn't miss all that much in terms of rapid progress -- (laughter) -- which now has become, I'd say, in a sense, a very urgent problem. And although a full general solution to that is in some ways unachievable, for many very interesting properties this ideas of proof has come a long ways in helping us quite substantially just in the last year.

We call the system that does this kind of proof, it's a model-checking system. You describe the constraints, including things as simple as nobody should acquire the lock if they've already acquired it, nobody should release it if they haven't acquired it, certain things about the multi-threading aspect of the code that you want to make sure work very well. And you describe those things literally, in this case in the C code itself, and then the analyzer goes through and reduces the program, takes away anything that doesn't affect the path analysis that it's trying to go through to determine is there some path through the program that violates the constraints.

The initial domain we applied this in was in device drivers. You may know that in Windows we now have a system that whenever there's a system crash of any type, it offers to send a report back to Microsoft, and about 80 percent of people click "Yes." I hope no one here has seen that, but I'll bet most people here have at some time. And that information database, which we really just started having systematically a little over a year ago, has been a fascinating thing to look at in terms of where the mismatch is, where is it that drivers have problems.

We found there was a video driver that was causing 25 percent of all system crashes that were out there, literally 500,000 crashes a day caused by one video driver. And we'd always known that -- it was a third-party video driver -- (laughter, applause.). We anecdotally knew that it wasn't that great, but now we had hard evidence, in fact evidence that was a bit of a carrot, in the sense that we had this analysis, but it was also a stick in the case we could say, look, the system manufacturers who choose whether to use your chip and your driver, it might be interesting to them to know what this does to their support costs on a very numeric basis.

And one of the nice things about the analysis is that when it sees a crash of a certain type, it looks up on the Internet to see if we want to get more data. So it says, OK, it's in this video driver, give us these 10,000 additional bytes of information, so that it's a form of debugging where you look at later things of that class, get more information, decide, OK, what additional information do you want so you can really get to the bottom of those things, and there were a lot of things that came out of that and then, of course, we put the driver on Windows Update to improve it and complete the feedback loop and then that class of problem dropped down to .1 percent, so a fairly substantial improvement in that over a period of about three weeks.

But it became clear to us that an area of great vulnerability is the complex specification of how a driver is written and, of course, one response to that is to abstract out a lot of that complexity. But because people want to do so much performance optimization, you can't get rid of that altogether, and it's this idea of being able to go in and do proofs against these drivers has worked very well. It's working now on drivers up to hundreds of thousands of lines of code, and coming up with some amazing things that never would have been spotted without this kind of capability.

Now, making this so that it can be broadly deployed in a development tool is still a challenge that we have, but one that we're working very hard on.

Another thing is static analysis, looking at data paths and trying to use this symbolic path execution to detect certain classes of defects. Now, this is different than an overall proof. With the proof you take something somebody stated, and say whether or not there's a path where that can occur. Here, you're looking for a large class of bugs, and the amount of path analysis you can do is limited because you're doing it across millions of lines of code.

As we've used this tool, the things that it has pointed out, the number of things it's been able to point out, is very large, say, in the many thousands of things that get pointed out. And because it was very compute-intensive, we were doing this well after the check-in process on an every-other-day basis, running through these types of checks.

We realized that there was a way that many of these could be done on the desktop, and so we actually took and went from prefix, which was the big tool and said, OK, what subset of that can be done literally at check-in time? And it turned out about 80 percent of it could ,and so it's simply part of the compilation process to go in and have this take place. And that makes a very big difference; certain classes of problems this can eliminate all together.

The science of looking at programming errors is one that I think was neglected for longer than it should have been. Say, five or six years ago I think the best piece was the thing Donald Knuth did where he catalogued all the programming errors he'd made during his life. It wasn't a very long list actually. If the world had all programmers like that, then probably things would be pretty fantastic, but there's no known scalable model for that other than cloning, which is probably not a good approach.

So the question of the real science of digging through the entrails essentially of these things, I think it's really in these last five years it's grown up, both in Microsoft Research and in universities at large, to try and study the phenomenon and do taxonomy, and therefore use that as the guidance of what can be done better in the development process and the tools themselves.

Another important thing for us is the whole area of testing. When you look at a big commercial software company like Microsoft, there's actually as much testing that goes in as development. We have as many testers as we have developers. Testers basically test all the time, and developers basically are involved in the testing process about half the time, so you even get a balance for testing. We've probably changed the industry we're in. We're not in the software industry; we're in the testing industry, and writing the software is the thing that keeps us busy doing all that testing.

Well, one of the interesting questions is, when you change a program, say you have an urgent fix that you're doing for a customer or a security update of some type, what portion of these test cases do you need to run? The test cases are unbelievably expensive; in fact, there's more lines of code in the test harness than there is in the program itself. Often that's a ratio of about three to one.

Well, if you want to get quick turnaround on something, going through that, even though it's a fully automated process, doesn't actually work. And so the insight that Microsoft Research had is to take all the test case execution and the basics, put into a repository the execution paths of all those tests against the existing source code and then have that data to reason against so that when you make changes you can understand the relationship between the changes you've made and which test things might make sense.

We've just deployed this literally in the last four months or so in all of our major groups and the impact has been very dramatic.

To give you a sense of how this works and why it's so important, I'd like to ask Marne Staples, who's a group manager in the research group that did this work, to come up and just give us a little glimpse of what's it like to use this prioritization system that we call Scout. Welcome.

MARNE STAPLES: Thank you, Bill. (Applause.)

Good afternoon. I'm excited to be here today to tell you a little bit about Scout, and especially how it works within Windows. As Bill mentioned, reliability has been a key goal in research and we've got many projects under way. But one place we've really narrowed our focus is at the heart of the development process, and at this point we really want to push quality earlier in the product cycle than ever before. So we've been investigating for many years how can we change a system, how can we change any part of a system without breaking anything else.

Now, take, for example, Windows. We need to be able to make a very small but critical change to this large and complex system and we have to ensure reliability. I will show you today how Scout or our recent breakthrough in test prioritization will actually help us to answer some of these questions.

Now, basically test prioritization consists of three parts. As Bill mentioned, the first one and the key part of this is the repository and I'm going to show you how we actually go through and build this repository. In this last year it has exceeded 600 gigabytes of data and the repository is a key foundation for testing prioritization.

The second part of test prioritization is detecting change. and in each build of Windows, which is produced daily, there are over 12,000 or 13,000 binaries and files that we have to detect change on from one build to another.

The third part of test prioritization is leveraging tests. The last count in the last few months the tests for .NET Server for all of these binaries exceeded 10 million. So we've got a lot of work and a lot of scaling to be able to do for Scout.

So let's just take a quick look at Scout and see how do we actually go through and build this repository.

You can see here that I've got just the last several builds of Windows, Windows .NET Server, and in the actual repository it holds a lot of data and this data is imported throughout the entire product cycle. In fact, it begins at the very beginning and it goes through to the very end when the last change is made to the product before it heads out the door. So very detailed information is captured and you'll see here not only the binary information is captured but also symbols and in addition to that source files, classes, functions, even down to the very basic block and arc of a program.

Now, in addition to this the repository also holds daily coverage of tests running on these programs. This coverage, as I mentioned earlier, exceeds into the millions and you can see here just a sampling of some of the tests that have been collected. We've got security and authorization, admin tools, distributed system tests, networking. All of these tests are being pushed into this repository.

So we've got a repository, we've got all of this data, build to build every day. The next key thing is how can we detect change? We've got to be able to know exactly what changed in this repository.

When we're looking for change, Scout uses BMAG. This is an acronym we use for a Binary Matching Algorithm, and with speed, accuracy and very fine granularity, it actually goes through and rapidly disks the binaries in the current build to the build prior. So Scout uses that algorithm, and then second uses another very greedy algorithm to grab those changes that it detected first and map them directly to the test cases that it actually found.

Now, what I'd like to do is just show you an example, a real example of some of the results that Scout has actually detected.

Let's look at a DLL named S-Channel. S-Channel is a security DLL. As you can see, it's got a number of source files. In the last few weeks one particular source file that changed is Mapper.C . Now, Mapper.C has about 12,000 blocks total; if you were to look at it, it has about 12,000, but only 60 of those blocks needed to be changed.

Now, the developer can view this source to see what's the impact of these changes, and any of the sources that he changes. Here's an example. You'll see on the screen the green represents code that has actually been covered, the red at the bottom of the screen shows code that has never been covered by a test and the blue with the arrows, the blue arrows shows code that changed. This is part of the 60 blocks that changed and it's also been covered. So the developer can see that not only the change that he or she made is apparent, but also that a test case has indeed covered it.

Another more valuable view to the developer is also a binary disk view. Now, this is interesting because it graphically represents the control flow of the binary. On the right hand side is the current build and on the left hand side, my left, is also the build prior. The magenta color represents the actual ordering that has changed in this binary. The red shows code that has been removed and the yellow shows code that has been added, while gray means there's been a match. So he or she can also see in this control flow graph that there have been changes happening in the binary and exactly where this is. This is the 60 blocks of code that has been changed.

So, back to our example of how we're actually leveraging these tests. We've got the repository, we've detected this change. Now we want to know in S-Channel what test actually affected those changes. You can see here that the 12,000 blocks, about 11,253, 15 new were added and 60 were changed on this particular build.

When Scout goes through and actually leverages those tests it found on the bottom frame of this screen about six steps that actually hit those impacted blocks. Now, remember, the repository contains millions and millions of tests and it must go through and be able to find which ones affected it.

The interesting thing about the prioritization is under the very first sets are the tests that will impact the change the most. The second set will impact it slightly less, the third, fourth and so on. So prioritization is key, because that's where it pushed the reliability earlier.

So you can see that in here test prioritization is actually found the ability to leverage tests in this repository and the interesting thing is we've got several groups in Microsoft using it. Right now Windows, Exchange and SQL are all using it in their current process, and they have made some big strides. We've also been able to use it to ship our hot fixes and QSEs, because they can see the reliability sooner.

An interesting thing is one of the groups is now using Scout for the pre-check-in process for developers. They select a developer, before even checking in code, to be able to run Scout, see the impact of those changes, see the tests that have to pass before they ever get to the full build, so again reliability has been pushed sooner.

So what we've got here is the ability to actually see reliability in software sooner than ever before, and what you can imagine is that this kind of a technology can benefit everyone and we do look forward to the day to being able to extend this to notify third party developers or vendors or ISVs, any of the other consumers of the changes made to the system and exactly what kind of results or effects those changes will have.

As you can see and what I've talked about today, test prioritization has a very unique binary code-based method that detects granularity at a very fine grain and then has a unique way of mapping the test to the impacted blocks. It can also scale to very complex systems some of the products and programs that I've defined.

And if you're interested in any of these technologies, both papers have been published on BMAG and test prioritization, and you can find these in the Microsoft Research Web site (http://research.microsoft.com/).

Thank you. (Applause.)

BILL GATES: Well, now let me talk a little bit about XML Web services. This goes back, you could say, to the precursor of XML, the SGML work, or even to all the ideas of having abstract, self-describing data from long ago. You can say that the people who pursued object-oriented databases were looking at getting this type of rich description in, and having the system deal with it as an intrinsic capability.

Well, no such approach has ever gotten to critical mass, and yet now it's very necessary for these programs communicating over the Internet that we get that to work. If we do it, the applications of this are quite broad. It's not just people buying and selling across the Internet; the very relationship of all software components to each other can be handled using this approach. So we see, for example, a whole new generation of management tools that take the idea of schematization and the rich protocols, and do a far better job of management than anything that exists out there today.

You know, take, for example, a data center. If you schematize each of the servers in the data center, if you schematize the services or the applications that you want to run and what kind of responsiveness or service level of backup you want for those applications, then for the first time it's possible to have software actually manage the allocation of the resources, and monitor what's going on and to have a fully lights-out automatic data center.

This vision of moving management to a new level I think is recognized as an industry-side problem. IBM talks about it, Sun talks about it, and everybody knows we've got to get there because for the first time people are trying to manage thousands of servers, or even hundreds of thousands of clients, and the old tools simply don't fit to that. When XML Web services were being developed it wasn't targeted to that domain, but, in fact, it applies very well to that domain.

Application integration within a company has always been a tough thing, lots of glue code written by a fairly expensive consulting process just to move data from one application to the other. Well, now if the company is careful about controlling its schemas, the data dictionary, the amount of complexity in moving that data back and forth really comes down to simply mapping the semantics, which should be a very focused effort, very easy to do.

So the benefits are not just in the field of something like e-commerce, where this is a necessary advance, but a very broad set of benefits. The agility to take a piece of code and think of it as a component is pretty dramatic.

Now, this is not something that one company, even an IBM or a Microsoft, can get to critical mass. After all, if it's really about standard protocols, nobody wants to limit who they're doing e-commerce with. And so it's got to go through a standardization process, it's got to be pulled together into testable profiles. Standards that don't have fairly clear tests that relate to them often drift off in ways that undermine the full benefits of that standard, particularly if there's no formalization there. So in this case that energy has to be put in as well.

And many of the pieces have come out of different groups. The UDDI standard was done somewhat separately from WSDL. There's some pulling together of those things. The XML type system, of course, is not exactly the same as we have native to a lot of languages we have today. There is some impedance mismatch there. But my view is that the progress on driving these things as a standard has really been quite phenomenal.

WS-I is the name of the organization that pulled together a lot of companies that are behind this and it includes virtually all the major players in the industry and not only is the organization in place but the adoption of even the higher level protocols, things like the WS Security piece, has been very rapid.

Now, many of you here can look back on previous similar efforts and say, well, why is this one different. Well, one reason I think it's different is that it will be tested across corporate boundaries where somebody has a complete software stack from one provider and somebody has a complete software stack from another provider. It's all very independent of what language these things are written in and there will be an absolute demand for perfect interoperability.

In a sense, it's more like what the market expects from TCPIP than something like Corva that was positioned as an interoperation thing but primarily inside one corporation and there was never a black and white test of is this meeting the needs that are out there.

And so we're putting all our energy behind this. You could go as far as to say that when we announced .NET, that was about betting the company on Web services. And it has a fairly profound impact on all the software we do. Some parts are obvious. The development tool, Visual Studio .NET, was a major effort for us to build the tool for building Web services. For our database SQL Server, the idea of not just having XML mapping on top of the database, but having XML down in the core of the query processor and the data representation, that's the major work we're doing in the next release of SQL Server. And it's very important to us because, by getting that rich heterogeneous capability for the first time, we'll be able to take the SQL store and apply it to domains like e-mail storage, file server storage, directory storage where we've had to have special stores that we used for those things today, leading to additional complexity of the backup model and programming model for those different kind of stores.

The commitment to Web services even touches Microsoft Office. Today, when you think about Office and something like Excel, the XML schemas are not a native data type. If you express typing or constraints, the Excel spreadsheet, as you import those things in, it's not preserved as you the right script. So even Office has to have this idea of the XML of being very native to the applications themselves.

The next version of that, that comes out the middle of 2003, is a huge advance in getting XML down into the core of Office itself, even building a new module called the XDocs that relates to that.

So .NET is simply our implementation of these XML Web services across all the different Microsoft products.

In Visual Studio .NET we've said that moving to XML is not something that's restricted to a single language. In fact, there will be two classes of XML capable applications: those that were written with XML structure from the beginning, and there will be lots of those, the new ones are coming out, but there will be applications where the XML layer has been put on top of an existing application and, of course, you're not going to change and rewrite that code.

The notion that the world will rewrite everything and move to one programming language is not only unrealistic, I think that would shut off the kind of innovation that you want to see in extending languages and letting new languages come into the mix. And so the idea of having the runtime be able to host all the different languages, and even being willing to extend the runtime as new novel languages come along, is a very strong commitment we have, so that somebody who's inventive in that domain doesn't have to build an editor or the special compiler piece or the special runtime piece or have a debugger that's not quite as good as what people are used to. They can just focus in on the one contribution and then have it, because of the extensibility of the tools platform, get all those other things that are really independent of the language to come along.

We've had some very good partnerships in doing work with people on this, and I'm excited to have that moving ahead.

One thing to make clear is that we do see that existing C and C++ code will continue to be a substantial part of the source code that's not only out there but that people are continuing to do work on. If you take Microsoft's code bases, they are over 90 percent C, C++ code bases. We've had a lot of work to improve in the C++ area at the same time that we're putting forward C#. I think there were some other sessions at the conference this year where people like Anders talked about C# and the good results we've seen there.

But in parallel with that, we're making innovations in C++. Some of the things that people have given us a hard time about in terms of standards conformance, we hear the message, we are now implementing all of those things. C++, because of the native pointer capability, really lets you do high-level, low-level, high-performance type code, and it will have a significant role.

We see a mixture of C++ code with managed code, and having that boundary be quite efficient is one of the breakthroughs that we were able to make in the CLR runtime.

Well, I mentioned that languages won't be static, so I have some thoughts here about where things might change and almost a solicitation from people who are working in these areas to collaborate with Microsoft, because we think there's a lot of impact that can be had by doing these things right.

A lot of development should be more modeling based and yet the idea of the model being separate from the source code, we think, is one that leads to lots of problems. The modeling and the source code, you should have viewing tools that allow you to work at the modeling level, work at the source code, and it's almost a duality as you go through refinement and so the two don't drift apart from each other; in fact, the model is used for a lot of the static analysis that tries to catch any problems in the implementation. So modeling will be very critical.

In the language extensions, native connection to XML; in the same way that SQL has always looked foreign inside primary languages today, some of the XSL things that you get in there have certainly a different flavor, one might say, than typical code. Well, how do you get rid of some of that complexity?

Asynchronous programming is a rich area where there's been many proposed extensions. I think some of those will absolutely be moving into the mainstream.

Messaging primitives built into the system: A move towards Web services puts messaging sort of on top. Yes, you can build object-oriented interface underneath that, but messaging is a very basic primitive that you ought to be able to reason about and write code related to directly in the language itself.

Richer typing, we think, is very important.

Parallel programming, this may sound like another Holy Grail that's been out there for a long time, but it's actually a very actionable thing. If we look at microprocessor evolution, there are sort of two tracks to go down. One is the one that Intel is focused on most energetically, which is trying to reduce cycles per instruction. And that's very important; to the degree progress can be made there for linear code it's really fantastic. The amount of silicon area that's devoted to getting an extra 10 or 15 percent off of CPIs is pretty phenomenal. It's gotten quite extensive. There are diminishing returns there.

Now, if you have code that can express some parallelism, taking the equivalent amount of silicon and forking off multiple paths to run in parallel, that is fairly cheap, and the place we've seen that in a dramatic fashion is actually in the graphics processor. Those processors have moved from having very dedicated pipelines where pixel shading and (VRTAC ?) calculations were very specialized; now they moved to where those two things have become unified and really it's about general purpose parallelization.

And we want to do this in a way that you can describe it in a high level language. They're not married to any particular semiconductor implementation so you're abstract across the GPUs and these GPU capabilities move into the mainstream so it's not just when you're dealing with graphics data, it's when you're dealing with data of any type.

So we think that, because of the diminishing returns in one space, this is very right for a lot of things that are quite performance sensitive.

And finally there's another point about XML, which is that as we get these abstract stores, the database is extended. The idea of reducing the amount of code that's pulling things out of the database and putting things back into the database once again comes up as an opportunity that a lot of improvements can be made.

So in no way do I see this decade as one where we'll stay on the same plateau in terms of languages, tools or modeling. We have to make advances there, not just because of the kind of large-scale software that we do, but also because customers around Web services will be defining their business flexibility according to their implementation of those services and they'll need much lower cost ways to build code, but code that has to be extremely reliable and extremely secure, and without advances that simply isn't within the reach of the broad set of programming skills that are out there in the companies that will want to use Web services.

This decade is one where we think that pervasiveness of digital activity will become almost common sense. Although coming off of the hype of dot-com valuations and all that, we find a lot of people fairly sober, and certainly the hiring situation and some of the cutbacks in R&D in many of the companies make this several-year period here one that doesn't have the ebullience of a few years ago.

And yet the key advances, whether it's chip improvements or fiber-optic improvements or storage advances, are tackling some of the frontiers that we've had for so long. Pervasive connectivity, finally getting broadband out, you know, it won't happen overnight, but over the course of this decade, with existing approaches and new approaches around wireless mesh type networks, we think that will be very pervasive and fairly low cost.

Things like a natural form factor for the computer, we think that that is going to come along in a pretty dramatic way. In fact, what I've got up here is my little Tablet PC. It was just yesterday in New York that the actual official launch of the Tablet PC took place, and we were there with a number of manufacturers showing the hardware and software combinations that are put together.

Well, why is that a big deal? Well, as you get the PC into the small form factor that you can carry around and use without a keyboard, as you get broadband and you get the wireless network, the idea that you're going to just sit and sketch an idea and share it with a friend or that you're going to take some article that you read from a periodical or an academic paper and be able to write simple notes on it and send that off to people who might be interested in your comments, and they'll get that, those comments, and if other people have commented on the same thing, they'll get that all brought together; some of these things that are really commonsense scenarios have been blocked by the fact that you can't hold the device in your hand.

Even reading, you know, we thought four or five years ago that if we got the resolution better and made advances like ClearType, reading would move to the screen. But what research found was that if it was in a fixed position, you would never read long documents no matter how nicely they were on the screen, because the fatigue of looking at that single point defeats immersive reading. And so although it varies by person how large the document has to be before you print it out, there's some size that everyone for some subconscious reason for them, maybe I'll just print this out and read it. Well, then you take your notes on it and there's a question of, well, how do you get those back in the system.

And so we've been in this bifurcated world of partly digital, partly not. You see that when you go into a meeting sometimes where, in the environments I'm in, I need to know the protocol in advance; am I allowed to bring my computer to this meeting or not. You know, in some meetings it's considered that having the screen there is a problem. Well, with the Tablet PC, the fact you can just sit and take notes and not have any blockade there, it makes quite a difference.

So this is one of the frontiers, getting into the small form factor. And although I'm thrilled by the initial hardware implementations of these devices, one thing we can say for sure is that the hardware will get better and better. NEC showed an ultra-thin type device; it was the thinnest of all of them, and it really was the thickness of, say, two legal pads there. Some of the systems like the Toshiba had large screens, 12.1-inch screens. So out along this learning curve, which is just an evolution of portable computing, because we use all the same operating system and applications, we can take that learning curve and move it down and have it be far more pervasive.

One of the interesting problems when you get a tablet and ink is trying to recognize the ink; you know, what is it, not just what text it is, but how are things grouped together so you can do analysis and layout. And it's kind of an interesting debugging problem. We needed to make ink a standard data type of the system so you can send ink instant messages and ink e-mail. Ink e-mail, I send it to people and they go, "Whoa, what is that," because they've never seen that before and any machine can receive the ink e-mail because it's just a graphics metafile. You have to have a Tablet PC to create it, but consuming it can be done anywhere.

Well, some of the toughest software work we've ever done is in this whole ink and ink-recognition area. In 1992 when tablet pen-type computing flopped, the commitment I made to our research group was we would fund the handwriting recognition year after year without knowing when it would get ready for prime time because the idea was that, with improvements in hardware and software cleverness, that would be a problem that would be solved eventually, and it's a big enough contribution that almost any arbitrary period of time justifies that kind of investment.

The investments added up over the years in this whole tablet area to many hundreds of millions of dollars, but if you think about something that gives you extra hours to get value out of your PC, there hasn't been anything quite as impactful as saying the tablet form factor will just be a standard feature of all these portable machines.

To show you some of that work and how we dealt with some of those challenges I'd like to ask David Jones, who's a program manager on the Tablet PC team, to come out and show us a little bit about ink and ink recognition. (Applause.) Welcome, David.

DAVID JONES: Thanks, Bill.

I don't know how many people have actually seen a Tablet PC yet, since we just launched yesterday, so I'll just take a second to show you mine. I'm running an Acer TravelMate 100 here. I'm using it in laptop mode because I need to keep my hands free for the presentation. But I can easily just swivel the screen over and turn it into a slate so I can take notes on it. But for the presentation I'll turn it back to laptop mode.

So today I would like to give you a demonstration of our ink-analysis technology, and to begin I want to give you a demonstration of the problem that it was designed to solve. What you see in front of you here on the screen is an early prototype of the Windows Journal note-taking utility, which allows you take handwritten notes on a screen and allows you to convert your notes to text.

Now, if you write neatly between the lines you'll find that you get pretty good handwriting recognition results, which is right down here. But when we gave this to users what we found was that they don't take notes this way. Real world notes look kind of like everything else on the screen. People like to write on angles like this, they don't write between the lines, they write really big and go outside the lines. They draw diagrams. They write really small sometimes. And when you take ink like this and you hand it off to a handwriting recognizer you get results that look kind of like this. Garbage. (Laughter.)

So the problem here is not with the handwriting recognition technology itself, the problem is with the way we are feeding ink to the recognizer. You see, the way our recognition algorithms work is, I use state-of-the-art pattern algorithms, pattern-recognition algorithms that are trained on tens of millions of handwriting samples that have been gathered all around the world, and each one of these samples come from somebody sitting in a lab who's been asked to give a handwriting sample where they write between some guidelines. So if you want to get really good handwriting recognition results, you need to feed the ink to the recognizer in a way that it was trained to recognize it.

So another way of looking at the problem is when you're dealing with a page of free-form handwritten notes, you need to be able to step back and analyze the entire page and find where are the lines of writing on the page so that you can normalize them and feed them to the recognizer in the way they're trained to recognize them.

Now that you understand the problem, I'll show you our solution to this problem.

Here's the same file loaded in our ink analysis test application. When we begin our analysis of a page of ink, the first thing we do is we start by assuming that every single stroke on the page is its own word and every single word belongs to its own line, and I'm going to turn on some bounding box visual feedback here so you can see that. I'll turn on the lines first. So a pink bounding box means lines. I'll turn on the words next, and a blue bounding box means words.

Now, the ink analysis algorithm uses a multi-path bottom up approach where we slowly start grouping strokes together into words and words into lines using layout and robust statistics. We start doing it very conservatively at first and then in subsequent passes we can rely on global robust statistics to start doing it more aggressively.

So watch now as I step through the algorithm. In the first step we use a dynamic programming model to minimize the convex distance between stroke fragments and you'll see that we start getting pink lines appearing on the screen. It's important to note that many of our statistics are calculated on stroke fragments rather than the entire strokes themselves, because this allows us to treat cursive writing and printed handwriting as pretty much the same thing.

Now, as I go through the next pass we get more aggressive in grouping lines together and you see, for example, here in the big line that every word is now appearing on its own line.

In the third pass, we get even more aggressive and now we've actually successfully identified all the lines on the page.

In the fifth and sixth passes of the algorithm, we turn our attention to the actual lines we found themselves and in the fifth step we start grouping the individual strokes on a line into words. And now you see that we've successfully found all the words on the page.

And in the final step, we now take a look at each line and we take a look at it and decide is this line writing or is it drawing and we do this by feeding in both global and local features into an SBM classifier and that gives you this result.

So here everything on the page we have determined to be writing is now colored blue and everything that we decide is drawing is now colored in red.

Now, having shown you this, we also go through and we order the lines on the page. So it's relatively straightforward for a developer now to walk through the list of lines of ink on the page, to rotate them back to horizontal, to normalize the timing information of the strokes in case the user has gone back and corrected them and changed the timing information, and then it can be fed to the handwriting recognizer and so you get the results that you'd expect.

So I showed you how this works in our test application. Now I'd like to show you it running for real inside of Windows Journal and also running incrementally as we perform the analysis as each stroke is added to the page.

So this is Windows Journal that shipped as part of the Tablet PC. However, this is a special debug version that also shows colored bounding boxes for the structure on the page. And watch what happens as I write.

So as I'm writing, Windows Journal is running our analysis algorithm in the background and it's figuring out where the lines and words are on the page, and then in a separate thread it's also sending off these lines to the handwriting recognizer and storing the recognition results in the document. And that is exactly how Journal is able to support searching on your handwritten ink regardless if you've written sideways or upside down.

There's one more thing I want to show you here. If I write a numbered list down the page, you see that we identify this as a line running down the center of the page, which makes sense. But if I go back and start adding to it we are smart enough to take this new context information into account and reanalyze the page and we now realize that these are, in fact, lines running across the page. So we're always taking advantage of as much information as possible to produce the best results.

Well, I'll just draw a box around this so you can see that in this case Journal interprets that this is classified as a drawing so it would not get sent off to the recognizer.

So what I've just shown you is how our ink analysis technologies can be used to take a page of ink, analyze it and irrespective of size or orientation of the ink locate where the writing is and where the drawings are and how in note-taking applications such as Windows Journal can then use this functionality to support handwriting recognition on a page of free form notes.

Going forward, we hope to extend this technology to identify different types of ink structure such as a paragraph, lists, tables and flowcharts and we hope that providing this functionality in the Tablet PC platform that will enable developers to design powerful ink manipulation functionality in future versions of the Tablet PC.

Thank you very much. (Applause.)

BILL GATES: Well, I just want to close by saying that there's a real interesting set of tough problems that need to be solved; trustworthy computing, a very broad domain, bringing Web services in and making them easy to develop against, building up the standards, whether it's C++ or the new standards around Web services and pulling those together. There's a lot of investment in R&D that's required to go after these things. There's a lot of collaboration that's required in terms of reaching out and finding whose got the best ideas.

So for this community we're very anxious to know what new things can be done in languages, what new things can be done in tools, because the impact of the good work you do in that area will be quite dramatic. So we look forward to working with you on solving these problems.

Thank you. (Applause.)