Remarks by Bill Gates
CHI 2001 Conference on Human Factors in Computing Systems
Seattle, Wash., April 2, 2001

MR. GATES: Well, good morning, and welcome to Seattle. It was here in Seattle, about 30 years ago, that I first had a chance to use a computer. It was a teletype connected up to a GE timesharing system, and I can say that the user interface wasn't very good: very limited bandwidth, upper case only, but I did like the paper tape capabilities.

What I wanted to do today is share some of the projects that Microsoft is involved in that we hope will create new frontiers for human/computer interfaces. I also thought I could take some of my favorite Microsoft error messages and get your feedback, this is one I just got recently. I think it's fair to say that Microsoft has done good things in computer/human interface, and also done its fair share of bad things as well. And so we look forward to participating in the conference, and seeing what new things we can do.

It's fair to say that things have come a long way. The computing power we have today to dedicate to the human interface is quite awesome, and the freedom created by things like wireless networks, large storage, incredible bandwidth, every one of those things can really be devoted to doing a better job for the user. Personal computing has been through several major eras. The very first machines of the late 1980s had a character mode interface. And there were no standards for what an application interface should look like, dBASE, WordPerfect, 1-2-3, all the different applications appeared in a different way. They completely took over the screen. There was no way to share information between those applications. And yet, there was a lot of experimentation that went on there that was very important. This was the era when people first realized the personal computer would become the most important tool mankind has created to leverage our creativity and to become an incredible communications device.

We got a chance to take what we learned and essentially start with a clean sheet of paper when we moved from the character mode world to the graphics interface world. The applications didn't look the same at all. In fact, it was like being in two different worlds. There was a period of six or seven years where people straddled the two interfaces. But eventually there was enough memory, enough power, that graphics interface came to be taken for granted. There were style guides, there were multiple applications working together, data exchange. And this is something that led to a whole new generation of rich applications.

Only six years ago the next era began, which was the idea of universal connectivity, being able to browse for information. However, the browsing world and the productivity world were kept separate. So, if you wanted to take information from multiple sites and annotate that information, or combine it in a rich view, you were manually copying down the information and moving it to your application.

So, although we have this great connectivity, we don't yet have the richness of the Internet fully integrated into the environment. And so one of the tasks ahead, and one that the industry I see is very focused on is taking the Internet to a new level, really being able to view the Internet as an information source, and pulling that in so that everything you're used to doing in the productivity world can be done with those kind of browser connections.

This new era is bringing with it some very interesting challenges in user interface. For the first time, we will have multiple devices. Multiple devices that are the same, so a PC at work and a PC at home. Today it's very hard to synchronize the information between those devices. In fact, in some ways, you want differences, so you want the system to understand you're in a different mode, a different context, and so some information spans and some does now.

Even more difficult are machines whose interface is quite different. So, for example, if you organize a trip, or some of your favorite stocks, certainly that information should show up on your smaller screen devices, but it needs to show up in a completely different format. And, in fact, the proliferation of screen sizes, and different input capabilities, even just the different keypads on the smart mobile phones is confounding application developers who don't want to have to customize every single Web site and every single application to these different devices. Today, the user is very much involved in moving the information around. Even as you go from Web site to Web site, the fact that you have to have different user names, different passwords, the fact that you have to resupply the same information again and again, the fact that that information, once you supply it, you have no idea how it's retained and how it's used. That represents a very significant problem.

In the world of communication we have today, there's a tendency to get lots of junk mail, information you don't care about. Certainly I think I get my fair share of junk mail as well. Generally about three or four thousand pieces a day, and so that's really prioritized the Microsoft Research on filtering technology. There's no doubt that even the PC itself is still fairly difficult to use. If you think how many different concepts do you have to learn. You know, say that you simply want to learn how to search, well, searching files is different than searching emails, is different than searching for a printer, or a font. The number of commands and overall concepts to deal with a full set of applications is simply way too large, and there has to be a way to clasp that down to a very, very small set.

When something goes wrong with the system today, it's a very laborious process to deal with that. Typically you pick up the phone and try to describe to someone on the other end of the phone exactly what you've seen, what you might have done over the last few weeks, and they guide you, turn you almost into a robot, and have you type in cryptic commands, and see what might be done to fix the system.

Furthermore, whatever problem you run into, it's likely that someone else will run into that same problem. So, rather than having the fix that was discovered on your behalf automatically be available on all the machines, millions of people might run into that same problem. And so this idea of turning software, thinking of it not just as a product that you install one time, but something that is almost a service, where it's being made up-to-date on an ongoing basis, that's a very important element of what we mean.

We also have to really complete the feedback loop, the idea that whenever somebody is sitting in front of a device and they're frustrated, they ought to be able to log that. Sometimes we've talked about having a yellow button, or a red button. If you're just slightly confused, you push the yellow button, and if you're sort of mad at what happened you push the red button. And taking that information and really on a broad basis constantly striving to reduce the number of frustrating experiences.

The Internet, of course, gives us the opportunity to complete that feedback loop, to gather that information in a very deep way. It also gives us a chance to take whatever improvements are made, and make them easily available. And so I do think that in the next four or five years, a lot of what we'll do will be to take technology and simply improve the experience of what people are doing today. Really making the tool that much more effective in reducing the costs of support, and training that goes with these machines.

Now, we have many techniques that are very important to us in deciding how to do these things. Throughout the industry, I think the seriousness about user studies, or usability has been going up very, very substantially. The goal here, of course, is to really get real-life information to see how confusing things are, to understand when people try to do a task, how do they go about it. Microsoft has started doing usability research back in the 1980s. In the late '80s, it was still a very small scale effort. A couple of usability engineers, one lab with the ability to have four people at a time, and basically a single approach. Today, it's very much expanded from that. All of the large product groups do very significant usability research, including pure research activities around completely new approaches. And in a typical year, we do close to 1000 different studies. We continue to be surprised by the results we get out of these things, and we continue to be able to use these things as a very major component of deciding what the product should look like and how it should work. So it's really become a science of its own with lots of expertise, lots of new methodologies, and something that I think will continue to grow as the techniques in this area improve, driven by many of the people here at this conference.

And one thing that has really been incredibly important to Microsoft is a dedication to research. So in addition to the Microsoft product labs, we have about 500 people who work on long-term research. This includes a lab in Beijing, China, a lab here at our headquarters, and a lab in Cambridge, England. And this is a very broad set of research topics. But, if you really look at it, the majority relates to human computer interface. They relate to user interfaces, for example: how can we use 3D environments? Are there things there that could be done, how do we predict what the user is interested in? User modeling with very rich technology underneath, including Bayesian model-type systems, is something that we think has an incredible amount of promise.

Of course, the holy grails of user interface, the use of handwriting and speech, are very important to us, because we believe those technologies will be moving into the mainstream in the next five to ten years. Visualization of information -- how do you take large databases that will be gathering with everybody doing things on a digital basis, and analyze that information, making it meaningful to people, how do you get people to work together. A wide range of topics. And although many companies have had a hard time taking their investment research and mapping it into actual product use, for Microsoft we've had very good luck with this. Partly it's that by being a software company, the transition from research into products is a very natural transition. Partly it was looking at sort of the examples where that hasn’t worked perfectly, including companies like Xerox, and saying, what was it that we were going to do to make that work in a much better way?

So I can say that the investments that we've made in research are really the best investments that we've made. And in many ways, these are driving the new products that you'll see coming out over the years ahead. It's important to keep that dedication to research over a long period of time. Many of these projects only pay out with the kind of ten years of focus that we're able to provide.

Now, one of the goals, the high-reach goals that we've had is the idea of working with information on a purely digital basis, the idea that then it would be accessible, the history of what you've worked with, you could go back and search it, you could share it with other people. And one of the things that's certainly been a barrier to this is the fact that the screen itself has not been attractive for reading long documents. All of us here can think about the process we go through when we get a very long piece of email sent to us. What is the threshold at which you decide to print it out instead of reading it off the screen. And it's almost a kind of subconscious process, you don't understand why is it so much more comfortable to read it off a paper. But, if you want to annotate it, if you want to have it to take around with you, and if it's simply long it is often preferable to do that. We can see this in many ways, for example, magazines today are able to put free copies, including all the content up on the Web, and people don't call them up and say, hey, I'm canceling my subscription. I mean, are you kidding, I get it more up to date on my computer screen, I can search it, send it along to other people, stop sending me this paper magazine. And there are very few documents whose digital form has replaced the paper form.

There are a few like the encyclopedia. Traditionally, the encyclopedia was very expensive, updated in whole about every ten years, with yearbooks in between, and that has shifted. Today the encyclopedia is overwhelmingly based on CD-ROM products sold for under $100 and used on a personal computer, and the richness of what you get there, the time lines, the pictures, the soundtracks, that interactivity, and lower cost has made that more attractive. So the question is how could we do this for documents of all types. What does it really take?

Well, talk about tough competition, the book has a lot of things going for it, and you have to match up to those things, as well as having a lot of additional benefits. And so Microsoft set the goal of saying, let's do what it takes. In this case we knew it wouldn't just be Microsoft alone. We knew that we'd have to work with the screen manufacturers, the people who do hardware design to put the elements together. But, we said, we've got long term dedication to taking reading and putting it up on the screen. The benefits of doing this I think are quite dramatic, the fact that you'll have any choice of the material you're interested in, the fact that you'll be able to have links. Sometimes when you're reading a magazine and you'll see a URL, it's like 30 characters long, you'll say to yourself, who really types in that 30 character URL? You'll have lots of new technologies like scanners, or things where people are trying to embed in a bar code the idea of a URL on that paper device. There is a simpler way, which is to actually have the presentation be done electronically and then that link is live, it's just there, you can get as much in-depth information as you're interested in getting.

So to show you what we're doing to achieve this vision, and sort of talk about where we are, let me bring up Bill Hill one of the key people who has driven this forward, and have him talk to you about his philosophy, and what he's doing with the Microsoft reader.

Welcome.

(Applause.)

MR. HILL: Good morning, Bill.

MR. GATES: Good morning.

MR. HILL: Bill, you were speaking of the benefits of books and the benefits of reading. Can I ask when is it you learned to read, you probably read a lot, right?

MR. GATES: Right.

MR. HILL: When is it you learned roughly, five, six?

MR. GATES: When did I learn?

MR. HILL: There's an accent problem, I'm from a bit further East of Seattle.

MR. GATES: Probably five.

MR. HILL: Five, right. Anybody ever ask you to learn reading version 2.0? There's no new menus, there's no new commands, this is a skill that you learn when you're knee-high, and you use it your whole life. You add lots more words to the pattern database that you have in your head, but the same basic skill lasts you the rest of your life. It's a terrible upgrade story. You can't sell many upgrades if you get something you use your whole life. I mean, the book is a magical thing, if you boil it down to its essence it's basically dirty marks on shredded trees. That's all it is. But, it's transparent, immersive, it's the closest thing to telepathy we have. Somebody on the other side of the world has a thought, writes it down, makes some dirty marks on shredded trees, sends it to me, I open it up, look at it, and I get that thought. And that's a really magical technology that these dirty marks can do that.

So how does it do that, that was the question we started out with, trying to figure out how does a book work, what's going on, what's the interaction between the human and the book. When you're holding this book the real book isn't going on out here, the real book is going on in your head, what's going on here. So that was really the starting question. We don't have to do it for print, there's been hundreds of years of research into how we read, how our eyes move, the ways to optimize type for reading. Tinkard and Patterson are the kind of authorities on that, they published their first paper in 1929, did their last work in 1960. By 1940 they produced a book called How To Make Type Readable, that was based on speed of reading and comprehension tests given to 33,000 people by then. So there's a huge amount of research in how to do for print. How do you translate that into the screen, what are the issues, what are the problems, because reading still isn't a first class citizen on the screen. You know, the representation of text originally grew out of WYSIWYG, the goal was to print documents so the screen was a good representation of what you get when you printed it, but nobody really wanted to read it.

So I did a lot of research, read all of the literature and so on, but the interesting thing was, the answer didn't come in the lab or sitting in front of the computer, the real answer came out in the woods of Western Washington, because it's really quite surprising -- Bill Gates is Homo Sapien version 1.0, right. You may be the top of the line model, Bill. But, we're all the same. There was a shift about 400,000 years ago, there's no upgrade in sight. As recently as 10,000 years ago we were all hunter-gatherers. So our visual perception and our brain developed to do that, and reading actually grew out of that perception system. So the thing is for the hunter-gatherer visual perception is our primary survival skill, it's the one we use to identify food, to identify threats, we have sensitivity to movement, et cetera, et cetera. And what we're really doing is we specialize in visual pattern recognition, because it's a survival skill it has to run in background.

The classic one is when you remember, you've been driving -- you drove here this morning, perhaps, and quite often you get the experience when you're driving, you arrive at your destination thinking, I just crossed the 520, I drove on the freeway, I obeyed all the road signs, et cetera, et cetera, and I don't remember any of it because I was thinking about something else. We've all had that experience. The visual pattern recognition is in background mode. You know, visual pattern recognition in humans is like a device driver in computers. On birth you start recognizing visual patterns, on death, end recognizing visual patterns. If you believe in reincarnation you have to add one more line to the program, loop until perfect.

But, what's going on with the book is there's a system at work. Tinkard and Patterson identified this as early as the 1940s, there's a whole system of interrelated variables. So it's not just art, there's a scientific system that you can analyze in there. I call this system OSPREY, optimized serial pattern recognition. And I had the realization of what was going on when I was actually tracking a coyote in the woods and realizing that I'm recognizing patterns one after another, that's what I do when I read.

So the trick was to, once you understand how that system works, to kind of hold OSPREY sacred, if you like, make that the core task and add value to it without breaking it. So for those who are interested I've written this Magic of Reading research paper which is available in the reader format. You can get that from the Microsoft.com/reader Web site. So I wanted to show you now how we put some of those principles into practice.

Here you see the Microsoft Reader, we're inside a book, we're inside War and Peace. I've never actually read the whole of War and Peace yet, I'm hoping one day I'll get to it. Does this look familiar? It kind of looks like the page of a book. Well, there's a lot going on here, or there's a lot not going on here, actually. The first thing that's going on is we've grabbed the entire screen, there's no user interface in here. One of the things about the book is there's no distraction from the content itself, there's nothing going in out there to pull your attention away. One piece of interface had to survive, which was the resize and close box so that you can switch applications, but even that we grayed out. Now, if I want to start at the beginning of a book, go to the end and then stop, I use the space bar. If I want to get really fancy, I can use the back arrows as well. But, that's it, that's the entire user interface that you require to read a book from start to finish.

Now, of course, there's a lot more you want to do. So the trick was to say how can we put a user interface in here, without breaking the reading experience. So we invented what we call contextual or on object user interface. It really means you get the user interface you want when you need it, and not the rest of the time. There are actually three objects on this page, there's a running header, there's a page number, and there's the text itself. They all do different things when you click on them. Click on the running header, for example, I can navigate outside of the book, or I can change the settings, or I can go out to a bookstore like Barnesandnoble.com, buy another book online, pay for it, have it downloaded to my system.

One of the things that you want to be able to do in an electronic book that you can't do in a printed book, for example, is change the type size, if you're visually impaired, or if you're like me, you're 52 years old and it's getting kind of hard to read 10 point type first thing in the morning, you might want to do this. Now, what's going on here though is we're not just doing the simplistic thing of changing the type size. There are a lot of related variables, the spacing between the lines, the spacing between the words. So you have to re-hyphenate and justify. So that's what we're doing, we're in fact re-hyphenating, rejustifying, doing all of the spacing, repaginating the book when you change this. We're moving, if you like, from one harmonic tuning to the next. The reader doesn't have to care about that, they don't have to worry about the user interface, or the expertise that's going on behind it. They just make the type bigger.

The page number, if I right click on there, gives me a ripple control that allows me to page quickly through the book. You see, at this type size there's 2500 pages in War and Peace, so it's more than a light read. The interesting thing about digital stuff is it is a light read, it's no heavier than a 200 page book. The other object on here is the text itself, and this is a more complex object. This allows you to do a bunch of things, like add a bookmark to a page, highlight the text, add a text note to it, in Czechoslovakian, or something. If I put the text you'll notice there's a text icon there that tells me there's one there. I can draw on the page, by the way, I defy anybody to do anything very creative with a mouse, especially in front of an audience.

Other things you might want to do are things that you cant' do in a printed book. You might want to look up a word in the dictionary. So the Encarta Pocket Dictionary is shipping with the Microsoft reader. The dictionary itself is a piece of content, so if you don't understand a word in the dictionary you can click on that, look it up. On the basis that your knee bone is connected to your thighbone, if you do this for long enough you'll learn the entire English language. Just take this page, right, all of the marks that I made are stored in this annotations index. So if for example I'm doing a research paper I can highlight the passages I'm interested in, then come to the index and quickly navigate between them, I can delete them, I can move them around and so on. And I can jump back to the place I was reading. The other thing is, even though those marks are there, I can turn off the display of the marks. So I can still go back to the page, everything is still stored there, I can turn them back on again whenever I want to.

There was one other thing I wanted to show you. So we already have scalable text. There was one other thing I wanted to show you. What we can do with the -- I'm switching to a different version, because this is a prototype version, and the menus will do funky things. We should have sound.

(Audio clip.)

MR. HILL: Now, you see what's going on here, this was Dr. Martin Luther King's speech synchronized with the text. But, we're also working on a speech driven interface for the reader, and integrating the text to speech engine. Now what these things do -- you know, I've got a dream, too. And the dream is that one day a visually impaired person, or a blind person will just be able to go onto Barnes and Noble online, and buy exactly the same book that you and I buy when we buy it. They don't have to wait for the large print or the braille version, or the version that talks to them. So to access books -- that's never happened in history, you've always had to wait for a different version. But, they'll be able to buy exactly the same book, at the same time, at the same price as the rest of us.

So these are the kinds of things we've done in the reader, and we've invented technologies like ClearType to overcome problems of low resolution of the screen in the meantime, until devices get better. The challenge going forward now is to look at what we've done here and figure out how to make reading a first class citizen in the operating system and the applications, as well. Once we have that then I think the number of trees that have to get cut down in order for you to keep up with your reading will definitely go down.

MR. GATES: Excellent.

MR. HILL: Thanks, Bill.

MR. GATES: Thanks, Bill.

(Applause.)

MR. GATES: So reading with no compromise in the digital world. It's something that will still take a lot of work, but we really can see that with software like you've just seen that it's not that far away.

Let's talk about the era that we've just now entering. The Internet era, the GUI era, and the character mode era, every new thing builds no the work that came out of those eras. And so we have the PC from the first era, we have that rich graphical interface, we have the worldwide connectivity of the Internet. So what's the next thing that we're doing and why do we consider it so different. Well, first is this idea of many devices, of a small screen device you carry around that's connected to a wide area wireless network. These new devices will force us to think in a very different way about how we store information, and how we interact. So any time, any place, which is part of your theme of this conference actually is in the Microsoft vision statement.

We kept the same vision statement for 25 years, talking about the PC as a tool on every desk and in every home. It was only about 18 months ago we changed it for the first time in our history to talk about having the power of computing any time, any place, on any device. And so it is a different way of thinking about the software, a different way of thinking about the scenarios that make sense. Sometimes when people think about this world they think of these devices as being in competition with each other. Some people have said, the cell phone is so popular, doesn't that mean the PC is less important.

Well, in our view it's quite to the contrary. The two devices need to be complementary. If I plan a trip, if I want to look at different places and maps, and what the costs are, and talk to my friends about what they want to do, I want the full screen device. I want that same screen size I have to do my homework or to read a magazine. So I set up the trip there, but if later simply in my car if something drastic happens, like a flight is canceled or something is not available, I might want to be notified of that, and make a quick decision about some other choices. So that's a holistic connection where the fact that I did the work on the PC it should immediately know that I'm interested in changes and find out where I am and contact me on that device in the appropriate way. Likewise for schedule changes, or changes in stock prices, or things that happen in your business environment. You want all of those devices to think of you and not just have you have to think about all those different devices.

There are many new technologies that are part of this era. One of the central standard technologies is XML, and the redesign of all the different pieces of software to deal with the rich heterogenous self-describing data that XML allows. This is a profound change for protocols. It's a profound change for the operating system, for the database, and for the productivity software. The standards around XML are emerging quite rapidly, things like SOAP and UDDI, these are basically the protocols to make XML into a distributed computing platform, and to allow for things like who do I want to buy from, are there any sellers out there who might fit my needs, making sure that that works without having to go through any middlemen whatsoever.

For the first time it means computer systems where the designers never met each other, and yet those systems are working in an intimate way to not only find that you want to do a transaction, but monitor that transaction and deal with any of the surprises that come up along the way.

The Microsoft product family that addresses this new era we call .NET. So, in the same way that Windows was our product for graphics interface, .NET is our platform for this new era. It's the first time somebody has talked about an operating system that runs across the different devices. And we recently introduced a thing called Hailstorm, which is part of .NET. It's a set of schemas where the information is literally stored in the Internet, and it's accessible to the different devices. And coming up with standards for those schemas, the scheduling standards, the preference standards, all the things that all these different devices and applications are interested in, that's a key step forward here.

So, this era is about a new generation of software, a new generation of experiences. It's my view that the distinction between the phone and the PC will largely eliminate that. That may sound strange today because the phone is so good at voice connections, the PC is so good at presenting pages of information. But in my view, whenever you call someone up, you might want your screen and their screen connected to see documents, edit those documents, and work together. Likewise, when you're on the PC, you might want to talk to somebody before you make a purchase on a Web site. And so things like instant messaging will move to have voice connections.

Even in the case where the voice doesn't travel over the Internet, when you make a call, we can see you're near your PC, and connect your PC up to the person that you're calling, and allow you to do that kind of real-time collaboration. And so there you have the PC as a communications device, not just asynch, but a real-time communications device as well.

The investments in these wireless networks are an important element of this. We're thrilled to see the incredible rise of 802.11 as a strong standard. One of the things we want to do in this new era is take the way that information is stored and get rid of a lot of the variety there. I mentioned earlier that the way we deal with email and files, and other objects, that they're very different. Well, what if the file system was a very rich database, an XML oriented database. Couldn't we take all of that information and store it the same way, and so that your techniques of navigating and filtering would be absolutely identical. That's one of the challenges we're taking on in this new era, is to replace the file system with the rich database and, therefore, radically reduce the variety of user interface things that people have to learn.

Now, another part of the vision here is that you can take the computer with you wherever you go. We talked a little bit about reading. Well, one of the things we discovered in the whole reading area is that you need a device that you can move it and hold it, instead of being in a fixed position, which is very fatiguing, you, like you do with a book, want to have it in your hands and move it around so that you can read for long periods without fatigue. There's some incredible benefits that can come by having this new form factor we call the Tablet PC. Think of it in some ways as revolutionary, because you'll use it where you'll take it to meetings, you'll be able to take notes on it. But in some ways it's evolutionary because we're going to run all of today's software, we're simply going to bring in reading, annotations, meetings, note taking as part of it. So using the latest Windows, we take the pen, and we give all software developers the ability to work with this as a new interface.

In fact, my view is that all PCs will eventually be like this, even your desktop machine, although it will have a much larger screen, you'll have a pen that you can sit there and do things very directly. This vision has been pushed forward by a really strong group at Microsoft and Bert Keely, who is one of the key architects in that group, is going to come out and give us a little look at what they're thinking about of what will the Tablet PC be like.

(Applause.)

MR. KEELY: Hi, Bill.

MR. GATES: Good morning, Bert.

MR. KEELY: Good morning. Hi everyone.

So this is my Tablet PC. I say mine because it's the main PC that I've been using for the last four months, actually. It's 8-1/2 by 11, and less than an inch thick. And it's actually one of several shapes that you'll be seeing over the years to come. What defines the Tablet PC, besides being a full-featured PC, is that it has an additional mode of use, that is that you can hold it in one hand and use the other hand to directly manipulate things on screen.

So that doesn't mean that it can't be used the way that a traditional PC can. In fact, this type of tablet you can think of as a desktop monitor, perhaps one of couple of desktop monitors that you can have plugged into your stereo speakers, your DVD, et cetera, even a mouse and keyboard. And then, when it's time to go to a meeting, you can grab this display, take it with you, and all the information is still right there.

Some tablets will be more like laptops whose display can fold back around, and you can use it in this mode, or you can use it in a more traditional mode. But some of us early adopters like the idea of leaving the keyboard at the desk because we love the level of mobility.

This device I'm using right now, four or five different windows is what I have up right now, and so there's my 1.2 gigabyte email inbox with all those attachments. Here are my pictures, several hundred pictures. Here's my media player. Here are all my other files, all of my Office documents, CAD software packages, direct manipulation of 3D models is really fun on a tablet. And then here is some ink notes that I took in preparation for this discussion.

So, you can really see this is a full-fledged PC, and our vision is to combine the simplicity of paper with the power of the PC. I don't know if you know this, but the way that I was flipping among Windows was just by pushing a button which is right here -- if you think of this as a big stack of pages, we're flipping through windows as if they were pages. So, what I love is the handiness.

I would like to show you some other things, though, and to give you a bit more up close I need to put it on this overhead projector. So, when we talk about direct manipulation, there are already many features built into existing software that become a whole lot better when you have a tablet. So, for example, this is a Word document, and this is Word XP, actually the new generation of Word, configured for optimal use on a tablet. And things like being able to paint the format directly can be really pleasant on a tablet. So there are editing functions that feel very natural on a tablet, including undo.

Bill Hill actually told you a lot about how he focused on achieving the simplicity of paper for reading materials. One of the next things that we've been focusing on is achieving the simplicity of paper for capturing thoughts, and the capture of thoughts is very natural with a pen. And so we've been working on some note taking software in conjunction with this tablet hardware, and we've been doing a ton of user research to understand what the key issues were going to be. And boy have we found that when you have a device that's this simple and portable, and expects to be able to work with a pen, there are many, many latencies that you have to get rid of in order to make it feel natural.

So, what we've done in our note taking application, we've actually worked on an interaction model that really doesn't get in the way of handwritten ink, but it also allows mouse functions to come in. Now, the first step, of course, was just make the handwritten ink look like handwriting. And, I'll tell you what we're after. Here's a piece of paper that depicts what happens when a person is losing motor control, you can see nice clean ink, and then line by line it gets progressively messier.

Now, most people think of digital ink as being something like what we have down here on about this fourth or fifth line, and it was really important to us on the tablet PC to get to the level of quality not of existing digital ink, but of the individual's handwriting. Now, this may look a little bit messy, but I know this is exactly the way I write. And having it exactly the way I write means that -- reminds me of what I meant when I wrote it, so we see that as very, very critical.

Now, as far as the way that you mix the ability to use a mouse with the ability to use a pen, it's a very delicate system, so we've been doing a lot of prototyping, and actually Evan Feldman, who many of you may know, was the first person that we hired onto the Tablet team, he's the usability expert, and he's been conducting lab tests. It's really been amazing to see the kinds of things that happen if you put just a little bit of trouble in the way of digital handwriting. So, one of the first principles that we found was very important was to leave the ink as it is.

But then, second, people really want to be able to use the power of the PC. One of the things that people really like to be able to do is to insert space to add a font to an existing note. So here's how easy that is. When I just tap, I get an insertion point. If I drag that insertion point, it gives me space in which to write. And the ink comes out nice and clean. In fact, I'll zoom in on that and show you how much it is like handwriting.

Okay, that's because we're giving the ink note taking process the first priority at the system level, and then we're anti-aliasing the edges to make them smooth, and then as the ink dries, we're going back and doing some fine Bezier lines in order to smooth the ink even a little bit further.

So, other types of manipulations that are really important to us, and of course, again, it's got to stay out of the way of the handwriting, it's got to be a simple, direct experience. We want to be able to modify the appearance of the ink, and formatting things like you might do with a word processor are also very easy. So, I can use an on object menu, just like with the reader, in this case to do things like bold, italicize, even highlight the ink.

Now another thing is, if I want to find somewhere else in my notes a particular word, I ought to be able to search my handwritten ink, at least I'd like to. So one thing we've done is just to make sure that if we select a word, we have an easy way to find the next instance of that word. So, here's the sketch that I was looking for, that I wanted to find. You can see that it's searching through the handwritten ink. It doesn't have to replace it with text. It uses the text in the background, okay. So that gives us the best of both worlds.

Now, finally, one of the things that people really are telling us they want to be able to do with their handwritten notes is repurpose them into other documents. And so, I want to show you the simplicity of repurposing this ink. I simply copy the ink, and when I go back to this other document and paste, it inserts as text. So that's the repurposing of ink that we're really focusing on, okay.

Well, that's a glimpse of what we've been doing with ink on the Tablet PC. It really is dependent on a great interaction model, and we're looking forward to next CHI, next year, where we should be able to show you some tools with which developers can apply the interaction model that we've been working on, so we can all work on achieving the simplicity of paper with the power of the PC.

Thank you.

(Applause.)

MR. GATES: Well, I think you can see the tablet really can open up some new frontiers in terms of rich usability and having access to your information wherever you go.

One of the challenges to this environment, though, is that often you want information that's most valuable to you. You don't want your instant messaging to be active when you're in a meeting, you don't want something that's unimportant to interrupt you if you're thinking about an important document.

And you want all these different email addresses or phone numbers, things like that, to be brought together in a way that you get back into control based on who is sending you the information, and what you're up to that the computer does the right thing. And it's not just human communication, it's also any of these alerts from the machine, or from the events out there on the Internet. You know, sometimes you might be interested in when a check clears, or when a bill comes in, but you certainly wouldn't like to be interrupted with that all the time.

Now, part of the idea here is to simply go back to the fact that the scarce resource, the most valuable resource is the user's time. And so, the computer or all the different devices, as they work together, have to respect that, and have some sense of how the user wants to use that time.

This kind of idea has been talked about for a long time, the idea of an agent that knows things, and that can deal with the uncertainty, and can deal with the context. It's a deep enough problem that we've had people in Microsoft Research thinking about this for some time. And we have, we think, made a lot of progress in this. And to show you where we are in this work, I would like to ask one of the top people, Eric Horvitz to come out and show us what he's doing in the area of priorities and notification.

(Applause.)

MR. GATES: Welcome, Eric.

MR. HORVITZ: Hi, Bill.

So, thanks. I would like to briefly tell you about some work on methods for sensing and inferring a user's context, and then leveraging that context sensitivity to give users greater control of information and communications, so they can get the right information at the right time on the right device, in just the right way.

So, if you could look at my laptop over here, we see a client for an application named Priorities, that's been in development for several years now, Priorities actually learns from users about how to guide alerting about appointments, email and tasks. The system actually comes to understand a user's sense of urgency, assigning an expected cost of delayed review to every item coming into a user's inbox. I typically don't share my urgencies with colleagues, but once in a while for illustrative purposes it might be okay.

You see here, this is yesterday's email and this morning's, Carl Kadie in our group got a 92 score, you see a sorted list of emails on the right right now, by urgency, prioritized by urgency, 92 by Jennifer Rowland, a message on the DTAS lunch, a scheduling item is coming up very strongly there, Ken Hinckley on some new product impact. And going down a little bit we see Susan Dumais had some first-come, first-serve CHI T-shirts in her office yesterday, and it got an 87 score. Going down lower we see some notions about the announcements like colloquia and so on, like Terry Winograd's People, Computers and Design seminar. And then towards the bottom here, Repair Your Creditr, Delete Bad Credit and so on. So in some ways our priority system gives you an automatic direct mail filter, as well.

So just a little bit of background here on what's going on with the system. The Priorities system is actually considering multiple facets, or features in messages, including the structure of the header and body, patterns of text, it does a linguistic analysis, even most of the communication history of a user, who you've been responding to, and how fast you respond and so on. And it considers sender-recipient relationships by looking at the online org chart. What is the relationship between people that you send urgent messages to, or receive urgent messages from, and so on. So depending on an inferred urgency, the information about my current context gleaned by watching my activity, listening to ambient acoustics in my office, even looking at my calendar, the system actually decides whether to interrupt a meeting, for example, with some incoming information, or to make a decision to send a message to my mobile device.

Now, Priorities is just one piece of a larger project at Microsoft Research, named the Notification Platform. If you think about it there's a lot more in life than email, tasks, and appointments. The Notification Platform considers multiple sources of information, including interpersonal communications, news and finance and other kinds of information, and even deliberates about information coming from those helpful agents, like an agent offering helpful assistance, or scheduling assistance, or background query information, as well as error messages. And Bill and I have spoken about this before, we both keep a list of our favorite error messages around to deliberate about future interfaces.

The idea is to tie these information sources in a very standard way with standard interfaces to a notification manager that knows about all your devices and usage patterns, and to reason about, in an ongoing way, contextual information, like desktop activity, your calendar, the ambient acoustics, and potentially, in advanced versions of the system a video analysis and accelerometer data, and so on. And the idea is to have standard interfaces for devices and notifications to inform the notification manager as to how best to get information to you in the right time and the right place.

So let's actually take a look at what's going on on my desktop here, we have a context palette that's been listening to me and watching me. You see, the system knows we're coming in through this device. I set up my calendar to make this a meeting at my office, Bill, just because we'd never get bothered with anything if it knew I was sitting up on stage with you here in front of a large number of people. The computer actually thinks we're in a private location right now, but it hears me talking here, and sees me working at the top level of the system. It also notices that it's acquired my face here, and the whole time on this stage this system has been just waiting around for people's heads to come into focus. And it can see that I'm actually sitting here gazing at the display right now. And that information is used, along with other sensors, some noisy, in a sort of fail safe way, you can plug in and snap out sensors and they can fail, even a calendar is a noisy sensor if you think about it, and all that information is being pulled together to generate some information about a number of things, including my location, and what's going on right now in my life.

So using some information about my activity level and my location, we can reason about notifications coming in. And let's actually stream in some more notifications here into my universal inbox here. You see now that the system has some MSN Messenger information, some financial information, some news information and so on, and even agents will post their notifications in this universal inbox in this notification manager. The idea is to basically render important information at the right time, on the right device. So in this case Carl Kadie, who might even be here, wants to know about the meeting today, what time. By hovering I can actually treat this as voice mail from messenger and tell Carl Kadie that I've now seen his message and so on.

If I was on a small device right now the system would actually understand how long I've been gone, and know the probabilities of where I am and the importance of the message, and reason about the cost of delayed review to consider the value of sending that information to me. In fact, if we go offline right now, offline from my event source, and pull in a scenario, like "away for four hours," notice now that we go to pager of some information her, and the information would actually be going to my pager right now.

So that gives you a sense for what's happening with the Notification Platform project. We're excited to say that we've been working very closely with the product groups at Microsoft, as a whole. In one application that is actually a descendent of Priorities we recently shipped about two weeks ago the Microsoft Outlook Mobile Manager, and the idea with Mobile Manger is you can set up profiles for different places in your life work, home, out of office, and through a variety of easy to use controls set your preferences about messaging and communications for mobile settings. Perhaps even more exciting for our team is the integration of some of the technology into the .NET infrastructure. And I guess most of you probably weren't here about a week and a half ago to hear Bill's talk on Hailstorm, but the Hailstorm system of Internet services has some deep integration of notification platform into the schema, context management, the decision policies, and the user profiling in Hailstorm. And we're looking forward to seeing third parties and researchers build on top of this general framework.

So just for the CHI audience I want to mention that beyond the machine learning, and the decision making, and the statistical methods, and the sensing and inference, there are some very, very deep challenges in HCI with a general notification system. And this includes notions of coming up with great visualizations and abstractions that let users define and inspect profiles about how they're alerted, and when they're alerted with different kinds of information. And also, different ways to summarize in place on different devices, potentially, multiple items about the world, and about things the user wishes to be aware of. There are also some really deep questions about the psychology of disruption when a user is flowing or headed down a path in their work. And so we have several psychologists on our team, including Mary Czerwinski and Ed Cutrell, we have all been looking very deeply in our labs about the fundamental fabric of disruption and learning about the cost and value of information.

So, again, we're very excited about the kind of value we might deliver in the world with new kinds of cross device experiences for users.

MR. GATES: Thanks, Eric.

(Applause.)

MR. GATES: Well, so you've seen that the frontiers out there are pretty amazing. What kinds of scenarios computers are going to be addressing will be far broader than in the past. And so the kind of work that all of you do, thinking about how to take these capabilities, present them in a simple, understandable way, to have it so that when the system is not working it's understandable what's going on and what needs to be done, these are very tough problems. They're not problems that are going to get solved in the short term, and so the kind of patient, academic research that many of you are involved in, the kind of research that some of the software companies are really funding I think will play into this. And we'll also be getting massive feedback from the users as we put these products out in the marketplace. By taking advantage of that feedback, by continuing to fund the research, and I think all of our dreams about computers, whether it's reading, notification, the tablet device, or many, many others that people have, I think we'll have a chance to realize those in the decade ahead.

Thank you.

(Applause.)