REDMOND, Wash., Jan. 29, 2007 — Gathering user input to develop superior software is nothing new at Microsoft. The company has done it for years in the quest for a better product, drawing on everything from users of technical betas and focus groups to usability labs and consumers
With Windows Vista — Microsoft’s most tested operating system ever — the company drew on user feedback to an unprecedented extent. In order to ensure Windows Vista’s readiness for the general public this week, those efforts included recruiting 50 real-life families in seven countries to live with the operating system through successive builds over a two-year period until release to manufacturing (RTM) last November.
A screenshot showing the "send a smile" tool used by families in the “Life with Windows Vista” program to provide feedback to developers about the OS. Users activate the tool by clicking on the smile/frown icons at the bottom right hand corner of the screen.
For more insight into Microsoft’s Life with Windows Vista program, how developers gathered feedback from the families and what new processes have helped provide a superior customer experience with Windows Vista and potentially future versions of Windows, PressPass spoke with three Microsoft officials: Trish Miner, research manager for the Life with Windows Vista program; Tjeerd Hoek, director of user-experience design for Windows Vista; and Richard Russell, development manager in the Windows Core operating system team.
PressPass: How was the Life With Windows Vista program different from consumer input gathered with previous versions of Windows?
Miner: We’ve collected feedback from consumers before, but this program is larger than any program we’ve had in the past. We gathered feedback from 50 families in the U.S., Israel, Mexico, India, Japan, Finland and Germany. We also had the opportunity to collect feedback from the families every single day and earlier in the process, so we got much more feedback than in the past. One method we used to gather feedback was to visit customers in their homes, which we have done previously, but the number of visits was much higher with Windows Vista. Some of these families were visited 10 times by a number of different people from Microsoft. Also, the length of time the families were using the different versions of the beta product was much longer with Windows Vista than with previous versions of the OS. Some families were involved in the program for about two years. They started using the product in August 2005 with our beta 1 build and they used it until they received the final RTM version.
PressPass: What was the impetus behind the Life With Windows Vista program?
Miner: People do more and more things with their PC at home and are more dependent on the PC for different aspects of their daily life. We wanted to capture as much feedback as possible, as early as possible, about the unique issues facing this group of people. We focused on people at home doing the kinds of things with their PC that you would expect consumers to do. They use the product differently from how business users or our tech beta people might use it. We specifically tried to find people who were not running home businesses, and instead recruited people who did things like photos or music or had a large number of files on their computer. Those were some areas where we planned to innovate in Windows Vista and we wanted to make sure we had people to give us feedback in those areas.
PressPass: Please describe the logistics of the program. How were families chosen and how did they get started?
Miner: We went out and interviewed a large number of families, then the selected families were invited into the program and given a computer with Windows Vista on it, a printer and a monitor. They had to set it all up themselves. It was just as if they had brought it home from the store. Some of them got different computers, depending on the types of things they were using the computer for and to help us see the breadth of hardware issues, performance issues or other types of issues. Initially, there was no differentiation in the versions of the OS, but in later builds they were running Windows Vista Home Premium Edition and then Windows Vista Ultimate Edition.
PressPass: Talk about the feedback itself. How was it different from, say, the feedback you got from beta testers?
Russell: The feedback from the families was very different from the bulk feedback that we got from the beta testers. We had about 2.5 million beta testers for Windows Vista, and of course we couldn’t have a one-on-one relationship with that many testers, so we used a lot of automated ways to collect data from them. They could submit bugs, their OS was instrumented to report data back and we’d aggregate that and report some of it. By contrast, we could maintain one-on-one relationships with the families, so their feedback was a lot richer and more interactive and timely. For example, we’d get feedback or input from one of the families about something they were seeing onscreen, and the development teams would be able to look at it and get questions back to the families. We used both types of feedback to find and fix things in the OS that needed work and to validate design decisions and things of that nature.
Miner: Another thing that made the families different was that we went out and visited them in their homes and actually took team members along so that we could see what they were doing and follow up on issues in person if we needed to. It wasn’t just them sending in comments and feedback. Often, the feedback was discussed in meetings where we were actually creating the design, so it wasn’t like we did it, then threw it over the wall for people like Tjeerd and Richard. The feedback we got was very tight and integrated into the development process.
Hoek: There’s also a difference in the kind of people we’re talking about here. As important as our beta testers are, obviously, to get the quality and finished product, a lot of them tend to be more technology oriented or really interested in software for business reasons. But the people in the family program were just regular families, so they gave us a very different perspective on everything we were putting out.
PressPass: What new feedback mechanisms did Microsoft put in place to gather feedback from the families – and future customers – to help make the best possible product?
Miner: We created a special tool for the families and installed it on each of the computers to enable them to send us a smile or a frown just by clicking on a smile icon or a frown icon any time of the day or night. It would automatically take a screen shot of whatever was on their screen at that moment, and they could type in a comment, then hit send. That would come immediately to us at Microsoft, and we could see those comments and take action on them. We also had what’s called instrumentation. Windows Vista itself has some data points in it that were automatically sent to us, enabling us to know, for instance, how many times a user clicked on different items on the Start Menu, or how many times they got one of the User Account Control prompts. We wanted to make sure they weren’t getting too many.
We also used existing feedback mechanisms, such as surveys to collect data at different points in the process. As we said earlier, it was very timely feedback. We could actually go out and ask the families questions to get that feedback right away. Another thing we did was to videotape the families actually using the product because we wanted to be able to take as many team members virtually out to these homes as possible. We showed the videos to the team members so they could see how our customers were experiencing the product.
Russell: These videos were very helpful. The [developer] teams really liked seeing them because when you see real people struggling to use your product it really motivates you to go back and address those issues and make it better and get it back into the hands of those people. So it was really interesting to see that on a regular basis during the development process. It was a dose of reality for the developers.
PressPass: Please describe how these new processes have helped provide a superior customer experience with Windows Vista and future versions of Windows.
Russell: One of the things we did differently with Windows Vista over Windows XP was a much broader focus on the full end-to-end scenarios. For example, we didn’t just focus on the display of photos but the whole end-to-end scenario of capturing photos, from the moment you plug the camera into your PC to downloading the photos and tagging them, viewing them, selecting them, then burning them to a DVD or printing them or archiving them or e-mailing them. Tjeerd’s team focused on the user interactions and how it looked and felt and the flow and the ease of use. My team focused under the covers to make sure the entire end-to-end scenario worked smoothly and efficiently and was fast and snappy. And that really changed things. When we looked at that whole flow, it really pushed us to look at things in a much broader way. In other words, we didn’t just focus on making specific things in the OS work well, but on making everything work well together. And that was particularly important given the strong consumer focus of Windows Vista.
Miner: Given the number of families we had, we could see those end-to-end scenarios played out in different ways because not everyone approaches the experience of, say, capturing photos in the same way. Family A might do it way differently from Family B, so it was important to see that diversity as well, which would be hard to gather in a usability lab, for instance, where they’d all be doing it the same way versus how they do it when they’re at home.
PressPass: Can you quantify the feedback contributed by the families?
Miner: We got over 5,000 comments using the “Send a Smile”/”Send a Frown” tool. The families also identified over 800 bugs that were never noted by any of the other customers using Windows Vista, which I think really illustrates how the families are different from the other people that we had using the product and helping us out with making sure it’s good before we ship.
PressPass: Can you name some features that got smiles?
Miner: One thing they liked is, when you mouse over the items in the task bar, you get a thumbnail view of what that item is. The families really resonated with that, not just because it’s visual, but because they felt it made it easier for them to more quickly get to where they really wanted to be. They also liked the QuickPick feature in Internet Explorer 7.0, which allows you to see all of the icons of all of the tabs you have open by simply mousing over and clicking it. Also very important to some of the families was the ability to change the color of the glass. They had expressed very early on that they wanted additional customization and they were very pleased to see those coming into the later builds. Other popular features were the sidebar gadgets, the Flip-3D feature and – in the area of security – the Parental Controls feature. Often it was the little things they liked. For example, we changed the hourglass icon from earlier Windows versions into a kind of a circular icon. We didn’t expect the families to take the time to say that they liked such small changes, but they did.
Hoek: We also got a number of smiles for the new Control Panel where people said it was finally better than the Classic View. I was specifically looking at that area because in Windows XP we had the same sort of user interface, but a lot of people were found to be switching back to the Classic Control Panel where you basically just use a bunch of icons. That was another area that people really liked and that really delivered in terms of ease of use and efficiency. Again, in the area of security, we got several smiles for a new Windows Vista feature requiring a password before installing a program – which one would think would be an annoying factor. One comment says, “Excellent – even installing the Reader requires a password. I like it!” It’s fascinating to see people realizing it’s an inconvenience, but also appreciating it as a protective aspect of the OS.
Miner: One important difference is that these were spontaneous comments. If we bring people into the usability lab and have them look at the Control Panel and ask them what they think, it’s one piece of data. But it’s great to see these families just giving us spontaneous feedback and saying, ‘Hey, I love the new Category View.’ That really cements that we’re moving in the right direction, and we can take that with the usability lab data too and see if they’re consistent with each other.
PressPass: What about frowns? What features earned those?
Russell: The very earliest versions of Windows Vista needed some work on performance. Things were a little slower than they needed to be, the OS wasn’t quite as responsive and certain operations took longer than the users felt they ought to take. Of course, the OS was very young, but it was something the families struggled with and that was a little frustrating to them. We took that to heart and the next version of the OS they got – the February 2006 CTP (Community Technology Preview) that went to a broad number of people – performed much better than the earlier versions from the previous fall. They were quite happy with that. This is a good example where the input from the families helped us focus on the things that were most important to consumers.
PressPass: What features are currently in Windows Vista as a direct result of input from the families?
Miner: Each of our families would probably give you a different example. A prime example is the burn button used to burn CDs and DVDs. There was no burn button in the Photo Gallery, but there was one in the Pictures Folder. One of the family members wanted the ability to burn from the Photo Gallery as well, so we put it in. Another family would probably point to DVD Maker. In this case, a family member wanted to take photos of his kids and put them on a DVD and send them to the kids’ grandmother. He thought it would be great if Microsoft had a feature that would walk him through really quickly and spit out a DVD that grandma could play on her DVD player and watch on her TV. We added that capability in later versions of the product and he was delighted.
Another area where the families had a lot of input was readability on the transparent glass of open windows. Early versions of the glass were very transparent, so people had challenges reading the information on a window. The design team did a lot of work and iteration to retain the glassy look and feel but still have the readability that people needed. We relied heavily on feedback from the families to find the sweet spot there. One of the things we hear from the families over and over is that they saw changes based on their feedback. That’s what kept them going and kept them committed when they had an early build that was kind of slow or kept crashing. Each build they would see changes.
PressPass: Is the family trial program here to stay for future versions of Windows?
Russell: Definitely, because this type of input is absolutely essential. We want to use as broad a set of feedback and mechanisms as we can, but the families are an extremely important and unique type of feedback. It isn’t some anonymous, scrubbed, academic, bureaucratic report. It’s very earnest and honest feedback from real people using our product in real ways and you really can’t duplicate that.
Hoek: In general, these kinds of programs tend to evolve on. We’ve been doing user-involved software development for years, and every time we get a little smarter and better about it and extend these programs. I am really impressed with how incredibly positive the families have been about this experience because, truth be told, these people did some pretty challenging things. They worked with really early builds. This was their only computer in the house and they needed to use it. They made a real investment.
Miner: And just as we added new things this time around, I’m sure in the next incarnation you’ll see new things and improvement so that we can have greater impact. We already have some families we have recruited to continue where our Life with Windows Vista families left off, so we’re already moving on to look at the next versions we’ll be working on.