Customer Input Key to Developing MCP Exams
If you were to peek into the halls of the Microsoft Training and Certification group headquarters, you'd see people walking briskly to conference rooms and offices, yellow legal pads under their arms, and Microsoft coffee cups in their hands. What you might not realize is that some of these busy people aren't Microsoft employees—they're information technology (IT) professionals who help create the Microsoft certification exams.
"From the conception of the exam until its release, customers are involved every step of the way,” says Dan Truax, Director of Certification. This involvement includes giving feedback on already developed exams, working with the development group with job task analysis, creation of exam questions, and the testing of beta exams. “Customers have a huge impact on the exams," says Truax.
Listening to customer feedback
Truax interacts mostly with IT pros who have something to say about the exams already on the market. His team is in charge of product planning and product management for certification. The team actively seeks out feedback from customers about their needs so the team can then design the right products to meet those needs.
"We're talking to customers almost on a daily basis," Truax says. "We spend a lot of time in newsgroups, interfacing with customers. I hold specific online chat sessions where the whole goal is to discuss with customers what they think about our program, what they think about our exams, what we can be doing better, and what we're doing well. We also hold many focus groups, and we go to events and speak to customers there."
The result of all this active listening is that the exams Microsoft Training and Certification develops are almost always the result of customer feedback.
Creating job-relevant exams
One piece of feedback that the team kept hearing was that IT pros and the IT industry needed the Microsoft certification exams to be more relevant to the jobs that IT people perform in their daily work.
"In the exam development process, many companies—including Microsoft historically—start with the products and then look at what people do with them from a feature perspective," Truax explains. "The problem with that method is that the way we might think customers use a product doesn't always turn out to be exactly the way they use them in the real world."
As a result of this feedback, the team now starts by looking at job roles and tasks that are most important to people in their daily work. Called job task analysis (JTA), this part of the exam development process identifies all tasks that make up a specific job function. Then, those tasks are mapped back to Microsoft products in order to create the exam questions. The first exams to be developed using JTA focused on Microsoft Windows 2000.
January 2001 marked another milestone for this process. "The Microsoft Certified System Administrator (MCSA) exam was another big step because we completed a new analysis of that job role," says Truax. Going forward, this type of analysis will take place for future versions of the Microsoft Certified Application Developer (MCAD) and Microsoft Certified Systems Engineer (MCSE) certifications. Truax adds, "I think where we're going to take even bigger steps is around Windows Server 2003 when we launch our updates to the IT professional exams. They will be totally redesigned based on a brand new task analysis."
Many ways for customers to participate
Customers are involved in exam development not just in the JTA phase, but throughout the entire process. As Truax notes, "We have a community of about one and a half million people that are certified, so it gives us a great customer base, a great community to interact with, and great pool of very skilled people."
Often, the development teams engage customers who have participated in the Joint Development Program (JDP). Through the JDP, some customers adopt new Microsoft technologies before release, which lets the exam development group talk to people who have used unreleased products and thereby derive test content early.
In addition, the team often finds customers with relevant experience through the Microsoft Partner Advisory Councils. These councils are composed of 30-50 Microsoft partners who are focused on a particular initiative or technology area and meet two or three times a year to give Microsoft feedback on its programs and plans.
If customers receive the MCP News Flash and demonstrate expertise in a particular technology, they may be invited to take beta exams, allowing them to participate in the exam development process. Taking the free beta exams allows IT pros to be some of the first customers to be certified, and gives them a chance to provide feedback on the exam questions.
Another way that customers take part in the exam development process is by helping to define the objective domain of the exam, where job tasks are translated into measurable skills. "In this phase, customers help us scope the content for a particular exam and define the exact tasks," says Mary Beth Thome, Content Development Manager.
Other customers are subject matter experts (SMEs) who review exams to ensure technical accuracy and relevancy of exam questions to the skills being tested. SMEs also create exam questions by: turning exam objectives into testable exam questions; taking part in the alpha technical review of exam questions; helping to select questions that go into the final version of the exam; and helping to set the standards for each exam.
While some customers participate in the whole exam development cycle, most simply attend one or two meetings. For many customers, this is a rewarding process in which to be involved, and they continue to be involved for many years (see sidebar). "The customers who help us in the exams are very dedicated to making sure that they're doing this the right way and that they're helping us measure the right things because they understand what kind of an impact it's going to have," says Truax.
For more information
| • | How MCP exams are developed |
| • | Beta exam process ensures credential validity |