White papers
An enterprise simulation platform for education:
Building a world game for pre-college students with Microsoft ESP
Exhibit 5: Assessment in virtual environments
An automated assessment system works to provide the student, teacher, or researcher with a rich stream of information over time. As the user interacts with the game, a computer collects and analyzes data regarding the user‘s performance in real time and then provides a set of graphic, textual, and numerical representations of the information collected, either during the session or shortly after.
Recent work stemming from adaptive testing (Almond, Steinberg, and Mislevy 2002) shares a core of ideas with other research on assessment, which holds that every assessment of student learning involves three fundamental components: “a model of how students represent knowledge and develop competence in the subject domain, tasks or situations that allow one to observe students‘ performance, and an interpretation method for drawing inferences from the performance evidence thus obtained” (Pellegrino, Chudowsky, and Glaser 2001, 2). Mislevy, Steinberg, and Almond (2003) refer to these as three components of a conceptual assessment framework (CAF): the student model, the task model, and the evidence model (Figure 5-1).
Figure 5-1. Components of a conceptual assessment framework
The student model in a computer-based automated assessment is a computational representation of “the knowledge, skills, and abilities to measure for each participant” (Almond, Steinberg, and Mislevy 2002, 35). The task model includes “the presentation material to be presented to the user [and] a description of the work products that will be returned as a result of user interaction with the task” (24). The evidence model acts as a bridge between the student and task models in that it describes in numerical and text variables how to analyze the evidence called for by the task model in order to assess the student‘s understanding. Each of these models is represented in concrete terms as variables being captured, stored, and analyzed by a computer.
In planning for assessments in a virtual environment for learning, the CAF framework can help guide design and implementation decisions. For example, the task model characteristics can be used to guide decisions about how to represent what the user needs to know and do (Table 5-1). Different states of key variables can be recorded at a variety of times — for instance, before, during, and after the user encounters something in the virtual world or practices a skill within the learning environment — to analyze the difference between actual student performance and a target performance and to show change over time.
Prompt or Challenge | How does the game or sim set a context? What’s the story line or setup? |
Content | What does the learner need to know already versus need to acquire during play in order to succeed? |
Validity | What assumptions about the real world are embedded in the game engine’s world model? |
Performance opportunity | Is relevant performance evidence produced? |
Affordance | What does the object, tool, situation allow the user to do that creates variables for the computer analysis of the evidence? |
Rich picture | Are several kinds and pieces of evidence elicited? |
Trustworthiness | Is the user’s record of interactions with the computer reliable for making inferences about what they know and can do? |
Ideal versus real variables | What are the ideal game states for performances that exhibit the knowledge and skills of interest (for example, what is the target state of a numerical variable that represents the best answer or performance) and how does each user‘s actual performance (for example, the number stored when the student attempted the performance) compare with those states and change over time? |
Table 5-1. Example task model characteristics
In the evidence model, quantitative evidence of the student‘s entry, midpoint, and ending status on important variables are recorded and analyzed to create an estimate of what the student knows. For example, if the expected performance of an expert is to complete an array of x and the student is currently completing an array of y, the computer can model that difference along an array of complex variables. The computational representation of an expert performance in computer variables captures an expert community‘s common vision of excellence. This is a trivial matter to illustrate when the expert performance is a “right answer” to an objective item on a test, but such representations can also be generalized to assess more complex performances when there is more than one right answer and when the performance requires more subtle uses of knowledge and skills in a computer-based learning environment. In addition, the evidence model analysis can be used for other purposes, such as adapting the game play, giving feedback to the user, and documenting change in knowledge and skills over time.
Return to top
References
Almond, Russell, Linda Steinberg, and Robert Mislevy. 2002. “Enhancing the Design and Delivery of Assessment Systems: A Four Process Architecture.” The Journal of Technology, Learning, and Assessment 1 (5). http://escholarship.bc.edu/cgi/viewcontent.cgi?article=1008&context=jtla (accessed May 27, 2008). Archived at http://www.webcitation.org/5XWimvcb2.
Mislevy, Robert, Linda Steinberg, and Russell Almond. 2003. “On the Structure of Educational Assessments.” Measurement: Interdisciplinary Research and Perspectives 1:3-67.
Pellegrino, James, Naomi Chudowsky, and Robert Glaser, eds. 2001. Knowing What Students Know: The Science and Design of Educational Assessment. Committee on the Foundations of Assessment, Board on Testing and Assessment, Center for Education, National Research Council. Washington, D.C.: National Academy Press.
Return to top
