This example is similar to your assignment and will help you become familiar with the affective evaluation, the intrinsic motivation inventory (IMI) questionnaire, and the cued debrief recall technique.
First, find a simple online game that you can evaluate. It should be easy to startup, learn, and get into real play. Just do a web search for free online games if you aren’t sure of what to play.
Second, determine how you will capture the first person view of the player playing the game (either screen capture, or over the shoulder to capture player’s head/hands and screen if possible). Some screen capture options are QuickTime, OBS (open broadcast software), Windows built in screen record (Windows key G in Windows 10).
Third, you will also need a way to record the debrief session with the player and evaluators so that you can each look at that video to code it after.
Study
Pick one person from your team who has access to a device to play the online game, a device to record themselves playing (recorded from behind — see below) and a quiet place to work. Ideally you would have worked this out ahead of class time.
One person from the team will play the game and the other team members will conduct the study as investigators.
1. The player will first familiarize him/herself with the game by playing for 5 minutes.
2. The player plays the game for one level (e.g., 5-10 minutes) and is video recorded from a first-person view (behind them facing the screen). You can use a cell phone or webcam from a laptop to record the video. Share the screen and video of yourself while you are doing this using Zoom.
3. The investigators conduct the cued recall debrief technique by reviewing the participant’s video of them playing and asking about their play at various points in time. They should video record the participant in a third-person view while they watch their own video and talk about it. You should be able to do this in Zoom.
4. The player fills out the IMI questionnaire. Using google drive so everyone can see the results.
Analysis
Now perform the analysis. You must use the answered IMI questionnaire along with the video of the person talking about their play when they did the cued recall debrief.
1. Analyze the cued recall debrief video and fill out the coding sheet, one sheet per investigator (do this separately, then share and compare).
2. Compare the coding sheets of the two investigators by computing the inter-rater reliability. Are they consistent? Is there a correlation? You should have an R-value >= 0.90. If you don’t, discuss the codes that each evaluator chose and understand why there were discrepancies.
3. Score the IMI questionnaire using the scoring sheet and scoring instructions.
Use the results to assess the overall affective enjoyment of the gameplay, problem areas with the design, and areas of success.