IAT432 – Assignments

Assignments – 35% Individual and 40% Group

  • #1 Ethics – Individual (5%) – due at the start of class (2:30 PDT), May 19 by email to TA Niloofar Kazemi [nka51@sfu.ca].
  • #2 Controlled Experiment – Individual (15%) – due at start of class, May 28 by email to TA Niloofar Kazemi [nka51@sfu.ca].
  • #3 Affective Evaluation – Individual (15%) – due at start of class, June 9 by email to TA Niloofar Kazemi [nka51@sfu.ca].
  • #4 Evaluation Project – Group (40%) due 5 pm PDT, June 23, by email to Instructor Alissa Antle [aantle@sfu.ca] and TA Niloofar Kazemi [nka51@sfu.ca].
  • Bonus — 2% due 5 pm PDT, June 24, by email to Instructor Alissa Antle [aantle@sfu.ca].
Midterm – 25%

 

Assignment 1: Ethics – Individual (5%)

When conducting design evaluation studies with people as participants, it is very important to follow ethical practices.  The Government of Canada has created guidance for conducting studies with humans that you need to follow when working or being a student at Simon Fraser University.  It is called the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans or TCPS2 for short.  Even once you are finished your time at SFU, it is advised that you continue to follow approved ethical standards when conducting studies.

For this assignment you must complete the TCPS2 course online at this address individually:

https://tcps2core.ca/welcome

The tutorial will likely take you two to three hours to complete if you go through all of the modules properly and read the content.  Email a copy of your completed certificate to the TA before class on the day this assignment is due.
 

Assignment 2: Controlled Experiment – Individual (15%)

This assignment is a hands-on exercise on quantitative evaluation. Its immediate purpose is to give you experience conducting a controlled experiment, performing a simple statistical analysis, interpreting the results, and considering its implications to design decisions. Its other purpose is to provide you with enough knowledge of the experimental process to help you understand and appreciate the interaction design evaluation and HCI literature that uses this methodology.

Imagine you have been asked by a company to evaluate their product where they are interested in knowing if Interface Style A is better than Interface Style B.  You will plan and perform a controlled study and create a study report detailing the findings of your study and recommendations for system design.

You will complete the assignment individually.

Steps

1. Plan the Study:

  • Decide on a Design to Evaluate: Choose a design that you can evaluate and compare to a different design. For example, imagine driving a telepresence robot using one of two techniques: a mouse and keyboard vs. a gaming controller. However, you will need to choose a design that your participants can access remotely. 
  • Hypotheses and Variables: You need to decide on hypotheses and null hypotheses, as well as independent, dependent, and control variables. You will need to ensure you can measure your dependent variables remotely. For example, you could run the experiment with a screen share to measure task time and error, or you could have your participants record data related to dependent variables and report it back to you.
  • Pre-Test Questionnaire: include questions that ask the user about their general demographics, computer skills, how often they have used user interfaces like those in question, and what they think of the user interfaces in question.
  • Post-Test Questionnaire: include questions that ask the user how they felt about the user interfaces, what was easiest to do, what was hardest, what suggestions they would have for improving the interface.
  • Representative Tasks: Participants will use both interfaces . You need to design a series of tasks for them to do using the system. Try to construct tasks such that they will take about 10-20 minutes per participant.

2. Perform the Study: Normally, you’d run around 20 people in a study like this but we will reduce that number because it is a class assignment. You must run the study with at least 6 people.  In total, this shouldn’t take more than a couple of hours to do if you plan your tasks and remote data collection appropriately.

3. Analysis: Analyze your findings using known statistical methods that are taught in class.

4. Report: Create a report that details your findings.

Deliverables

1. Introduction: describe the situation you are studying and why

2. Study Methods:

Participants: describe your participants’ demographics briefly.

Hypotheses: describe your hypotheses and rationale for them.

Method: explain that you performed a controlled study and describe your study steps including the tasks. Use figures to illustrate the tasks.

Environment: describe where you conducted the study and how you controlled the environment.

Data collection: explain how you collected your data.

Validity: describe any potential concerns with validity; talk about how you ensured validity.

Questionnaires: place your completed questionnaires in an appendix.

3. Results: Describe your data and statistical tests. Include any graphs to show your descriptive statistics.

For your statistical testing, report:

  • What data we are running the t-test on.
  • The result (p-value).
  • Whether you can reject the null hypothesis and what this means.

For example (M is mean, SD is standard deviation):

“An independent-samples t-test was conducted to compare memory for words in sugar and no sugar conditions. There was a significant difference in the scores for sugar (M=4.2, SD=1.3) and no sugar (M=2.2, SD=0.84) conditions; p = 0.02. Thus, we can reject the null hypothesis at a 95% confidence interval (p < 0.05). These results suggest that sugar really does have an effect on memory for words. Specifically, our results suggest that when humans consume sugar, their memory for words increases.” (Source)

4. Discussion and Conclusions: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?

5. Appendix 1: include all raw data from your study. You need to Include your Excel spreadsheet twice. This might include printing out multiple data sheets/tabs within Excel depending on how you structure your spreadsheet:

– Show the Excel spreadsheet with the data output (e.g., the mean, standard deviation, p values)

– Show the Excel spreadsheet with formulas visible (to show formulas, see the instructions on this page)

6. Appendix 2: include your pre and post-test questionnaires

Email a PDF copy of your report to the TA before class on the due date listed in the course timeline (and announcements). Failure to email it in at the start of class means it will be considered late.

Resources 

Example Instructions To Participants

Example Pre and Post-Study Questionnaire (yours should be more detailed)

Post-Study Questionnaire (System Usability Scale)

Example Data Analysis

Marking Guide

Assignment 3: Affective Evaluation – Individual (15%)

You are going to evaluate a video game’s user interface for emotional and affective experiences. Your goal is to find out how participants felt during and after game play. You will use this information to suggest improvements to the design of the game.

The assignment must be done individually; however, you will need to recruit a player, who may or may not be a classmate, and there is one portion where you must pair up with a classmate to analyze the data (reliably). You should be able to do all this work remotely as needed.

The Location

Ideally you will run this as a (remote) field study. This means that you should evaluate the player in any environment where you might naturally find them playing these kinds of games. For example, you could host a video game party with several friends playing remotely and then conduct the evaluation of one player in their living or party room. If this is not possible due to Covid, you can run the study remotely with your single participant playing a video game on their laptop in their own home space. You will need to ask them to set up their camera (for video) to record themselves playing if you will not be there. You should explicit in your report that you ran the study remotely as a field study unless you are able to run the study remotely with several players (one of which you will capture data for) or in person with a person/people you are sheltering with.

The Evaluation Methods

You will be evaluating each of participant’s affective experiences using two methods:

1. A simple post-play questionnaire called the Intrinsic Motivation Inventory (IMI) which measures the overall emotional experience of enjoyment of the game or activity.

2. The cued-recall debrief method is a first person, video-based method of reviewing and talking about the participant’s play session with the participant after the play session in order to elicit information about the short term affective states the player experienced during the game or activity.

Study Protocol

A study session should look like this for each participant.  You will run the study with just one participant.

1. Play: One participant will play a game for about 15 minutes. Their experience will be video recorded in first person – the video should capture what the participant is seeing. That is you/they will need to place a camera directly behind them or mounted on their head to capture what they see while they play.

2. IMI Questionnaire: After they are done playing, the participant is to fill out the IMI questionnaire using the enjoyment, perceived competence, perceived choice and pressure/tension subscales.

3. Cued Debrief Recall: Immediately after the questionnaire part is the review or debriefing part. One team evaluator and the participant will sit together and review the video of the play session and “debrief” or talk about what the participant was feeling during the play session. The participant will talk about this as they watch the video of the session. You may need to use prompts (like in think aloud) to encourage them to talk about what they were feeling. The debrief part should also be video or audio recorded. This means you will need to find a quiet place where you can replay your video and discuss the session. Note that you need one camera to play the video (or alternative play back display) and a second camera to video record the interview (focusing on audio).

When you are done the session you will have a video of the participant playing (shot in first person) and a video of the participant talking about their play (shot in third person).

Data Analysis

1. IMI Enjoyment Questionaire

The Intrinsic Motivation Inventory (IMI) is a multidimensional measurement device intended to assess participants’ subjective experience related to a target activity in laboratory (or contrived field) experiments. The questionnaire assesses participants’ interest/enjoyment, perceived competence, perceived choice and felt pressure and tension while performing a given activity, thus yielding four subscale scores for our purposes.

The interest/enjoyment subscale is considered the self-report measure of intrinsic motivation. Although the overall questionnaire is called the Intrinsic Motivation Inventory, it is only the one subscale that assesses intrinsic motivation, per se. As a result, the interest/enjoyment subscale often has more items on it that do the other subscales. The perceived choice and perceived competence concepts are theorized to be positive predictors of both self-report and behavioral measures of intrinsic motivation.

Pressure/tension is theorized to be a negative predictor of intrinsic motivation.  You will score the IMI questionnaire using the procedure described in class.

2. Affective Content of Cued Recall Debriefing Sessions

After the play-debrief sessions are finished, you will analyze the debrief video/audios (i.e., not the video of playing). To do this you need two evaluators – thus, you need to pair up with someone else from the class for this part of the assignment.

Two evaluators will each review each debriefing session to listen for affect comments. They will record these comments on a data sheet.  Have two evaluators individually (i.e., separately) review each participant’s debriefing session. Focus on what the participant said (i.e., audio only). Your goal is to write down (on the data sheet) any comments the participants make which contain affective words or contain affective expressions. Count (add up) the total number of positive, neutral and negative comments.

3. Inter-rater Reliability on Cued Recall Debrief Affect Comment Coding

For each session/participant, compare the results of the two evaluators/raters. Did you find the same kinds of affective statements? Did you find the same number of for each kind? Did you find the same overall number of positive, neutral and negative affect comments?
You also need to calculate the inter-rater reliability value for each session. The inter-rater reliability or R value indicates how similar the results from the two evaluators are. For each session your R should be better than .90. If it is not, have the two evaluators work together to reach a better consensus.

Deliverables

You will provide a detailed report (maximum 2 pages in length, single spaced + appendices). It should include the following sections:

1. Introduction: describe the situation you are studying and why.

2. Methodology: describe your study methods.

  • Participants: describe your participants’ demographics briefly.
  • Method: explain that you performed a field study and describe your study steps.
  • Data Collection: explain what data was collected and how.

3. Results: describe your data and the generalized results.

  • What does your analysis of the IMI/Enjoyment questionnaire data tell you about each of the player’s experiences with game play?
  • What does the affective content analysis of participant’s cued recall debriefing session tell you about each of the player’s experiences with game play?
  • For each participant, were there any parallels or inconsistencies between the questionnaire and the cued recall debrief results?

4. Discussion and Implications: describe what the implications are from your results. What should be do moving forward based on the results of your study? What design implications does it present?

5. Conclusion: conclude your report by summarizing the study and its findings.

6. Appendices: include all raw data from your study.

Email a PDF copy of your report to the TA before class on the due date listed in the course timeline (and announcements). Failure to email it in at the start of class means it will be considered late.

Resources

Cued Recall Coding Sheet

IMI Scoring

IMI Scoring Instructions

Interrater Reliability Example

Marking Guide

Assignment 4: Evaluation Project – Group (40%)

You will work with an industry or community partner to conduct a real world evaluation of one of their products.  This will involve you meeting with the partner and exploring their needs for design evaluation.  You will think carefully about the design challenge posed to you by the company or organization and decide on the best method for evaluating the design.  The evaluations that you can consider are:

Heuristic Evaluation

Usability Study

Controlled Study

Affective Evaluation

You will be graded based on the quality of your work as well as the complexity of the design evaluation you conduct.  For example, conducting a fairly straightforward heuristic evaluation of a single user application will likely not generate a strong grade.  On the other hand, a field deployment of a new user application that requires multiple site visits and interviews will likely generate a stronger grade given the complex nature of the study.

You will work in a group of four people.

Steps

The following steps are presented in the course calendar based on when they should occur:

1. Industry or community partner:  You will first need to find an industry or community partner that you can work with.  You will have to meet with them to explore their needs and what design you can evaluate.

2. Study Plan:  You will draft a study plan that document how you could conduct a study.   This should include all components, including the questionnaires, study tasks, interview questions (if needed), etc.  You should also write one page that explains why you would not choose other methods to conduct the study.

2. Perform the Study: Conduct the study with an agreed upon number of participants, as detailed in class.

3. Analysis: Analyze your findings using known methods that are taught in class.

4. Report: Create a report that details your findings.

Deliverables

You will provide a detailed report that is 4 pages, single spaced (not including appendices and figures). It should include the following sections:

1. Introduction: describe the situation you are studying and why

2. Study Methods:

  • Participants: describe your participants’ demographics briefly.
  • Method: explain the method you used and exactly what you did.
  • Data Collection: explain how you collected your data.
  • Validity: describe any potential concerns with validity; talk about how you ensured validity.
  • Questionnaires: briefly describe them; place your questionnaires in an appendix.
  • Data Analysis: describe how you analyzed your data.
3. Results: Describe the main themes from your results.  What did you learn by conducting the design evaluation?

4. Discussion and Conclusions: Describe the implications from your results. What should be done moving forward based on the results of your study? How should the user interface be redesigned as a result of your study?  Show mockups of potential redesign ideas.

5. Appendix 1: include all raw data from your study.

6. Appendix 2: include any pre and post-test questionnaires, etc.

Email a PDF copy of your report to the TA and to the instructor  TA before class on the due date listed in the course timeline (and announcements). Failure to email it in at the start of class means it will be considered late.

Bonus Assignment

You can earn up to 2% bonus by participating in designated research studies within SIAT as a learning experience to broaden your understanding of research in interactive arts and technology. You can earn a 2% bonus for participating in a study that is 30 minutes or longer, or 1% for a shorter study (for a max of 2%). All studies during the pandemic (social distancing) are online.

The studies are posted in the study management system: https://sfu-siat.sona-systems.com/ (Links to an external site).

You need to create an account. In your profile, you can select a default course to apply credits to, or you can decide later.

Beware, that there is a limited number of slots for each study, and it is quite unpredictable when the new studies will be ready. If you see the study that interests you, go and do it.

Once you participate in a study, complete a User Study Reflection form and email it to the course instructor and TA.

Alternative Bonus Assignment

Choose one article on a research study.

Article 1: Buhler T, Neustaedter C, Hillman S. How and why teenagers use video chat. In Proceedings of the 2013 conference on Computer supported cooperative work 2013 Feb 23 (pp. 759-768).

Article 2: Antle AN, McLaren ES, Fiedler H, Johnson N. Evaluating the impact of a mobile neurofeedback app for young children at school and home. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 2019 May 2 (pp. 1-13).

Read the article, complete the Research Study Reflection form and email it to the course instructor.

Course Consent Form

This course has ethics approval from Simon Fraser University.  This means that any students registered in the course are permitted to conduct studies with human participants.  In doing so, you are required to use this consent form, which participants should read and sign before participating in a study:

IAT 432 Course Consent Form.