IAT 432 – Timeline (Course Calendar)

IAT 432 - Winter 2021

IAT 432 Design Evaluation is being taught online for the Summer 2020 & Winter 2021 sessions.

This is an upper division undergraduate course taught in the School of Interactive Art and Technology at Simon Fraser University in Surrey, BC, Canada. It explores the evaluation of user interfaces using a variety of design evaluation methods from the fields of human-computer interaction and interaction design.
Note: This is not an “online course”. It is a course being taught online. We will need to have flexibility and patience with another as
we learn this together. The following course schedule for IAT 432 is subject to change.
Instructor: Alissa Antle [aantle@sfu.ca]
TA: Niloofar Kazemi [nka51@sfu.ca]

Technology: You will need a laptop or PC, a stable internet connection, a web browser that can run Zoom (Google Chrome works well, you do not need the Zoom application), Microsoft Office available to SFU students  (Word and Excel), and the Slack application (download Win, Android or Mac). You will also need Flash for some in-class activities (check that pop-ups are not blocked). Let us know if you have concerns with the technology requirements. 

If you do not have access to a laptop or PC or have adequate internet etc … , see SFU Financial support for student technology equipment (see Need emergency assistance for technology-related funding?) 

#1 Winter 2021

Wed Jan 13 9:30 – 12:20 

To join today's class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted using the on/off buttons lower left of your Zoom screen. 

If you cannot join Zoom, please  contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888. 

If you do not get the Zoom link in an email or cannot attend, please email TA Niloofar Kazemi [nka51@sfu.ca]. 

Course Logistics [about an 1 hour] 

Zoom Whole Class

Check in Zoom — everyone on board and muted? practice raising hand, chat etc. 

Check in technology accessibility and other COVID issues.

Brief review course and communication expectations.

Today's Plan

Offline (15 minutes): Students go through Course Logistics [Slides]

Focus questions : Syllabus & Logistics — What are learning outcomes for course? Do you know what work are being assessed on (% each)? When are the assignments due? What is the group project about? When do I need to be on a team?  

What grade scale is being used? What happens if your assignment is late? Do you have to attend class times? What are required texts? 

Do you know how/when and what to expect re contacting your instructor/TA? What are key things you need to do to succeed in course? 

Other — What is clear? What is not clear? What do you need more information about? 

Zoom Breakout Rooms (10 minutes) — Breakout groups will discuss Focus Questions and report back to class.

Zoom Whole Class (10 minutes): Talk through Focus questions.  

Break

Course Content — Key Concepts for Design Evaluation [about an 1 hour]

Zoom Whole Class:  Go through Concepts and Evaluation Framework [Slides]

Zoom Breakout Rooms (10 minutes) Focus questions: Download these focus questions and work through the questions assigned to your breakout group — bring questions back to class.

Zoom Whole Class (10 minutes): Q&A + Homework — Use Slack Channel to answer focus questions and get feedback — one Q per thread. 


Zoom Optional: Q&A and Instructor and TA Office Hours (N/A first week)

Today's Learning Outcomes are to orient you to the course, go over course tools, set course expectations and communication guidelines, and give you key terms you will need to understand to succeed in  the course. 

Due before the start of next class — You will work in small groups of 4 people in the course and for the project.  Have one team member email TA with your preferred small working group of 4 people and CC your team members. If you do not do this, you will be assigned to a working group. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time : 1 hour)

Read Introductory Course Email that you should have received about the course. Respond by email (“IAT 432" in subject line) to instructor and TA if you have any technology accessibility issues, and/or challenges around online learning and COVID-19 that may impact your ability to participate in this course (e.g. different time zone, child or elder care responsibility, you are ill, mental health issues). If you have formally diagnosed disabilities, please contact the Centre for Accessible Learning (CAL) to set up special accommodations.  

Read Course Logistics Introduction — attached to introductory email you and here [Slides]. Bring questions to class.

Review: Syllabus including  IAT 432 Learning Outcomes

Course Web Site: Please look over and familiarize yourself the web site including assignments and assessment. Bring questions to class for the Zoom session.

Please join: SLACK course channel.  Please review netiquette resources.

Please review: SFU student conduct, what is academic integrity [video] and SFU's student academic integrity policy.

 

#2 

Wed Jan 20 9:30 – 12:20 

To join today's class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted. 

If you cannot join Zoom, please  contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888. 

If you do not get the Zoom link in an email or cannot attend, please email Niloofar Kazemi [nka51@sfu.ca]. 

Zoom Whole Class (15 minutes)

Check in Zoom — how was last class? questions? 

Today's Plan


Course Content —  IAT 201/334 Review (see Preparation (right column) slides/audio and readings) 

Questions from Slides and Audio Content?

Activity: Zoom Breakout Groups 

Meet your working group (mini-icebreaker).

Work through Review heuristic evaluation

Zoom Whole Class: Check in Heuristic Eval Activity

Introduction to Assignment 1: Ethics (5%) [allow 2-3 hours outside of class for this assignment]


Learning Outcomes for today are to review and summarize key materials from IAT 334 focusing on heuristic evaluations that you will need to use in the course and/or will be on the exam, and meet your small working group.

Prepare for Class — Please Do These Things Before Class  (estimated preparation time 2 hours) 

Content: Go through Slides and Audio for these Review topics

Heuristic Evaluation (Slides, Audio) [1:06 ]

Readings: As we go through the course and learn about each evaluation method, you should progressively scan, skim then read through Chapters 1-12 of this Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests” (2008) by Jeffrey Rubin, Dana Chisnell, Jared Spool (listed under Resources). The basic framework introduced in the first class, is expanded on in these chapters (what, when, who etc to evaluate?) as well as data collection, analysis and reporting information.  

 

#3 

Wed Jan 27 9:30 – 12:20 

Zoom Whole Class: Check in / this week is pretty easy! 

Course Content: walk through usability evaluation materials

Zoom Breakout Activity: Work through Review usability studies

Zoom Whole class: Share Usability Activity

Learning Outcomes for today are to review and summarize key materials from IAT 334 focusing on usability evaluations that you will need to use in the course and/or will be on the exam, and meet your small working group.

DUE: Assignment 1: Ethics (5%) Due at the start of class. Email ethics certificate to TA Niloofar Kazemi [nka51@sfu.ca]. 

Prepare for Class — Please Do These Things Before Class  (estimated preparation time 2 hours) 

Content: Go through Slides and Audio for this Review topic:

Usability Testing Plans and Methods (Slides Audio) [43:44]

Readings: As we go through the course and learn about each evaluation method, you should progressively scan, skim then read through Chapters 1-12 of this Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests” (2008) by Jeffrey Rubin, Dana Chisnell, Jared Spool (listed under Resources). The basic framework introduced in the first class, is expanded on in these chapters (what, when, who etc to evaluate?) as well as data collection, analysis and reporting information.  

#4

Wed Feb 3 9:30 – 12:20 

Zoom Whole Class – this week is intense! 

Add feedback on course so far and/or questions on assignment 2 to SLACK channel in assignment questions.

Course Content — Controlled Experiments — Topics: 

  • Scientific Method
  • Hypothesis testing
  • Independent & dependent variables
  • Ethics of controlled experiments
  • Reliability and validity
  • Between and within experimental designs

Zoom Lab (Breakout) Activity 2 (20 min): Work through controlled experimental study design with your working group using one interface set (will be assigned in class) — report back.

Exercise Sheets

Interface Set 1 Mac Tray

Interface Set 2 – Track pad vs. mouse

Interface Set 3 – Ribbon menu vs. traditional menu

Interface Set 4 – Wii remotes

Zoom Lab (Breakout) Activity 1 (10):  Work through Focus Questions on this weeks topics based on slides/audio/readings. Report back/questions.

Break 

Zoom Lab (Breakout) Activity 3 (60 min):  Work through with your working group Design an experiment for AimBooster – report back.

Learning outcomes or today are to understand and be able to apply the basic components of a controlled experiment in order to design a study to evaluate different interfaces and to assess a study design against criteria for reliability and validity.  

Prepare for Class — Please Do These Things Before Class (estimated preparation time 5-6 hours) 

Content: Go through Slides and Audio for Controlled Experiments (Slides, Audio) [18:00]

Content: Go through Slides and Audio for Reliability and Validity (SlidesAudio 31:38 minutes)

Required Reading

Doing Psychology Experiments, Chapter 2: How to Do Experiments

Doing Psychology Experiments, Chapter 7: How to Decide Which Variables to Manipulate and Measure 

Doing Psychology Experiments, Chapter 8: How to Decide on a Between-Subjects versus Within-Subject Design

 

#5

Wed Feb 10 9:30 – 12:20 

Zoom Whole Class: Walk through Assignment 2: Controlled Experiment (15%) [individual mark]

Course Content — Statistics for Controlled Experiments

Graphing frequency distributions

Descriptive statistics: mean, standard deviation, median, range

Graphing mean with error bars

Basic Inferential tests: t-test

Demo Excel (large/small mouse)

Questions from audio/slides/readings?

Zoom Lab (Breakout) Activity:  Work through exercise and check answers with your working group — ask for help as needed. Hand in exercise — individually. 

Exercise on statistics (compare two keypad designs, two data sets)

Note: Error in ans key: study design is within design so t-test parameter for type should always be 1 (paired) — not 3.

Excel Commands Cheat Sheet

Answers for the exercise on statistics (try to do the work first, then check!)

Learning outcomes for today are to understand and be able to run basic statistical techniques on data gathered in different forms of controlled experiments; interpret results in context of evaluation goals and design iteration; and understand the limitations of quantitative results.

Review: Assignment 2: Controlled Experiment (15%) [individual mark]

Prepare for Class — Please Do These Things Before Class (estimated preparation time 4 hours) 

Content: Go through Slides and Audio for Statistics (SlidesAudio 33 minutes)

For exercises, you will need Excel to have the Data Analysis Toolpak installed (for older versions of Excel) or the Solver analysis package installed (for newer versions of Excel). Instructions for how to get it are here for Mac or PC.

Required Reading

Doing Psychology Experiments, Chapter 12: How to Interpret Experimental Results

Doing Psychology Experiments, Chapter 13: How to Report Experimental Results 

#6 No Class 

Wed Feb 17 

Reading Week

DUE: Assignment 2: Controlled Study (15%) Due today. Email to TA Niloofar Kazemi [nka51@sfu.ca]. 

Catch up on readings.

Work on preparation for next week/assignment #3, mid-term exam practice questions, and setting up final projects — you should have a client lined up and know what your evaluation goals are.

#7

Wed Feb 24 9:30 – 12:20 

Evaluating Emotion and Affect

Measuring affect and emotion

Theories of emotion

Quantifying qualitative codes

Cued Debrief Recall

Studio Lab Activity (2.5 hours in classtime). See activity (below)  — choose participant, online game and practice video recording set up ahead of time.  Work out how you will share screens/data. 

Conduct an example study of gameplay and collect IMI questionnaire data and cued debrief recall data

 Learning outcomes for today are to understand what affect and emotion are, learn how to measure each understanding the limitations of different approaches, understand the basics of questionnaire and interview methods, learn to code and assess reliability of qualitative video data and then quantify that data, understand and be able to apply the cued recall method working remotely. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time 4.5 hours) 

Content: Go through Slides and Audio for Evaluating Emotion and examples  (Slides, Audio 30 minutes)

Pac Man game as an example cued debrief recall example

Inter-rater reliability example

Content: Go through slides and audio for Interviews, Surveys, Questionnaires including retrospective methods for assignment 3 (Slides  Audio) [36:40]  and for Qualitative Data Analysis (Slides Audio) [26:14]

Review: Introduction to Assignment 3: Affective Evaluation (15%) [individual mark]

Required Reading

Bentley, T., Johnston, L., and Baggo, K. Evaluation using cued-recall debrief to elicit information about a user's affective experiences, Proc. OzCHI, ACM Press (2005). (PDF)

#8

Wed March 3 9:30 – 12:20

Studying Field Deployments (see audio/slides)

Planning and time-frames

Managing home visits and remote “visits"

Minimum viable prototypes

Ethics for field evaluations, ethics of collecting data remotely

Remote methods for fieldwork (see reading)

Assignment #3 check in 

Studio Lab Activity – Practice questions for exam

Learning outcomes for today are to understand the main considerations in doing field evaluations, and how to do them remotely.

Prepare for Class — Please Do These Things Before Class (estimated preparation time 3 hours)

Content: Go through Slides and Audio for Field Evaluations SlidesAudio (30 minutes)

You should be working on assignment 3 and studying for your midterm, which is on content up to and including today's class.

Required Reading

Resources for doing fieldwork in a pandemic [check out methods including: remote video elicitation, diaries, online interviews]

Optional: Judge and Neustaedter, Studying and Designing Technology for Domestic Life, Morgan Kaufmann (2015). Chapters 9, 10

#9

Wed March 10  9:30 – 12:20

Choosing Evaluation Strategies

Greenberg & Buxton article: Usability evaluation considered harmful (some of the time).

Each team should come to class with a company and product you want to evaluate. Bring a document of up to one page that lists:

Your team members and email addresses.

The name of the organization that you plan to work with.

The name and email address of a contact person at the organization.

Details of how you have talked with them (remotely).

An initial idea of the type of evaluation you could conduct and on what.

In Class Exercise: Designing your evaluation: 5xWs

Evaluation Framework One Pager Template for in class exercise

Check in — Questions about exam

NEW DUE: Assignment 3: Affective Evaluation (15%) Due at the start of class. Email to TA Niloofar Kazemi [nka51@sfu.ca]. 

Required Reading

Greenberg S., and Buxton, B. Usability Evaluation Considered Harmful (Some of the time), Proceedings of the ACM Conference on Computer Human Interaction, ACM Press (2008) (PDF)

Watch this video of the paper's presentation: the video.

#10

Wed March 17 9:30 – 12:20

NEW DATE: Midterm Exam — during class time 9:30 – 11:30 PDT.  Note: As of March 14 it is now PDT in Vancouver.

Please write your name at the top of all pages on exam!!

Monday March 15 — We will return assignment 3.

After the exam, work with you team on completing the Evaluation Framework from last week.

Work with your team to do a draft of your team's Study Plan using template in week #11.

#11

Wed March 24 9:30 – 12:20

Design Evaluation: Workshopping and Refinement 

Share & feedback: Group Project complete Evaluation Framework (from week #9). Signoff unless further revisions required. If not signed off, revise again and email before midnight. 

Workshop: Group Project Study Plan. You will be split into groups of teams to present and receive feedback. It is highly advised to attend these presentations.

Study Plan Template for in class exercise. Prepare draft in advance of class. Revise and submit after class before Sunday midnight. 

 

 

Preparation

Come to class with complete Evaluation Framework (from week #9) and draft of Study Plan (Template).

After Class

If you did not get signoff in class from instructor on Evaluation Framework  — then after class today but before midnight email instructor and TA revised, final evaluation framework for one last round of feedback.

Revise and complete study plan. Email instructor and TA final study plan for signoff any time before Sunday midnight.

It is highly recommended you get sign off on both of these documents before you run your study. There are no marks assigned to this process. It is for your benefit. 

#12

Wed March 31 9:30 – 12:20

Study Plan/Initial Study Q&A 

Email instructor and TA in advance by Tuesday (March 30) midnight 2 questions/problems that have come up running your study. It’s normal to have challenges. We will work through these in class.

Google doc for Q&A

Required Reading

None

#13

Wed April 7 9:30 – 12:20

Mid-way Study Q&A 

Email instructor and TA in advance by Tuesday (April 6) midnight 2 questions/problems that have come up running your study, doing analysis or interpretation/recommendations and/or writing your report. It’s normal to have challenges. We will work through these in class again list last week. 

Google doc for Q&A

Assignment 4: Project (40%) Due by 5:00 pm, April 21 2021. Each group should email a single PDF containing all materials to TA and instructor.

Required Reading

None

#14

Wed April 14 9:30 – 12:20

Sign up for Team time w/ Instructor/TA

Sign up here .

Assignment 4: Project (40%) Due by 5:00 pm, April 21 2021. Each group should email a single PDF containing all materials to TA and instructor.

Required Reading

None