IAT 432 – Timeline (Course Calendar)

IAT 432 - Summer 2020

IAT 432 Design Evaluation is being taught online for the Summer session.

This is an upper division undergraduate course taught in the School of Interactive Art and Technology at Simon Fraser University in Surrey, BC, Canada. It explores the evaluation of user interfaces using a variety of design evaluation methods from the fields of human-computer interaction and interaction design.
Note: This is not an “online course”. It is a course being taught online. We will need to have flexibility and patience with another as
we learn this together. The following course schedule for IAT 432 is subject to change.
Instructor: Alissa Antle [aantle@sfu.ca]
TA: Niloofar Kazemi [nka51@sfu.ca]

Technology: You will need a laptop or PC, a stable internet connection, a web browser that can run Zoom (Google Chrome works well, you do not need the Zoom application), Microsoft Office available to SFU students  (Word and Excel), and the Slack application (download Win, Android or Mac). You will also need Flash for some in-class activities (check that pop-ups are not blocked). Let us know if you have concerns with the technology requirements. 

If you do not have access to a laptop or PC or have adequate internet etc … , see SFU Financial support for student technology equipment (see Need emergency assistance for technology-related funding?) 

Week 1 — Class 1 

Tues May 12, 2020

2:30 PM – 5:20 PM

To join today's class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted using the on/off buttons lower left of your Zoom screen. 

If you cannot join Zoom, please  contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888. 

If you do not get the Zoom link in an email or cannot attend, please email TA Niloofar Kazemi [nka51@sfu.ca]. 

Course Logistics [about an 1 hour] 

Zoom Whole Class

Check in Zoom — everyone on board and muted? practice raising hand, chat etc. 

Check in technology accessibility and other COVID issues.

Brief review course and communication expectations.

Today's Plan

Offline (15 minutes): Students go through Course Logistics [Slides]

Focus questions : What is clear? What is not clear? What do you need more information about?

Zoom Breakout Rooms (10 minutes) — Breakout groups will discuss Focus Questions and report back to class.

Zoom Whole Class (10 minutes): Talk through Focus questions.  

Break

Course Content — Key Concepts for Design Evaluation [about an 1 hour]

Zoom Whole Class:  Go through Concepts and Evaluation Framework [Slides]

Zoom Breakout Rooms (20 minutes): Focus questions: Download these focus questions and work through questions assigned to your breakout group — be prepared to discuss with the class.

Zoom Whole Class (20 minutes): Report back and talk through Focus questions.


Zoom Optional: Q&A and Instructor and TA Office Hours

Today's Learning Outcomes are to orient you to the course, go over course tools, set course expectations and communication guidelines, and give you key terms you will need to understand to succeed in  the course. 

Due Wed May 13, before 5 pm  PDT — You will work in small groups of 4 people in the course and for the project.  Have one team member email TA with your preferred small working group of 4 people and CC your team members. If you do not do this, you will be assigned to a working group. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time : 1 hour)

Read Introductory Course Email that you should have received about the course. Respond by email (“IAT 432" in subject line) to instructor and TA if you have any technology accessibility issues, and/or challenges around online learning and COVID-19 that may impact your ability to participate in this course (e.g. different time zone, child or elder care responsibility, you are ill, mental health issues). If you have formally diagnosed disabilities, please contact the Centre for Accessible Learning (CAL) to set up special accommodations.  

Read Course Logistics Introduction — attached to introductory email you and here [Slides]. Bring questions to class.

Review: Syllabus including  IAT 432 Learning Outcomes

Course Web Site: Please look over and familiarize yourself the web site including assignments and assessment. Bring questions to class for the 2:30 Zoom session.

Please join: SLACK course channel.  Please review netiquette resources.

Please review: SFU student conduct, what is academic integrity [video] and SFU's student academic integrity policy.

 

Week 1 — Class 2

Thursday May 14, 2020

2:30 PM – 5:20 PM

To join today's class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted. 

If you cannot join Zoom, please  contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888. 

If you do not get the Zoom link in an email or cannot attend, please email Niloofar Kazemi [nka51@sfu.ca]. 

Course Logistics  

Zoom Whole Class (15 minutes)

Check in Zoom — how was last class? questions? 

Today's Plan


Course Content —  IAT 201/334 Review (see Preparation (right column) slides/audio and readings) (2 hours) 

Questions from Slides and Audio Content?

Activities: Review Methods from IAT 334

Zoom Breakout Groups 

Meet your working group (mini-icebreaker).

Work through Review heuristic evaluation

Break

Work through Review usability studies

Zoom Whole Class

Introduction to Assignment 1: Ethics (5%) [allow 2-3 hours outside of class for this assignment]


Learning Outcomes for today are to review and summarize key materials from IAT 334 that you will need to use in the course and/or will be on the exam, and meet your small working group.

Prepare for Class — Please Do These Things Before Class  (estimated preparation time 2 hours) 

Content: Go through Slides and Audio for these Review topics

Heuristic Evaluation (Slides, Audio) [1:06 ]

Usability Testing Plans and Methods (Slides Audio) [43:44]

Readings: As we go through the course and learn about each evaluation method, you should progressively scan, skim then read through Chapters 1-12 of this Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests” (2008) by Jeffrey Rubin, Dana Chisnell, Jared Spool (listed under Resources). The basic framework introduced in the first class, is expanded on in these chapters (what, when, who etc to evaluate?) as well as data collection, analysis and reporting information.  

 

Week 2 – Class 1 

Tues May 19, 2020

2:30 PM – 5:20 PM

Zoom Whole Class 

Check in Zoom — how was last week? questions? 

Today's Plan


Course Content — Controlled Experiments — Topics: 

  • Scientific Method
  • Hypothesis testing
  • Independent & dependent variables
  • Ethics of controlled experiments

Questions from audio/slides/readings?

Zoom Lab (Breakout) Activity:  

Work through Focus Questions on this weeks topics based on slides/audio/readings. Report back/questions.

Zoom Whole Class: Walk through Assignment 2: Controlled Experiment (15%) [individual mark]

Break

Zoom Lab (Breakout) Activity: Work through controlled experimental study design with your working group using one interface set (will be assigned in class) — report back.

Exercise Sheets

Interface Set 1 Mac Tray

Interface Set 2 – Track pad vs. mouse

Interface Set 3 – Ribbon menu vs. traditional menu

Interface Set 4 – Wii remotes

Learning outcomes  for today are to understand and be able to apply the basic components of a controlled experiment in order to design a study to evaluate different interfaces. 

DUE: Assignment 1: Ethics (5%) Due at the start of class (2:30 PDT Tues, May 19). Email ethics certificate to TA Niloofar Kazemi [nka51@sfu.ca]. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time 4 hours)

Content: Go through Slides and Audio for Controlled Experiments (Slides, Audio) [18:00]

Review: Assignment 2: Controlled Experiment (15%) [individual mark]

Review: Breakout activity on controlled experiment design.

Required Reading

Doing Psychology Experiments, Chapter 2: How to Do Experiments

Doing Psychology Experiments, Chapter 7: How to Decide Which Variables to Manipulate and Measure

Week 2 – Class 2 

Thursday May 21, 2020

2:30 PM – 5:20 PM

Zoom Whole Class 

Check in Zoom — how is it going at end of week 2 – 1/3 through course? Add feedback on course so far and/or questions on assignment 2 to google doc

Today's Plan


Course Content — Reliability and Validity

Reliability, consistency, repeatability

Confounding variables

Face validity, construct validity, internal validity, external validity

Between and Within Design (see reading)

Questions from audio/slides/readings?

Zoom Lab (Breakout) Activity:  Work through with your working group Design an experiment for AimBooster – report back.

Learning outcomes for today are to advance your understanding and ability to apply to design a study to evaluate different interfaces and ability to assess a study design against criteria for reliability and validity.  

You should be working on assignment 2. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time 2 hours) 

Content: Go through Slides and Audio for Reliability and Validity (SlidesAudio 31:38 minutes)

Required Reading

Doing Psychology Experiments, Chapter 8: How to Decide on a Between-Subjects versus Within-Subject Design

 

Week 3 – Class 1 

Tues May 26, 2020

2:30 PM – 5:20 PM

Zoom Whole Class (10 minutes)

Check in Zoom. 

Today's Plan


Course Content — Statistics for Controlled Experiments

Graphing frequency distributions

Descriptive statistics: mean, standard deviation, median, range

Graphing mean with error bars

Basic Inferential tests: t-test

Demo Excel (large/small mouse)

Questions from audio/slides/readings?

Zoom Lab (Breakout) Activity:  Work through exercise and check answers with your working group — ask for help as needed. Hand in exercise — individually. 

Exercise on statistics (compare two keypad designs, two data sets)

Note: Error in ans key: study design is within design so t-test parameter for type should always be 1 (paired) — not 3.

Excel Commands Cheat Sheet

Answers for the exercise on statistics (try to do the work first, then check!)

Learning outcomes for today are to understand and be able to run basic statistical techniques on data gathered in different forms of controlled experiments; interpret results in context of evaluation goals and design iteration; and understand the limitations of quantitative results.

You should be working on assignment 2.

Prepare for Class — Please Do These Things Before Class (estimated preparation time 4 hours) 

Content: Go through Slides and Audio for Statistics (SlidesAudio 33 minutes)

For exercises, you will need Excel to have the Data Analysis Toolpak installed (for older versions of Excel) or the Solver analysis package installed (for newer versions of Excel). Instructions for how to get it are here for Mac or PC.

Required Reading

Doing Psychology Experiments, Chapter 12: How to Interpret Experimental Results

Doing Psychology Experiments, Chapter 13: How to Report Experimental Results 

Week 3 – Class 2 

Thursday May 28, 2020

2:30 PM – 5:20 PM

Evaluating Emotion and Affect

Measuring affect and emotion

Theories of emotion

Quantifying qualitative codes

Cued Debrief Recall

Studio Lab Activity (2.5 hours in classtime). See activity (below)  — choose participant, online game and practice video recording set up ahead of time.  Work out how you will share screens/data. 

Conduct an example study of gameplay and collect IMI questionnaire data and cued debrief recall data

 Learning outcomes for today are to understand what affect and emotion are, learn how to measure each understanding the limitations of different approaches, understand the basics of questionnaire and interview methods, learn to code and assess reliability of qualitative video data and then quantify that data, understand and be able to apply the cued recall method working remotely. 

DUE: Assignment 2: Controlled Study (15%) Due at the start of class. Email to TA Niloofar Kazemi [nka51@sfu.ca]. 

Prepare for Class — Please Do These Things Before Class (estimated preparation time 4.5 hours) 

Content: Go through Slides and Audio for Evaluating Emotion and examples  (Slides, Audio 30 minutes)

Pac Man game as an example cued debrief recall example

Inter-rater reliability example

Content: Go through slides and audio for Interviews, Surveys, Questionnaires including retrospective methods for assignment 3 (Slides  Audio) [36:40]  and for Qualitative Data Analysis (Slides Audio) [26:14]

Review: Introduction to Assignment 3: Affective Evaluation (15%) [individual mark]

Required Reading

Bentley, T., Johnston, L., and Baggo, K. Evaluation using cued-recall debrief to elicit information about a user's affective experiences, Proc. OzCHI, ACM Press (2005). (PDF)

Week 4 – Class 1 

Tues June 2, 2020

2:30 PM – 5:20 PM

Studying Field Deployments (see audio/slides)

Planning and time-frames

Pilot testing

Managing home visits and remote “visits"

Minimum viable prototypes

Group vs. individual interviews, doing interviews remotely

Data validity

Ethics for field evaluations, ethics of collecting data remotely

Remote methods for fieldwork (see reading)

Assignment #3 check in 

Studio Lab Activity – Practice questions for exam

Learning outcomes for today are to understand the main considerations in doing field evaluations, and how to do them remotely.

Prepare for Class — Please Do These Things Before Class (estimated preparation time 3 hours)

Content: Go through Slides and Audio for Field Evaluations SlidesAudio (30 minutes)

You should be working on assignment 3 and studying for your midterm, which is on content up to and including today's class.

Required Reading

Resources for doing fieldwork in a pandemic [check out methods including: remote video elicitation, diaries, online interviews]

Optional: Judge and Neustaedter, Studying and Designing Technology for Domestic Life, Morgan Kaufmann (2015). Chapters 9, 10

Week 4 – Class 2 

Thursday June 4, 2020

2:30 PM – 5:20 PM

Midterm Exam

Week 5 – Class 1

Tues June 9, 2020

2:30 PM – 5:20 PM

Choosing Evaluation Strategies

Greenberg & Buxton article: Usability evaluation considered harmful (some of the time).

Each team should come to class with a company and product you want to evaluate. Bring a document of up to one page that lists:

Your team members and email addresses.

The name of the organization that you plan to work with.

The name and email address of a contact person at the organization.

Details of how you have talked with them (remotely).

An initial idea of the type of evaluation you could conduct and on what.

DUE: Assignment 3: Affective Evaluation (15%) Due at the start of class. Email to TA Niloofar Kazemi [nka51@sfu.ca]. Required Reading

Greenberg S., and Buxton, B. Usability Evaluation Considered Harmful (Some of the time), Proceedings of the ACM Conference on Computer Human Interaction, ACM Press (2008) (PDF)

Watch this video of the paper's presentation: the video.

Week 5 – Class 2

Thursday June 11, 2020

2:30 PM – 5:20 PM

Design Evaluation Presentations and Refinement

Come to class and be prepared to preset an plan for the study you want to run. You will be split into groups of teams to present and receive feedback. It is mandatory to attend these presentations.


Update on bonus marks, midterm marking and assignment 3 marking. 

 

Preparation

Before class each team should email instructor and TA a complete study plan. Be prepared to present this in larger groups to refine your plan.

 That is, you should send us a complete set of draft study materials in a single PDF. This means that you should create an initial plan that includes details of:

Evaluation goals

Details of your participants (who, how many) and setting for study (where)

Method you will use and why (relate to evaluation goals and summative or formative nature of your evaluation and what you hope to find out)

Procedure (e.g. tasks, instructions, data collection points etc)

Dimensions of UX/Usability you will measure (e.g. details of all questionnaires/interviews/tests and how they are administered)

Data collection: what kinds of data (data type, qual/quant, subj/obj) you will collect and why

Data analysis: how you will analyze your data

Week 6 – Class 1 

Tues June 16, 2020

2:30 PM – 5:20 PM

Initial Report of Results

Email instructor and TA in advance by noon today — 1-3 questions/problems that have come up running your study. It’s normal to have challenges. We will work through these in class. 

Course Evaluation.

Return assignment 3. 

Required Reading

None

Week 6 – Class 2

Thursday June 18, 2020

2:30 PM – 5:20 PM

Final Reports of Results

Come to class and be prepared to do a informal talk about your final study design and results. Use slides. Informal. 7 minutes per team. 

 Assignment 4: Project (40%) Due by 5:00 pm, June 23. Each group should email a single PDF containing all materials to TA and instructor.

Required Reading

None