IAT 432 - Winter 2021
IAT 432 Design Evaluation is being taught online for the Summer 2020 & Winter 2021 sessions.
we learn this together. The following course schedule for IAT 432 is subject to change.
Technology: You will need a laptop or PC, a stable internet connection, a web browser that can run Zoom (Google Chrome works well, you do not need the Zoom application), Microsoft Office available to SFU students (Word and Excel), and the Slack application (download Win, Android or Mac). You will also need Flash for some in-class activities (check that pop-ups are not blocked). Let us know if you have concerns with the technology requirements.
If you do not have access to a laptop or PC or have adequate internet etc … , see SFU Financial support for student technology equipment (see Need emergency assistance for technology-related funding?)
#1 Winter 2021
Wed Jan 13 9:30 – 12:20
To join today’s class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted using the on/off buttons lower left of your Zoom screen.
If you cannot join Zoom, please contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888.
If you do not get the Zoom link in an email or cannot attend, please email TA Niloofar Kazemi [nka51@sfu.ca].
Course Logistics [about an 1 hour]
Zoom Whole Class
Check in Zoom — everyone on board and muted? practice raising hand, chat etc.
Check in technology accessibility and other COVID issues.
Brief review course and communication expectations.
Today’s Plan
Offline (15 minutes): Students go through Course Logistics [Slides]
Focus questions : Syllabus & Logistics — What are learning outcomes for course? Do you know what work are being assessed on (% each)? When are the assignments due? What is the group project about? When do I need to be on a team?
What grade scale is being used? What happens if your assignment is late? Do you have to attend class times? What are required texts?
Do you know how/when and what to expect re contacting your instructor/TA? What are key things you need to do to succeed in course?
Other — What is clear? What is not clear? What do you need more information about?
Zoom Breakout Rooms (10 minutes) — Breakout groups will discuss Focus Questions and report back to class.
Zoom Whole Class (10 minutes): Talk through Focus questions.
Zoom Whole Class: Go through Concepts and Evaluation Framework [Slides]
Zoom Breakout Rooms (10 minutes) Focus questions: Download these focus questions and work through the questions assigned to your breakout group — bring questions back to class.
Zoom Whole Class (10 minutes): Q&A + Homework — Use Slack Channel to answer focus questions and get feedback — one Q per thread.
Zoom Optional: Q&A and Instructor and TA Office Hours (N/A first week)
Today’s Learning Outcomes are to orient you to the course, go over course tools, set course expectations and communication guidelines, and give you key terms you will need to understand to succeed in the course.
Prepare for Class — Please Do These Things Before Class (estimated preparation time : 1 hour)
Read Introductory Course Email that you should have received about the course. Respond by email (“IAT 432" in subject line) to instructor and TA if you have any technology accessibility issues, and/or challenges around online learning and COVID-19 that may impact your ability to participate in this course (e.g. different time zone, child or elder care responsibility, you are ill, mental health issues). If you have formally diagnosed disabilities, please contact the Centre for Accessible Learning (CAL) to set up special accommodations.
Read Course Logistics Introduction — attached to introductory email you and here [Slides]. Bring questions to class.
Review: Syllabus including IAT 432 Learning Outcomes
Course Web Site: Please look over and familiarize yourself the web site including assignments and assessment. Bring questions to class for the Zoom session.
Please join: SLACK course channel. Please review netiquette resources.
Please review: SFU student conduct, what is academic integrity [video] and SFU’s student academic integrity policy.
#2
Wed Jan 20 9:30 – 12:20
To join today’s class — You will be emailed a Zoom link, which you can open in a browser. Please ensure your video and audio are muted.
If you cannot join Zoom, please contact IT Service Desk at it-servicedesk@sfu.ca or 778-782-8888.
If you do not get the Zoom link in an email or cannot attend, please email Niloofar Kazemi [nka51@sfu.ca].
Zoom Whole Class (15 minutes)
Check in Zoom — how was last class? questions?
Today’s Plan
Course Content — IAT 201/334 Review (see Preparation (right column) slides/audio and readings)
Questions from Slides and Audio Content?
Activity: Zoom Breakout Groups
Meet your working group (mini-icebreaker).
Work through Review heuristic evaluation
Zoom Whole Class: Check in Heuristic Eval Activity
Introduction to Assignment 1: Ethics (5%) [allow 2-3 hours outside of class for this assignment]
Learning Outcomes for today are to review and summarize key materials from IAT 334 focusing on heuristic evaluations that you will need to use in the course and/or will be on the exam, and meet your small working group.
Prepare for Class — Please Do These Things Before Class (estimated preparation time 2 hours)
Content: Go through Slides and Audio for these Review topics
Heuristic Evaluation (Slides, Audio) [1:06 ]
Readings: As we go through the course and learn about each evaluation method, you should progressively scan, skim then read through Chapters 1-12 of this Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests” (2008) by Jeffrey Rubin, Dana Chisnell, Jared Spool (listed under Resources). The basic framework introduced in the first class, is expanded on in these chapters (what, when, who etc to evaluate?) as well as data collection, analysis and reporting information.
#3
Wed Jan 27 9:30 – 12:20
Zoom Whole Class: Check in / this week is pretty easy!
Course Content: walk through usability evaluation materials
Zoom Breakout Activity: Work through Review usability studies
Zoom Whole class: Share Usability Activity
Learning Outcomes for today are to review and summarize key materials from IAT 334 focusing on usability evaluations that you will need to use in the course and/or will be on the exam, and meet your small working group.
DUE: Assignment 1: Ethics (5%) Due at the start of class. Email ethics certificate to TA Niloofar Kazemi [nka51@sfu.ca].
Prepare for Class — Please Do These Things Before Class (estimated preparation time 2 hours)
Content: Go through Slides and Audio for this Review topic:
Usability Testing Plans and Methods (Slides, Audio) [43:44]
Readings: As we go through the course and learn about each evaluation method, you should progressively scan, skim then read through Chapters 1-12 of this Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests” (2008) by Jeffrey Rubin, Dana Chisnell, Jared Spool (listed under Resources). The basic framework introduced in the first class, is expanded on in these chapters (what, when, who etc to evaluate?) as well as data collection, analysis and reporting information.
#4
Wed Feb 3 9:30 – 12:20
Zoom Whole Class – this week is intense!
Add feedback on course so far and/or questions on assignment 2 to SLACK channel in assignment questions.
Course Content — Controlled Experiments — Topics:
- Scientific Method
- Hypothesis testing
- Independent & dependent variables
- Ethics of controlled experiments
- Reliability and validity
- Between and within experimental designs
Zoom Lab (Breakout) Activity 2 (20 min): Work through controlled experimental study design with your working group using one interface set (will be assigned in class) — report back.
Exercise Sheets
Interface Set 2 – Track pad vs. mouse
Interface Set 3 – Ribbon menu vs. traditional menu
Zoom Lab (Breakout) Activity 1 (10): Work through Focus Questions on this weeks topics based on slides/audio/readings. Report back/questions.
Break
Zoom Lab (Breakout) Activity 3 (60 min): Work through with your working group Design an experiment for AimBooster – report back.
Learning outcomes or today are to understand and be able to apply the basic components of a controlled experiment in order to design a study to evaluate different interfaces and to assess a study design against criteria for reliability and validity.
Prepare for Class — Please Do These Things Before Class (estimated preparation time 5-6 hours)
Content: Go through Slides and Audio for Controlled Experiments (Slides, Audio) [18:00]
Content: Go through Slides and Audio for Reliability and Validity (Slides, Audio 31:38 minutes)
Required Reading
Doing Psychology Experiments, Chapter 2: How to Do Experiments
Doing Psychology Experiments, Chapter 7: How to Decide Which Variables to Manipulate and Measure
Doing Psychology Experiments, Chapter 8: How to Decide on a Between-Subjects versus Within-Subject Design
#5
Wed Feb 10 9:30 – 12:20
Zoom Whole Class: Walk through Assignment 2: Controlled Experiment (15%) [individual mark]
Course Content — Statistics for Controlled Experiments
Graphing frequency distributions
Descriptive statistics: mean, standard deviation, median, range
Graphing mean with error bars
Basic Inferential tests: t-test
Demo Excel (large/small mouse)
Questions from audio/slides/readings?
Zoom Lab (Breakout) Activity: Work through exercise and check answers with your working group — ask for help as needed. Hand in exercise — individually.
Exercise on statistics (compare two keypad designs, two data sets)
Note: Error in ans key: study design is within design so t-test parameter for type should always be 1 (paired) — not 3.
Answers for the exercise on statistics (try to do the work first, then check!)
Learning outcomes for today are to understand and be able to run basic statistical techniques on data gathered in different forms of controlled experiments; interpret results in context of evaluation goals and design iteration; and understand the limitations of quantitative results.
Review: Assignment 2: Controlled Experiment (15%) [individual mark]
Prepare for Class — Please Do These Things Before Class (estimated preparation time 4 hours)
Content: Go through Slides and Audio for Statistics (Slides, Audio 33 minutes)
For exercises, you will need Excel to have the Data Analysis Toolpak installed (for older versions of Excel) or the Solver analysis package installed (for newer versions of Excel). Instructions for how to get it are here for Mac or PC.
Required Reading
Doing Psychology Experiments, Chapter 12: How to Interpret Experimental Results
Doing Psychology Experiments, Chapter 13: How to Report Experimental Results
#6 No Class
Wed Feb 17
Reading Week
DUE: Assignment 2: Controlled Study (15%) Due today. Email to TA Niloofar Kazemi [nka51@sfu.ca].
Catch up on readings.
Work on preparation for next week/assignment #3, mid-term exam practice questions, and setting up final projects — you should have a client lined up and know what your evaluation goals are.
#7
Wed Feb 24 9:30 – 12:20
Evaluating Emotion and Affect
Measuring affect and emotion
Theories of emotion
Quantifying qualitative codes
Cued Debrief Recall
Studio Lab Activity (2.5 hours in classtime). See activity (below) — choose participant, online game and practice video recording set up ahead of time. Work out how you will share screens/data.
Conduct an example study of gameplay and collect IMI questionnaire data and cued debrief recall data
Learning outcomes for today are to understand what affect and emotion are, learn how to measure each understanding the limitations of different approaches, understand the basics of questionnaire and interview methods, learn to code and assess reliability of qualitative video data and then quantify that data, understand and be able to apply the cued recall method working remotely.
Prepare for Class — Please Do These Things Before Class (estimated preparation time 4.5 hours)
Content: Go through Slides and Audio for Evaluating Emotion and examples (Slides, Audio 30 minutes)
Pac Man game as an example cued debrief recall example
Inter-rater reliability example
Content: Go through slides and audio for Interviews, Surveys, Questionnaires including retrospective methods for assignment 3 (Slides Audio) [36:40] and for Qualitative Data Analysis (Slides, Audio) [26:14]
Review: Introduction to Assignment 3: Affective Evaluation (15%) [individual mark]
Required Reading
Bentley, T., Johnston, L., and Baggo, K. Evaluation using cued-recall debrief to elicit information about a user’s affective experiences, Proc. OzCHI, ACM Press (2005). (PDF)
#8
Wed March 3 9:30 – 12:20
Studying Field Deployments (see audio/slides)
Planning and time-frames
Managing home visits and remote “visits"
Minimum viable prototypes
Ethics for field evaluations, ethics of collecting data remotely
Remote methods for fieldwork (see reading)
Assignment #3 check in
Studio Lab Activity – Practice questions for exam
Learning outcomes for today are to understand the main considerations in doing field evaluations, and how to do them remotely.
Prepare for Class — Please Do These Things Before Class (estimated preparation time 3 hours)
Content: Go through Slides and Audio for Field Evaluations Slides, Audio (30 minutes)
You should be working on assignment 3 and studying for your midterm, which is on content up to and including today’s class.
Required Reading
Resources for doing fieldwork in a pandemic [check out methods including: remote video elicitation, diaries, online interviews]
Optional: Judge and Neustaedter, Studying and Designing Technology for Domestic Life, Morgan Kaufmann (2015). Chapters 9, 10
#9
Wed March 10 9:30 – 12:20
Choosing Evaluation Strategies
Greenberg & Buxton article: Usability evaluation considered harmful (some of the time).
Each team should come to class with a company and product you want to evaluate. Bring a document of up to one page that lists:
Your team members and email addresses.
The name of the organization that you plan to work with.
The name and email address of a contact person at the organization.
Details of how you have talked with them (remotely).
An initial idea of the type of evaluation you could conduct and on what.
In Class Exercise: Designing your evaluation: 5xWs
Evaluation Framework One Pager Template for in class exercise
Check in — Questions about exam
NEW DUE: Assignment 3: Affective Evaluation (15%) Due at the start of class. Email to TA Niloofar Kazemi [nka51@sfu.ca].
Required Reading
Greenberg S., and Buxton, B. Usability Evaluation Considered Harmful (Some of the time), Proceedings of the ACM Conference on Computer Human Interaction, ACM Press (2008) (PDF)
#10
Wed March 17 9:30 – 12:20
NEW DATE: Midterm Exam — during class time 9:30 – 11:30 PDT. Note: As of March 14 it is now PDT in Vancouver.
Please write your name at the top of all pages on exam!!
Monday March 15 — We will return assignment 3.
After the exam, work with you team on completing the Evaluation Framework from last week.
Work with your team to do a draft of your team’s Study Plan using template in week #11.
#11
Wed March 24 9:30 – 12:20
Design Evaluation: Workshopping and Refinement
Share & feedback: Group Project complete Evaluation Framework (from week #9). Signoff unless further revisions required. If not signed off, revise again and email before midnight.
Workshop: Group Project Study Plan. You will be split into groups of teams to present and receive feedback. It is highly advised to attend these presentations.
Study Plan Template for in class exercise. Prepare draft in advance of class. Revise and submit after class before Sunday midnight.
Preparation
Come to class with complete Evaluation Framework (from week #9) and draft of Study Plan (Template).
After Class
If you did not get signoff in class from instructor on Evaluation Framework — then after class today but before midnight email instructor and TA revised, final evaluation framework for one last round of feedback.
Revise and complete study plan. Email instructor and TA final study plan for signoff any time before Sunday midnight.
It is highly recommended you get sign off on both of these documents before you run your study. There are no marks assigned to this process. It is for your benefit.
#12
Wed March 31 9:30 – 12:20
Study Plan/Initial Study Q&A
Email instructor and TA in advance by Tuesday (March 30) midnight 2 questions/problems that have come up running your study. It’s normal to have challenges. We will work through these in class.
Required Reading
None
#13
Wed April 7 9:30 – 12:20
Mid-way Study Q&A
Email instructor and TA in advance by Tuesday (April 6) midnight 2 questions/problems that have come up running your study, doing analysis or interpretation/recommendations and/or writing your report. It’s normal to have challenges. We will work through these in class again list last week.
Assignment 4: Project (40%) Due by 5:00 pm, April 21 2021. Each group should email a single PDF containing all materials to TA and instructor.
Required Reading
None
#14
Wed April 14 9:30 – 12:20
Sign up for Team time w/ Instructor/TA
Sign up here .
Assignment 4: Project (40%) Due by 5:00 pm, April 21 2021. Each group should email a single PDF containing all materials to TA and instructor.
Required Reading
None