PhonoBlocks – TECI – Alissa N Antle

Project Overview

New technologies, such as tangibles and touch tablets with haptic feedback, may provide solutions for helping dyslexic children learn to read. Tangible letters that can be encoded with graphical or haptic information may enable dyslexic children to better decode them. For example, color coding, used by syntheses, may enable reliable 2D decoding. Haptic feedback can be used to encode letters with signature movements. Tablets combined with augmented reality and physical letters can also enable multimodal interaction and digital feedback. In this project we are exploring these strategies and others. The main research outcome is to determine if any of these strategies improve reading outcomes for dyslexic children, and how to scale and deploy solutions broadly to make them accessible to children at home and in schools.

Team

Dr. Alissa N. Antle: Project Lead
Dr. Maureen Hoskyn (previous advisor): Director, Centre for Research on Early Child Health and Education, Faculty of Education, Simon Fraser University.
Dr. Alyssa Wise (previous advisor): Professor of Learning Sciences and Educational Technology, Steinhardt School of Culture, Education, and Human Development, New York University.

Graduate Students

Min Fan, Post-Doctoral Fellow (now Assistant Professor, School of Animation and Digital Arts, Communication University of China)
Victor Cheung, Post-Doctoral Fellow (now Lecturer, Computer Science, SFU)
Shubhra Sarker, MSc. (now software developer, Fortinet)
Boxiao Gong, PhD student
Emily Cramer, MSc. (now developer, Google)
Ying Deng, MSc. (now user experience, Copperleaf)
Jianyu Fan, Ph.D. Candidate

Funded By

SSHRC Insight
GRAND NCE
McDonnell Foundation

Publications

2020
Fan, Min, Antle, Alissa N.

An English Language Learning Study with Rural Chinese Children Using an Augmented Reality App Proceedings Article

In: Proceedings of the Interaction Design and Children Conference, pp. 385–397, Association for Computing Machinery, London, United Kingdom, 2020, ISBN: 9781450379816.

Abstract | Links | BibTeX

@inproceedings{10.1145/3392063.3394409,
title = {An English Language Learning Study with Rural Chinese Children Using an Augmented Reality App},
author = {Min Fan and Alissa N. Antle},
url = {https://doi.org/10.1145/3392063.3394409},
doi = {10.1145/3392063.3394409},
isbn = {9781450379816},
year = {2020},
date = {2020-01-01},
urldate = {2020-01-01},
booktitle = {Proceedings of the Interaction Design and Children Conference},
pages = {385–397},
publisher = {Association for Computing Machinery},
address = {London, United Kingdom},
series = {IDC '20},
abstract = {Augmented reality (AR) apps have the potential to support early English learning for children. However, few studies have investigated how children from rural low socio-economic status (SES) schools, who learn English as a foreign language (EFL) used and perceived an AR app in language learning. In this paper, we present an exploratory case study of 11 EFL children and four school teachers from a Chinese rural county who used an AR app (called AR PhonoBlocks), for one week. The goal of the app is to support children to learn the alphabetic principle of English. The key features are overlaid dynamic colour cues on 3D physical letters. We present the results including themes related to children's interactional behaviours and motivations, and rural teachers' feedback on the opportunities and concerns around using an AR app in a rural school context. We suggest design implications and future research directions for designing AR apps to support EFL children from low SES schools in early English learning.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}

Close

Augmented reality (AR) apps have the potential to support early English learning for children. However, few studies have investigated how children from rural low socio-economic status (SES) schools, who learn English as a foreign language (EFL) used and perceived an AR app in language learning. In this paper, we present an exploratory case study of 11 EFL children and four school teachers from a Chinese rural county who used an AR app (called AR PhonoBlocks), for one week. The goal of the app is to support children to learn the alphabetic principle of English. The key features are overlaid dynamic colour cues on 3D physical letters. We present the results including themes related to children's interactional behaviours and motivations, and rural teachers' feedback on the opportunities and concerns around using an AR app in a rural school context. We suggest design implications and future research directions for designing AR apps to support EFL children from low SES schools in early English learning.

Close

https://doi.org/10.1145/3392063.3394409

doi:10.1145/3392063.3394409

Close
Cheung, Victor, Antle, Alissa N., Sarker, Shubhra, Fan, Min, Fan, Jianyu, Pasquier, Philippe

Techniques for Augmented-Tangibles on Mobile Devices for Early Childhood Learning Proceedings Article

In: Proceedings of the Interaction Design and Children Conference, pp. 589–601, Association for Computing Machinery, London, United Kingdom, 2020, ISBN: 9781450379816.

Abstract | Links | BibTeX

@inproceedings{10.1145/3392063.3394412,
title = {Techniques for Augmented-Tangibles on Mobile Devices for Early Childhood Learning},
author = {Victor Cheung and Alissa N. Antle and Shubhra Sarker and Min Fan and Jianyu Fan and Philippe Pasquier},
url = {https://dl.acm.org/doi/abs/10.1145/3392063.3394412},
doi = {10.1145/3392063.3394412},
isbn = {9781450379816},
year = {2020},
date = {2020-01-01},
urldate = {2020-01-01},
booktitle = {Proceedings of the Interaction Design and Children Conference},
pages = {589–601},
publisher = {Association for Computing Machinery},
address = {London, United Kingdom},
series = {IDC '20},
abstract = {Integrating physical learning materials with mobile device applications may have benefits for early childhood learning. We present three techniques for creating a hybrid tangible-augmented reality (T-AR) enabling technology platform. This platform enables researchers to develop applications that use readily available physical learning materials, such as letters, numbers, symbols or shapes. The techniques are visual marker-based; computer-vision and machine-learning; and capacitive touches. We describe details of implementation and demonstrate these techniques through a use case of a reading tablet app that uses wooden/plastic letters for input and augmented output. Our comparative analysis revealed that the machine-learning technique most flexibly sensed different physical letter sets but had variable accuracy impacted by lighting and tracking lag at this time. Lastly, we demonstrate how this enabling technology can be generalized to a variety of early learning apps through a second use case with physical numbers.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}

Close

Integrating physical learning materials with mobile device applications may have benefits for early childhood learning. We present three techniques for creating a hybrid tangible-augmented reality (T-AR) enabling technology platform. This platform enables researchers to develop applications that use readily available physical learning materials, such as letters, numbers, symbols or shapes. The techniques are visual marker-based; computer-vision and machine-learning; and capacitive touches. We describe details of implementation and demonstrate these techniques through a use case of a reading tablet app that uses wooden/plastic letters for input and augmented output. Our comparative analysis revealed that the machine-learning technique most flexibly sensed different physical letter sets but had variable accuracy impacted by lighting and tracking lag at this time. Lastly, we demonstrate how this enabling technology can be generalized to a variety of early learning apps through a second use case with physical numbers.

Close

https://dl.acm.org/doi/abs/10.1145/3392063.3394412

doi:10.1145/3392063.3394412

Close
Fan, Min, Antle, Alissa N., Warren, Jillian L.

Augmented Reality for Early Language Learning: A Systematic Review of Augmented Reality Application Design, Instructional Strategies, and Evaluation Outcomes Journal Article

In: Journal of Educational Computing Research, vol. 58, no. 6, pp. 1059–1100, 2020.

Abstract | Links | BibTeX

@article{doi:10.1177/0735633120927489,
title = {Augmented Reality for Early Language Learning: A Systematic Review of Augmented Reality Application Design, Instructional Strategies, and Evaluation Outcomes},
author = {Min Fan and Alissa N. Antle and Jillian L. Warren},
url = {https://doi.org/10.1177/0735633120927489},
doi = {10.1177/0735633120927489},
year = {2020},
date = {2020-01-01},
urldate = {2020-01-01},
journal = {Journal of Educational Computing Research},
volume = {58},
number = {6},
pages = {1059--1100},
abstract = {In this article, we present a systematic review of literature on augmented reality (AR) supported for early language learning. We analyzed a total of 53 papers from 2010 to 2019 using qualitative analysis with complementary descriptive quantitative analysis. Our findings revealed three main AR learning activities: word spelling games, word knowledge activities, and location-based word activities. Our findings also uncovered five main design strategies: three-dimensional multimedia content, hands-on interaction with physical learning materials, gamification, spatial mappings, and location-based features. Several combinations of design and instructional strategies tended to be effective: Learning gains were enhanced by using three-dimensional multimedia with advanced organizers (presentation strategy) and/or using location-based content with learners’ self-exploration (discovery strategy); and motivation was enhanced by using game mechanisms with discovery strategy. We suggest that future designers of AR early language applications should move beyond these basic approaches and consider how unique benefits of AR may be applied to support key activities in early language learning while also considering how to support sociotechnical factors such as collaboration between teachers and learners and different learning contexts. We conclude with a discussion of future directions for research in this emerging space.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}

Close

In this article, we present a systematic review of literature on augmented reality (AR) supported for early language learning. We analyzed a total of 53 papers from 2010 to 2019 using qualitative analysis with complementary descriptive quantitative analysis. Our findings revealed three main AR learning activities: word spelling games, word knowledge activities, and location-based word activities. Our findings also uncovered five main design strategies: three-dimensional multimedia content, hands-on interaction with physical learning materials, gamification, spatial mappings, and location-based features. Several combinations of design and instructional strategies tended to be effective: Learning gains were enhanced by using three-dimensional multimedia with advanced organizers (presentation strategy) and/or using location-based content with learners’ self-exploration (discovery strategy); and motivation was enhanced by using game mechanisms with discovery strategy. We suggest that future designers of AR early language applications should move beyond these basic approaches and consider how unique benefits of AR may be applied to support key activities in early language learning while also considering how to support sociotechnical factors such as collaboration between teachers and learners and different learning contexts. We conclude with a discussion of future directions for research in this emerging space.

Close

https://doi.org/10.1177/0735633120927489

doi:10.1177/0735633120927489

Close
2018
Fan, Min, Antle, Alissa N., Hoskyn, Maureen, Neustaedter, Carman

A design case study of a tangible system supporting young English language learners Journal Article

In: International Journal of Child-Computer Interaction, vol. 18, pp. 67–78, 2018, ISSN: 2212-8689.

Abstract | Links | BibTeX

@article{FAN201867,
title = {A design case study of a tangible system supporting young English language learners},
author = {Min Fan and Alissa N. Antle and Maureen Hoskyn and Carman Neustaedter},
url = {https://www.sciencedirect.com/science/article/pii/S2212868917300867},
doi = {https://doi.org/10.1016/j.ijcci.2018.08.001},
issn = {2212-8689},
year = {2018},
date = {2018-01-01},
urldate = {2018-01-01},
journal = {International Journal of Child-Computer Interaction},
volume = {18},
pages = {67--78},
abstract = {Many researchers have suggested that tangible user interfaces (TUIs) have the potential to support learning for children. While several tangible reading systems have been developed for children, few systems have been designed that explicitly target the first stage of reading where many children struggle, which is the alphabetic principle (letter-sound correspondences). We present a tangible reading system called PhonoBlocks that supports children learning English letter-sound correspondences. PhonoBlocks uses 3D tangible letters that change colour to draw attention to the moment that adding other letters changes the sounds. We then present a mixed-methods case study with ten Mandarin-speaking children in China using our system. Results showed that the Chinese children achieved significant learning gains relative to their baseline performance after PhonoBlocks instruction. The results also point to design features of our system that enabled behaviours that are correlated with learning. We compare the results of this study to a different study with eight at-risk monolingual English-speaking children in Canada using PhonoBlocks in learning to read and spell. By comparing results, we generalize and make three recommendations for designing tangible reading systems for all children who must learn the alphabetic principle. We also discuss three recommendations that are specifically for children learning English as a foreign language.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}

Close

Many researchers have suggested that tangible user interfaces (TUIs) have the potential to support learning for children. While several tangible reading systems have been developed for children, few systems have been designed that explicitly target the first stage of reading where many children struggle, which is the alphabetic principle (letter-sound correspondences). We present a tangible reading system called PhonoBlocks that supports children learning English letter-sound correspondences. PhonoBlocks uses 3D tangible letters that change colour to draw attention to the moment that adding other letters changes the sounds. We then present a mixed-methods case study with ten Mandarin-speaking children in China using our system. Results showed that the Chinese children achieved significant learning gains relative to their baseline performance after PhonoBlocks instruction. The results also point to design features of our system that enabled behaviours that are correlated with learning. We compare the results of this study to a different study with eight at-risk monolingual English-speaking children in Canada using PhonoBlocks in learning to read and spell. By comparing results, we generalize and make three recommendations for designing tangible reading systems for all children who must learn the alphabetic principle. We also discuss three recommendations that are specifically for children learning English as a foreign language.

Close

https://www.sciencedirect.com/science/article/pii/S2212868917300867

doi:https://doi.org/10.1016/j.ijcci.2018.08.001

Close
2017
Fan, Min, Antle, Alissa N., Hoskyn, Maureen, Neustaedter, Carman, Cramer, Emily S.

Why Tangibility Matters: A Design Case Study of At-Risk Children Learning to Read and Spell Proceedings Article

In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1805–1816, Association for Computing Machinery, Denver, Colorado, USA, 2017, ISBN: 9781450346559.

Abstract | Links | BibTeX

@inproceedings{10.1145/3025453.3026048,
title = {Why Tangibility Matters: A Design Case Study of At-Risk Children Learning to Read and Spell},
author = {Min Fan and Alissa N. Antle and Maureen Hoskyn and Carman Neustaedter and Emily S. Cramer},
url = {https://doi-org.proxy.lib.sfu.ca/10.1145/3025453.3026048},
doi = {10.1145/3025453.3026048},
isbn = {9781450346559},
year = {2017},
date = {2017-01-01},
urldate = {2017-01-01},
booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
pages = {1805–1816},
publisher = {Association for Computing Machinery},
address = {Denver, Colorado, USA},
series = {CHI '17},
abstract = {Tangibles may be effective for reading applications. Letters can be represented as 3D physical objects. Words are spatially organized collections of letters. We explore how tangibility impacts reading and spelling acquisition for young Anglophone children who have dyslexia. We describe our theory-based design rationale and present a mixed-methods case study of eight children using our PhonoBlocks system. All children made significant gains in reading and spelling on trained and untrained (new) words, and could apply all spelling rules a month later. We discuss the design features of our system that contributed to effective learning processes, resulting in successful learning outcomes: dynamic colour cues embedded in 3D letters, which can draw attention to how letter(s) position changes their sounds; and the form of 3D tangible letters, which can enforce correct letter orientation and enable epistemic strategies in letter organization that simplify spelling tasks. We conclude with design guidelines for tangible reading systems.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}

Close

Tangibles may be effective for reading applications. Letters can be represented as 3D physical objects. Words are spatially organized collections of letters. We explore how tangibility impacts reading and spelling acquisition for young Anglophone children who have dyslexia. We describe our theory-based design rationale and present a mixed-methods case study of eight children using our PhonoBlocks system. All children made significant gains in reading and spelling on trained and untrained (new) words, and could apply all spelling rules a month later. We discuss the design features of our system that contributed to effective learning processes, resulting in successful learning outcomes: dynamic colour cues embedded in 3D letters, which can draw attention to how letter(s) position changes their sounds; and the form of 3D tangible letters, which can enforce correct letter orientation and enable epistemic strategies in letter organization that simplify spelling tasks. We conclude with design guidelines for tangible reading systems.

Close

https://doi-org.proxy.lib.sfu.ca/10.1145/3025453.3026048

doi:10.1145/3025453.3026048

Close
2016
Cramer, Emily S., Antle, Alissa N., Fan, Min

The Code of Many Colours: Evaluating the Effects of a Dynamic Colour-Coding Scheme on Children's Spelling in a Tangible Software System Proceedings Article

In: Proceedings of the The 15th International Conference on Interaction Design and Children, pp. 473–485, Association for Computing Machinery, Manchester, United Kingdom, 2016, ISBN: 9781450343138.

Abstract | Links | BibTeX

@inproceedings{10.1145/2930674.2930692,
title = {The Code of Many Colours: Evaluating the Effects of a Dynamic Colour-Coding Scheme on Children's Spelling in a Tangible Software System},
author = {Emily S. Cramer and Alissa N. Antle and Min Fan},
url = {https://doi-org.proxy.lib.sfu.ca/10.1145/2930674.2930692},
doi = {10.1145/2930674.2930692},
isbn = {9781450343138},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the The 15th International Conference on Interaction Design and Children},
pages = {473–485},
publisher = {Association for Computing Machinery},
address = {Manchester, United Kingdom},
series = {IDC '16},
abstract = {Dyslexia is a severe impairment in reading and spelling that affects 10% of children in English-speaking countries. One area of difficulty is learning spelling rules that require attention to other letters within a word (i.e., context): for example, why grapple requires two ps while staple requires one. Poor visual attention contributes to children's difficulties. Computer-based programs that use multisensory cues have helped children learn simple letter-sound relations, but not contextual spelling rules. In this paper we present three theoretically derived principles that can be used to design dynamic colour codes for a variety of contextual spelling rules in software systems. We discuss how we used our principles to design the colour scheme for a single contextual spelling rule in our tangible software system, called PhonoBlocks. We evaluate its effectiveness in a field study with nine dyslexic children. On the basis of our findings, we conclude that our approach to using dynamic colour may help children with dyslexia to learn contextual spelling rules, but that individual factors impact the colours' effectiveness. We conclude by suggesting ways our dynamic colour-coding principles can be implemented in other systems taking into consideration individual factors that also impact their effectiveness.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}

Close

Dyslexia is a severe impairment in reading and spelling that affects 10% of children in English-speaking countries. One area of difficulty is learning spelling rules that require attention to other letters within a word (i.e., context): for example, why grapple requires two ps while staple requires one. Poor visual attention contributes to children's difficulties. Computer-based programs that use multisensory cues have helped children learn simple letter-sound relations, but not contextual spelling rules. In this paper we present three theoretically derived principles that can be used to design dynamic colour codes for a variety of contextual spelling rules in software systems. We discuss how we used our principles to design the colour scheme for a single contextual spelling rule in our tangible software system, called PhonoBlocks. We evaluate its effectiveness in a field study with nine dyslexic children. On the basis of our findings, we conclude that our approach to using dynamic colour may help children with dyslexia to learn contextual spelling rules, but that individual factors impact the colours' effectiveness. We conclude by suggesting ways our dynamic colour-coding principles can be implemented in other systems taking into consideration individual factors that also impact their effectiveness.

Close

https://doi-org.proxy.lib.sfu.ca/10.1145/2930674.2930692

doi:10.1145/2930674.2930692

Close
Fan, Min, Antle, Alissa N., Cramer, Emily S.

Design Rationale: Opportunities and Recommendations for Tangible Reading Systems for Children Proceedings Article

In: Proceedings of the The 15th International Conference on Interaction Design and Children, pp. 101–112, Association for Computing Machinery, Manchester, United Kingdom, 2016, ISBN: 9781450343138.

Abstract | Links | BibTeX

@inproceedings{10.1145/2930674.2930690,
title = {Design Rationale: Opportunities and Recommendations for Tangible Reading Systems for Children},
author = {Min Fan and Alissa N. Antle and Emily S. Cramer},
url = {https://doi-org.proxy.lib.sfu.ca/10.1145/2930674.2930690},
doi = {10.1145/2930674.2930690},
isbn = {9781450343138},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the The 15th International Conference on Interaction Design and Children},
pages = {101–112},
publisher = {Association for Computing Machinery},
address = {Manchester, United Kingdom},
series = {IDC '16},
abstract = {Tangible User Interfaces (TUIs) have been suggested to have the potential to support learning for children. Despite the increasing number of TUI reading systems there are few design guidelines for children, especially for those with dyslexia (a specific difficulty in language acquisition skills). In this paper we discuss four design opportunities and five design recommendations for designing tangible reading systems for children, particularly those with dyslexia. We ground our analysis using theories of the causes and interventions for dyslexia, best multisensory training practices and existing research on TUIs that support learning to read for children. We describe our tangible reading system, called PhonoBlocks, focusing on two core design features which take advantage of these opportunities. We also describe how we iteratively fine-tuned the details of our design based on our recommendations, an expert review and feedback from tutors who work with children with dyslexia every day. We include a discussion of design trade-offs in our process. This design rationale paper contributes to the growing research on designing tangible spelling and reading systems for children.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}

Close

Tangible User Interfaces (TUIs) have been suggested to have the potential to support learning for children. Despite the increasing number of TUI reading systems there are few design guidelines for children, especially for those with dyslexia (a specific difficulty in language acquisition skills). In this paper we discuss four design opportunities and five design recommendations for designing tangible reading systems for children, particularly those with dyslexia. We ground our analysis using theories of the causes and interventions for dyslexia, best multisensory training practices and existing research on TUIs that support learning to read for children. We describe our tangible reading system, called PhonoBlocks, focusing on two core design features which take advantage of these opportunities. We also describe how we iteratively fine-tuned the details of our design based on our recommendations, an expert review and feedback from tutors who work with children with dyslexia every day. We include a discussion of design trade-offs in our process. This design rationale paper contributes to the growing research on designing tangible spelling and reading systems for children.

Close

https://doi-org.proxy.lib.sfu.ca/10.1145/2930674.2930690

doi:10.1145/2930674.2930690

Close

Project Overview

Team

Graduate Students

Funded By

Publications

2020

2018

2017

2016