Techniques for Augmented-Tangibles on Mobile Devices for Early Childhood Learning

Abstract

Integrating physical learning materials with mobile device applications may have benefits for early childhood learning. We present three techniques for creating a hybrid tangible-augmented reality (T-AR) enabling technology platform. This platform enables researchers to develop applications that use readily available physical learning materials, such as letters, numbers, symbols or shapes. The techniques are visual marker-based; computer-vision and machine-learning; and capacitive touches. We describe details of implementation and demonstrate these techniques through a use case of a reading tablet app that uses wooden/plastic letters for input and augmented output. Our comparative analysis revealed that the machine-learning technique most flexibly sensed different physical letter sets but had variable accuracy impacted by lighting and tracking lag at this time. Lastly, we demonstrate how this enabling technology can be generalized to a variety of early learning apps through a second use case with physical numbers.

Reflection

This project is a product of numerous discussions and explorations on different technologies that target on an often overlooked but important application area: early childhood learning. Nowadays almost every kid has used a tablet. But the question is can we incorporate tangibles that are known to help children to understand the world via interactions with physical objects?

Our goal was to find ways that work with real-work constraints such as ease of development, cost-effectiveness, scalability and broad availability. We wanted adopters to be able to customize the applications and acquire the physical objects fast and cheaply. Amongst the three techniques we compared, we found that using computer-vision and machine-learning to recognize tangible objects through a training process, which allowed applications to augment them (e.g., overlay letters on top of the objects that shape like those letters), had most potential.

It’s a neat project with a meaningful application. However, we didn’t have the expertise to fully implement what we envisioned. Hopefully someone will take it up and make it into something (at the conference we heard some companies were working towards that direction, so it’s promising).

Recommended citation: V. Cheung, A. N. Antle, S. Sarker, M. Fan, J. Fan, and P. Pasquier. 2020. "Techniques for augmented-tangibles on mobile devices for early childhood learning". In Proceedings of the Interaction Design and Children Conference (IDC'20). ACM, New York, NY, USA, 589–601. https://doi.org/10.1145/3392063.3394412
Download Paper