Augmented reality based experience for EdTech in Dyslexia

Accessible design / AR Design / Unity Development / Vuforia / VUI Design
My Role
Designer and Developer
Team Members
Toshi Gupta
Ruchita Lodha
Duration
8 weeks
(March 2020-May 2020)
The application
Dyslexia is a language based learning disorder which is found ind kids who have difficulty often avoid reading because it's stressful. We wanted to use an immersive technology like AR/VR/MR for good together with digital game based learning to enhance the learning experience and help students retain things in a less stressful and playful way.

Final prototype of the Edtech Augmented-reality based mobile application
Playing with the developed prototype on an Android device
My Role
Designer and Developer
Team
Toshi Gupta
Ruchita Lodha
Duration
8 weeks (March 2020-May 2020)
Core responsibilities
  • Collecting insights based on research in the field of dyslexia and augmented reality
  • Designing workflows and prototypes
  • Developing and testing final application on Unity using Vuforia on Android phone
Problem
We wanted to work on the problem of building an accessible learning experience for kids diagnosed with dyslexia between the age of 3-6 years. The main motivation of solving this problem was to help kids be less stressed mentally while learning to read and write alphabets
Challenges
Let's look at the specific set of challenges faced by dyslexic children in a classroom step:
Solution
We decided to build an alphabet learning application for kids diagnosed with dyslexia between the age of 3-6 years that uses augmented reality and digital game-based learning. To validate out idea of this solution, we did qualitative research through literature review of research papers published in the field of dyslexia and augmented reality.
The vision of the application being used on phones and iPads
Literature Review
As the format of this project required a faster solution and iteration based on usability, my teammate and I collectively conducted secondary research by reading research papers and diving in the online videos of dyslexia learning .
The work ofProf. Luz Relloin her venturechangedyslexia.comhas also motivated us to pursue this project.
Why AR?
I did a qualitative research on ways and methods adopted to help dyslexic kids learn faster like the Orton Gillingham method and the Wilson Method. The team studied the online videos available for teachers to pick out the key components that are very important while helping in dyslexia.
We realized thats supporting these methods with augmented reality for multi-sensory learning (Kinesthetics, Visual, Tactile, Phonetics) coupled with game based learning would help students get accurate and faster in alphabet identification.
User Journey Mapping
Our potential users are the kids who find it difficult to read, learn and identify alphabets.
The idea is to build an application that augments the alphabet cards/alphabet book with 3D holograms and support the phonetic learning through sound assistance. It will have 3 main features:
Sketching
The best way to convey how the learning experience would look like would be to go the traditional way of sketching. The tricky part about sketching AR experiences is to show the device and differentiate between real world and augmented objects. For solving this, I have created a legend based on color of the objects. The drawings in black are the real world objects, blue ones are augmented objects and the green drawings represent sound in the application.
Technology decisions
*This project was carried out during COVID-19 and therefore our plan of enabling the application on Hololens could not go through.
Developing and testing application in Unity and Vuforia using marker
Voice assistant
To help the students engage and navigate better and the nature of dyslexia as health condition, there was a need of a voice assistant. We introduced a voice assistant - a friendly hippo, Jojo, who mainly helped in the following ways:
3D hologram of friend Jojo - the voice assistant
Sound Design
"The sounds in a UI should work like a well-coordinated orchestra, both with each other and within the surrounding interface." - Material design
I decided to go with narrative patterns in the voice interface where the assistant walks the user through the whole application and also responds to the actions with a feedback. Keeping the principles of VUI design by Google Conversation design in mind, we designed the voice over of Jojo in the application:
‍
Interactions

2D UI and menu
The application menu with Jojo introducing you to how the different options
The game starts with a 2D menu with an introduction by a friendly assistant JoJo. JoJo guides the user to the different sections of the application like Learn, Play and draw.
You can listen to JOJO's instructions in the final video attached.
Drawing in the air
Hold the pen (in future, the mobile) and follow the directions to draw
The students are asked follow the blue line and repeat the drawing with their marker in the air. The idea is to register the shape and drawing related to the alphabet by practicing with an augmented pen. For prototyping purposes the pen is attached to a marker but in ideal scenario the pen is controlled by phone movement.

Shooting with the ball
Throw the object in the alley to score points
For letter identification, the application is supported with physics enabled bowling game. The students' are asked to shoot the associated hologram to the right letter glasses.
The right and wrong answer are distinguished with sounds.
Drag & Drop | Success & Failure
Drop your object in the right ring till you get it right!
To test students' capability to identify the learned alphabet, they are asked to drag and drop the associated 3D hologram into the ring options. Based on the answer the application reacts gently with animations of right and wrong answer.

Reflections
Due to COVID-19, the prototype could not go under usability testing, yet. We are in talks with the organization based out of University of MichiganDyslexia Help.I believe there is very high potential of the emerging technologies of XR in educational learning and training.
My teammate was also my homie and we had fun moments building this in home quarantine
01
Team and Role
My role
Designer
Developer
Team
Toshi Gupta
Ruchita Lodha
Duration
8 Weeks
Mar'20-May'20
Tools
Unity
Vuforia
ARCore
Figma
02
Motivation
Dyslexia is a language based learning disorder which is found in 1 in 5 students and kids who have difficulty often avoid reading because it's stressful.
We wanted to use an immersive technology like AR/VR/MR for good together with digital game based learning to enhance the learning experience and help students retain things in a less stressful and playful way.
03
Problem Statement
To build an accessible learning experience for kids diagnosed with dyslexia between the age of 3-6 years. To help kids be less stressed mentally while learning to read and write alphabets.
04
Challenges in a classroom setup
There are specific set of challenges faced by dyslexic children in a classroom setup:
1. Manual Learning: Lack of multimedia based interaction to address learnability issues.
2.Individual learning approach: Teachers are less focused on students who have dyslexia in a classroom setup.
3. Less variation in learning: Lack of use of variety of senses such as seeing, hearing, talking, and touching.
05
Our Solution
To build an augmented reality game-based alphabet learning application for kids between age 3-6 years by augmenting the real world objects around them. The application would help students learn alphabets with phonetics and associated real world objects, identify and differentiate between alphabets and write them.
06
Why Augmented Reality?
We did a qualitative research on ways and methods adopted to help dyslexic kids learn faster like the Orton Gillingham method and the Wilson Method. We studied the online videos available  for teachers to pick out the key components that are very important while helping in dyslexia.
We realized thats supporting these methods with augmented reality for multi-sensory learning (Kinesthetics, Visual, Tactile, Phonetics) coupled with game based learning would help students get accurate and faster in alphabet identification
07
User Journey Mapping
Potential users: Kids who find it difficult to read, learn and identify alphabets.
The idea is to build an application that augments the alphabet cards/alphabet book with 3D holograms and support the phonetic learning through sound assistance. It will have 3 main features:
1. Learning about the shape and usage of alphabet
2. Identification of alphabets after learning through games
3. Learning to draw the alphabets through guidance
08
Lo-Fi Sketches
09
Introducing JoJo - the voice assistant
‍To help the students engage and navigate better with the learning and games, we introduced a cute animated hippo!
The main reasons behind having a voice assistant are:
1. Help students navigate between different parts of the application
2.Lower the burden of reading instructions for dyslexic kids
3.Give a comfortable learning experience through their "Friend- JoJo"
10
Technology decisions
1. The application is developed in Unity and tested on an android and iOS mobile device
2. To enable enhancements on a book/alphabet card, we decided to go for a marker-based approach in Vuforia

*This project was carried out during COVID-19 and therefore our plan of enabling the application on Hololens could not go through.
11
High Fidelity prototype and interactions
2D menu and instructions: The game starts with a 2D menu with an introduction by a friendly assistant JoJo. JoJo guides the user to the different sections of the application like Learn, Play and draw.
You can listen to JOJO's instructions in the final video attached.
Dragging/dropping and success/failure animations: To test students' capability to identify the learned alphabet, they are asked to drag and drop the associated 3D hologram into the ring options. Based on the answer the application reacts gently with animations of right and wrong answer.
Shooting with a ball: For letter identification, the application is supported with physics enabled bowling game. The students' are asked to shoot the associated hologram to the right letter glasses.
The right and wrong answer are distinguished with sounds.
Drawing: The students can reference the drawing technique animation and try drawing with a physical marker.
12
Final video
12
Reflection and learnings
All Projects
Next Project
Related projects
© TOSHI GUPTA 2020