An Augmentative System with Facial and Emotion Recognition for Improving the Skills of Children with Autism Spectrum Disorders

Autism spectrum disorders (ASDs) are one of the complex, pervasive, and multifactorial
neurodevelopmental conditions which affect one in 68 children. Scientific research has
proven the efficiency of using technologies to improve communication and social skills of
autistic children. The use of technological devices, such as mobile applications and
multimedia, increase the interest of autistic children to learn while playing games. This
thesis presents the re-engineering, extension, and evolution of an existing prototype
Windows-based mobile application called Ying to become an Android mobile application
which is augmented with facial and emotion recognition. This mobile app complements
different approaches of traditional therapy, such as Applied Behavior Analysis (ABA).
Ying integrates different computer-assisted technologies, including speech recognition,
audio and visual interaction, and mobile applications to enhance autistic children’s social
behavior and verbal communication skills. An evaluation of the efficacy of using Ying has
been conducted and its results are presented in the thesis.