Human-Computer Interaction

The collaboration between human and computer systems has grown astronomically over the past few years. The ability of software systems adapting to human's input is critical in the symbiosis of human-system co-adaptation, where human and software-based systems work together in a close partnership to achieve synergetic goals. However, it is not always clear what kinds of human’s input should be considered to enhance the effectiveness of human and system co-adaptation. To address this issue, this research describes an approach that focuses on incorporating human emotion to improve human-computer co-adaption. The key idea is to provide a formal framework that incorporates human emotions as a foundation for explainability into co-adaptive systems, especially, how software systems recognize human emotions and adapt the system’s behaviors accordingly. Detecting and recognizing optimum human emotion is a first step towards human and computer symbiosis. As the first step of this research, we conduct a comparative review for a number of technologies and methods for emotion recognition. Specifically, testing the detection accuracy of facial expression recognition of different cloud-services, algorithms, and methods.
Secondly, we study the application of emotion recognition within the areas of e-learning, robotics, and explainable artificial intelligence (XAI). We propose a formal framework that incorporates human emotions into an adaptive e-learning system, to create a more personalized learning experience for higher quality of learning outcomes. In addition, we propose a framework for a co-adaptive Emotional Support Robot. This human-centric framework adopts a reinforced learning approach where the system assesses its own emotional re-actions.

It is becoming increasingly important for an autonomous system to be able to explain its actions to humans in order to improve trust and enhance human-machine collaboration. However, providing the most appropriate kind of explanations – in terms of length, format, and presentation mode of explanations at the proper time – is critical to enhancing their effectiveness. Explanation entails costs, such as the time it takes to explain and for humans to comprehend and respond. Therefore, the actual improvement in human-system tasks from explanations (if any) is not always obvious, particularly given various forms of uncertainty in knowledge about humans.
In this research, we propose an approach to address this issue. The key idea is to provide a structured framework that allows a system to model and reason about human personality traits as critical elements to guide proper explanation in human and system collaboration. In particular, we focus on the two concerns of modality and amount of explanation in order to optimize the explanation experience and improve overall system-human utility. Our models are based on probabilistic modeling and analysis (PRISM-games) to determine at run time what the most effective explanation under uncertainty is. To demonstrate our approach, we introduce a self-adaptative system called Grid – a virtual game – and the Stock Prediction Engine (SPE), which allows an automated system and a human to collaborate on the game and stock investments. Our evaluation of these exemplars, through simulation, demonstrates that a human subject’s performance and overall human-system utility is improved when considering the psychology of human personality traits in providing explanations.

Autism spectrum disorders (ASDs) are one of the complex, pervasive, and multifactorial
neurodevelopmental conditions which affect one in 68 children. Scientific research has
proven the efficiency of using technologies to improve communication and social skills of
autistic children. The use of technological devices, such as mobile applications and
multimedia, increase the interest of autistic children to learn while playing games. This
thesis presents the re-engineering, extension, and evolution of an existing prototype
Windows-based mobile application called Ying to become an Android mobile application
which is augmented with facial and emotion recognition. This mobile app complements
different approaches of traditional therapy, such as Applied Behavior Analysis (ABA).
Ying integrates different computer-assisted technologies, including speech recognition,
audio and visual interaction, and mobile applications to enhance autistic children’s social
behavior and verbal communication skills. An evaluation of the efficacy of using Ying has
been conducted and its results are presented in the thesis.

Consumers are exposed to thousands of marketing messages every day. In such a
cluttered environment, gaining consumers· attention becomes an increasingly important
business objective. This study expands the concept of attention from a simple view of
attention as se lection of stimuli to a more elaborate two step process consisting of (I)
preattention and (2) focal attention. The focus of this research is on preattention, which is
determined by physical characteristics of objects in a visual scene. This study also
i1Pproves the measurement ofpreattention by surveying the neuroscience literature and
using a computational model to measure preattention. This improved measure allows us
to provide an enhanced explanation of how preattention f::tcilitates mere exposure effects.
Results confirm that preattentive processing of an ad in a visual scene affects liking of
that ad even when people do not remember previously seeing the advertisement. The study also finds that subtle. preattentive processes require increasing amounts of time in
order to affect focal attention and attitude toward the ad.

A user interface that has objects familiar to the user will be easier to use. In this thesis, a user interface that is customizable to any color bitmap is proposed. The most significant problem with this approach is the problem of finding objects in a color bitmap. A solution to the problem is proposed and evaluated using an analysis tool, developed for this thesis, called Workbench. Current image detection methods are evaluated and compared to the solution proposed using Workbench. The proposed solution is then evaluated for the YIQ and HSI color mappings. The results of this investigation and recommendations for future work is proposed.

Computers are increasingly a part of college and university instruction. Interactive hypermedia applications are being introduced throughout much of the curriculum as a possible solution to both improving educational outcomes and expanding educational horizons. The purpose of the present study was to investigate the effects of an interactive hypermedia application and a section of text on a measure of learning and understanding called concept mapping. The inter-rater reliability of concept map scores has not been reported previously in the literature. Results in this study concerning the reliability of concept map scoring procedures indicated that the continued improvement of inter-rater reliability is desirable if concept mapping is to actualize its potential as a practical, useful, and unique learning tool. Results suggest that concept mapping appears capable of assessing: (a) baseline knowledge, (b) meaningful learning, (c) the construction of new knowledge, and (d) knowledge change. Its usefulness in these areas and as an alternative or addition to standardized assessment is contingent, however, upon demonstrations of validity and reliability. Suggestions for further concept map research included: (a) replicating the present study with other measures of cognitive style across a wide variety of interactive hypermedia software applications, (b) doing longitudinal studies of concept mapping, (c) improving the reliablity of concept map scoring and evaluation, (d) looking at other aspects of cognition and information processing related to concept mapping, (e) using computer-based concept mapping tools, and (f) using concept maps as templates for the organization and integration of hypermedia elements.

Perceptual video coding has been a promising area during the last years. Increases in compression ratios have been reported by applying foveated video coding techniques where the region of interest (ROI) is selected by using a computational attention model. However, most of the approaches for perceptual video coding only use visual features ignoring the auditory component. In recent physiological studies, it has been demonstrated that auditory stimuli affects our visual perception. In this work, we validate some of those physiological tests using complex video sequence. We designed and developed a web-based tool for video quality measurement. After conducting different experiments, we observed that in the general reaction time to detect video artifacts was higher when video was presented with the audio information. We observed that emotional information in audio guide human attention to particular ROI. We also observed that sound frequency change spatial frequency perception in still images.

While new technologies are often used to facilitate regular people's lives, they often fail to see their potential in helping disabled people. Augmented reality, one of the newest state-of-the-art technologies, offers users the opportunity to add virtual information to their real world surroundings in real time. It also has the potential to not only augment the sense of sight, but also other senses such as hearing. Augmented reality could be used to offer the opportunity to complement users' missing sense. In this thesis, we study augmented reality technologies, systems and applications, and suggest the future of AR applications. We explain how to integrate augmented reality into iOS applications and propose an augmented reality application for hearing augmentation using an iPad2. We believe mobile devices are the best platform for augmented reality as they are widespread and their computational power is rapidly growing to be able to handle true AR applications.