Huang, Shihong

The collaboration between human and computer systems has grown astronomically over the past few years. The ability of software systems adapting to human's input is critical in the symbiosis of human-system co-adaptation, where human and software-based systems work together in a close partnership to achieve synergetic goals. However, it is not always clear what kinds of human’s input should be considered to enhance the effectiveness of human and system co-adaptation. To address this issue, this research describes an approach that focuses on incorporating human emotion to improve human-computer co-adaption. The key idea is to provide a formal framework that incorporates human emotions as a foundation for explainability into co-adaptive systems, especially, how software systems recognize human emotions and adapt the system’s behaviors accordingly. Detecting and recognizing optimum human emotion is a first step towards human and computer symbiosis. As the first step of this research, we conduct a comparative review for a number of technologies and methods for emotion recognition. Specifically, testing the detection accuracy of facial expression recognition of different cloud-services, algorithms, and methods.
Secondly, we study the application of emotion recognition within the areas of e-learning, robotics, and explainable artificial intelligence (XAI). We propose a formal framework that incorporates human emotions into an adaptive e-learning system, to create a more personalized learning experience for higher quality of learning outcomes. In addition, we propose a framework for a co-adaptive Emotional Support Robot. This human-centric framework adopts a reinforced learning approach where the system assesses its own emotional re-actions.

It is becoming increasingly important for an autonomous system to be able to explain its actions to humans in order to improve trust and enhance human-machine collaboration. However, providing the most appropriate kind of explanations – in terms of length, format, and presentation mode of explanations at the proper time – is critical to enhancing their effectiveness. Explanation entails costs, such as the time it takes to explain and for humans to comprehend and respond. Therefore, the actual improvement in human-system tasks from explanations (if any) is not always obvious, particularly given various forms of uncertainty in knowledge about humans.
In this research, we propose an approach to address this issue. The key idea is to provide a structured framework that allows a system to model and reason about human personality traits as critical elements to guide proper explanation in human and system collaboration. In particular, we focus on the two concerns of modality and amount of explanation in order to optimize the explanation experience and improve overall system-human utility. Our models are based on probabilistic modeling and analysis (PRISM-games) to determine at run time what the most effective explanation under uncertainty is. To demonstrate our approach, we introduce a self-adaptative system called Grid – a virtual game – and the Stock Prediction Engine (SPE), which allows an automated system and a human to collaborate on the game and stock investments. Our evaluation of these exemplars, through simulation, demonstrates that a human subject’s performance and overall human-system utility is improved when considering the psychology of human personality traits in providing explanations.

The Agile methodologies have attracted the software development industry's attention due to their capability to overcome the limitations of the traditional software development approaches and to cope with increasing complexity in system development. Scrum is one of the Agile software development processes broadly adopted by industry. Scrum promotes frequent customer involvement and incremental short releases. Despite its popular use,
Scrum’s requirements engineering stage is inadequately defined which can lead to increase development time and cost, along with low quality or failure for the end products. This research shows the importance of activity planning of requirements engineering in improving the product quality, cost, and scheduling as well as it points out some drawbacks of Agile practices and available solutions. To improve the Scrum requirements engineering by overcoming its challenges in cases, such as providing a comprehensive understanding of the customer’s needs and addressing the effects of the challenges in other cases, such as frequent changes of requirements, the Design Thinking model is integrated into the Scrum framework in the context of requirements engineering management. The use of the Design Thinking model, in the context of requirements engineering management, is validated through an in-depth scientific study of the IBM Design Thinking framework. In addition, this research presents an Items Prioritization dEcision Support System (IPESS) which is a tool to assist the Product Owners for requirements prioritization. IPESS is built on information collected in the Design Thinking model. The IPESS tool adopts Analytic Hierarchy Process (AHP) technique and PageRank algorithm to deal with the specified factors and to achieve the optimal order for requirements items based on the prioritization score. IPESS is a flexible and comprehensive tool that focuses on different important aspects including customer satisfaction and product quality. The IPESS tool is validated through an experiment that was conducted in a real-world project

Autism spectrum disorders (ASDs) are one of the complex, pervasive, and multifactorial
neurodevelopmental conditions which affect one in 68 children. Scientific research has
proven the efficiency of using technologies to improve communication and social skills of
autistic children. The use of technological devices, such as mobile applications and
multimedia, increase the interest of autistic children to learn while playing games. This
thesis presents the re-engineering, extension, and evolution of an existing prototype
Windows-based mobile application called Ying to become an Android mobile application
which is augmented with facial and emotion recognition. This mobile app complements
different approaches of traditional therapy, such as Applied Behavior Analysis (ABA).
Ying integrates different computer-assisted technologies, including speech recognition,
audio and visual interaction, and mobile applications to enhance autistic children’s social
behavior and verbal communication skills. An evaluation of the efficacy of using Ying has
been conducted and its results are presented in the thesis.

Scrum is one of the Agile software development processes broadly adopted in industry. Scrum promotes frequent customer involvements and incremental short release. Sprint planning is a critical step in Scrum that sets up next release goals and lays out plans to achieve those goals. This thesis presents a Sprint Planning dEcision Support System (SPESS) which is a tool to assist the managers for Sprint planning. Among considering other Sprint planning factors, SPESS takes into consideration developer competency, developer seniority and task dependency. The results are that the assignments of the tasks of each Sprint to developers guarantee that each team member contributes to their fullest potential, and project planning is optimized for the shortest possible time. Keywords—Scrum, Sprint planning, planning poker, competence, task dependence, Hungarian algorithm, Essence.

The software development process is an incremental and iterative activity.
Source code is constantly altered to reflect changing requirements, to respond to
testing results, and to address problem reports. Proper software measurement that
derives meaningful numeric values for some attributes of a software product or
process can help in identifying problem areas and development bottlenecks. Impact
analysis is the evaluation of the risks associated with change requests or problem
reports, including estimates of effects on resources, effort, and schedule. This thesis
presents a methodology called VITA for applying software analysis techniques to
configuration management repository data with the aim of identifying the impact on
file changes due to change requests and problem reports. The repository data can be
analyzed and visualized in a semi-automated manner according to user-selectable
criteria. The approach is illustrated with a model problem concerning software
process improvement of an embedded software system in the context of performing
high-quality software maintenance.

In the globalization software development environments, where the development
activities are distributed geographically and temporally, it is increasingly important
for the Computer Aided Software Engineering (CASE) tools to maintain the
information (both syntactic and semantic) captured in the design models. The Unified
Modeling Language (UML) is the de facto standard for modeling software
applications and UML diagrams serve as graphical documentations of the software
system. The interoperability of UML modeling tools is important in supporting the
models exchange, and further support design reuse. Tool interoperability is often
implemented using XML Metadata Interchange (XMI). Unfortunately, there is a loss
of fidelity of the design documentation when transforming between UML and XMI
due to the compatibility of different versions of UML, XMI and add-on proprietary
information, which hinder reuse. This thesis evaluates the interoperability of UML
modeling tools by assessing the quality of XMI documents representing the design.
Case studies in this thesis demonstrate a framework of preserving the fidelity of UML
model 's data when importing and exporting different UML models in a distributed
heterogeneous environment.

People with Autism Spectrum Disorders, particularly Asperger's Syndrome, have
a set of symptoms, such as difficulty processing emotive facial expressions and making
less eye contact, that affect the individual's social interactions. The prototype developed
is geared to help children with Asperger's Syndrome recognize facial expressions,
improve eye contact, and to reinforce connections between facial and verbal components
of expressions. This application utilizes a computer generated human head, an avatar, to
interact with the children while displaying various emotions and using games to increase
the children's eye contact. The unique properties of the avatar utilized allow for a
comprehensive coverage of the children's attention spectrum to maximize their learning
potential. Knowing that about 17% of Autism Spectrum Disorders belong to the
Asperger's Syndrome subclass, the positive impact of this application on their life would
be of great importance for their families and the community.

Modern software systems have grown in complexity and expense, even while costs for the supporting
hardware have decreased. Humans still comprise a significant cost from the development to operation
and evolution of software, and must be increasingly more skilled as systems grow more complex.
Further, legacy software has become more prevalent, as sophisticated enterprise systems developed
over many years approach complexity that is too expensive to replace or re-engineer. One of the major
software engineering challenges today is the maintenance and evolution of these existing legacy
systems to extend the business value of the software, while mitigating the costs and increased
complexity. Approaches practiced by researchers and engineers include software reuse, serviceoriented
architecture, and autonomic computing principles to improve the reliability of systems, while
removing the need for human monitoring and intervention where possible. In this paper, we describe a
real-world project in which a multi-channel vendor of musical equipment sought to extend their legacy
enterprise system to automatically process customer orders originating from a third party retailer
specializing in ecommerce. To realize the solution, an addition to the existing enterprise system was
developed using autonomic computing, software reuse, and service-oriented architectural approaches,
producing a highly automated order processing stream that was self-monitoring, self-adaptive, and
fault-tolerant, while minimizing the need for additional human resources to monitor the system and
process the data.

As computing technology continues to advance, it has become increasingly difficult to
find businesses that do not rely, at least in part, upon the collection and analysis of data for
project management and process improvement. The cost of software, for example, tends to
increase over time due to its complexity and the expense of employing humans to develop,
maintain, and evolve it. To help control costs, organizations often seek to improve the process by
which software systems are developed and evolved. Improvements can be realized by
discovering previously unknown or hidden relationships between the artifacts generated as a
result of developing a complex software system. The objective of this the work was to engineer a
visualization software tool that helps managers and engineers better plan for future projects by
helping them discover new knowledge gained by synthesizing and visualizing data mined from
software repository records from their own previous projects.