|
|
|
|
|
|
|
|
|
|
 Top |
Emotion Recognition
One of the most interesting topics in the research in affective computing is the Emotion Recognition. Humans express emotions through a variety of different channels: Autonomic Nervous System (ANS) sygnals, facial expressions, voice tone, body posture and gestures are only some of the different channels to exploit.
Emotion recognition is for sure one of the most important factors for the developing of MAUIs as the computer will always need to recognize an emotion in order to define a suitable adapted behaviour.
At the affective computing group we are focusing on recognition of the user's emotion through ANS signals but we are also exploring different kind of recognition, in particular facial expressions and voice tone.
Emotion Recognition Through ANS Signals
The Autonomic Nervous System (ANS) is a part of our nervous system, which has bee shown to bemodulated when we experience emotions. We can access to the ANS by measuring signals such as the skin conductance, the heart beat, the temperature, etc. Thus we have an indirect measure of the individual's emotion through the study of such signals.
Real time emotional features extraction in physiological signals.
To measure emotion of the user of computer application, we are focus on (1) the use and design of wireless hardware/software system to measure heart beat and skin conductance. We also work on (2) the real-time extraction of features which are related to emotions in such signal. Moreover, (3) we build a model of interpretation of such signals in terms of emotion by combining subjective emotional mental state with physiological signals involved in emotion toward a real-time emotional measure.
Related to the design of a user model based on psychophysiology, we work on the real time extraction of features related to emotion into the physiological signals. This leads to the design of software system which can extract significant features (as Heart Rate Variability) synchronized with multimedia stimuli presentations. Synchronization is also researched among the different features extracted.
Interpretation of emotion based on physiogical features: design of a user parametric model.
To be able to measure emotion trough computer, We are focusing on Pychophysiology (combining subjective emotional mental state with physiological signals involved in emotion) toward a real-time emotional measure. Our proposed approach consists on applying the concepts of first and third person methodology to ANS emotion recognition to do an interpretation of physiological signal tailored to individuals.
The goal is to design a model which include the psychophysiological mappings of the average population, the specificities of each user, and the specifities of each user under specific conditions (as the day-dependance).
The approach we propose [Villon & Lisetti 2006a] consists on applying the concepts of first and third person methodology to ANS emotion recognition to do an interpretation of physiological signal tailored to individuals. The goal is to design a model which include the psychophysiological mappings of the average population, the specificities of each user, and the specifities of each user under specific conditions (as the day-dependance).
Top |
Paradigms for Dynamic Facial Expression Recognition
At the ASCG we are working on the definition of the steps which are foundamental to the recognition of facial expressions using Scherer's [Scherer 2001-2006] component process theory (CPT) of emotions (Swiss Centre for Affective Sciences).
The approach we propose bases on the recognition of Ekman and Friesel's [Ekman & Friesel 1971] action units (AU) and on the direct application of Scherer's CPT. The process is developed in two steps:
- Recognition of Ekaman's AUs from real time video
- Recognition of Scherer's sequential evaluation checks (SEC) from Ekman's AUs
We have defined the interfaces between the two different models and we are developing the modules performing the recognition of the SECs from AUs. We will feed these modules with simulated data to demonstrate the feasability of the project.
We are exploring contributions with vision groups for the development of the first half of the work.
Top |
Multimodal Fusion of Affective Cues
Emotions are a multimodal phenomena that imply facial expression, voice tone, autonomous nervous system (ANS) signals (e.g. hear-beat and blood pressure), body posture, and others. While we can expect to have good results with a single modality on specific scenarios (with specific limitations), we cannot expect similar results in the open scenario including all possible human computer interactions.
Fuse the information coming from the different modalities can, and generally does, increase the system reliability, precision and recognition scores.
Multimodal Fusion of Affective Cues Paradigm [Paleari & Lisetti 2006]
Using Scherer's Component Process Theory (CPT) of emotions [Scherer 1982 - 2006] and basing on the user modeling of emotions we had previously defined, we have, at the Affective Social Computing Group (ASCG), defined a multimodal fusion paradigm [Paleari & Lisetti 2006c].
The new paradigm novelty stays not only on the underlying psychological model of emotions but also on the modular approach which easily allows to add new modalities and processing units to the system.
Furthermore the fusion system can, thanks to its particular design, offer more reliable and precise representations of the user affective states. The fusion system not only works recognizing emotions but also basic mood and some personality characteristics.
The system is designed to return two extimations: one branch return fast (near to real-time) estimations of current affective state by fusing affective information mainly coming from ANS, facial expression, voice (but other modalities are possible like body posture and can be easily integrated). A second branch elaborate the signals and tries to synchronize them to return a more reliable and precise extimation.
Top |
User Modeling
User Modeling of Emotions
Using Scherer's Component Process Theory (CPT) of emotions [Scherer 1982 - 2006] we have defined an user modeling of affective states (i..e which regards phenomena such emotions, moods and personality).
In particular we designed an user modeling of the emotions based on Scherer's CPT Sequential Evaluation Checks (SECs).
Such a kind of model allows to differentiate among all the possible emotion state felt by humans. Furthermore the model we designed will made available to Artificial Intelligence (AI) new possibilities in terms of sensory-motor, behavioral and cognitive reactions. These kind of reactions will be, in fact, biased by the simulated affective state of the AI agent.
Top |
User Modeling of Personality
We aim at creating agents with which people enjoy to interact with. It has been suggested that robots need to adopt social behavior such as to express their emotions or to respect social conventions in order to be integrated in human environment and improve human-robot interaction.
To give pleasantness in this kind of interaction, robot needs human-like reactions, behavior, social abilities. For example Heering and others [Heering et al. 2006] have studies the iCat acceptance by elderly users. They give characteristics traits to iCat in order to obtain a pleasant interaction. The result of this work is the well-acceptance of iCat among elderly people.
One of our current objective is to create a game interaction with iCat by giving it different personalities in order to evaluate the acceptance rate.
In psychological literature, there exist many definitions of personality. The Big Five Taxonomy regroups these definitions (John, 1999) and contains five dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism and Openness (OCEAN). We used the Big Five theory to create our different personalities that will be used during game interactions. Then, we will evaluate the iCat acceptance in function of its personalities (extravert, introvert, pleasant etc.).
We also would like to study the influence of human personality on iCat robot perception. We found some questionnaires that evaluate human personality like: Trait Descriptive Factors (TDA) or Big Five Inventory (BFI).
So we planned to use the UTAUT (Unified Theory of Acceptance and Use of Technology) questionnaire to evaluate the acceptance of the robot during a game interaction and BFI questionnaire to evaluate user personality. Then we will try to find a correlation between the results of these two questionnaire.
Top |
Synthesis of Dynamic Emotion Facial Expressions
During human face to face communications people exchange informations that are not linked to the chosen words. In particular it is argued that an important part of the meaning of the communications stays on the emotive channel [Besson et al. 2004, Mehrabian 1971,1972, Merola & Poggi 2004, Picard 1997].
One of the most important channel togheter with the voice tone is for sure the facial expression. It is generally argued [Picard 1997 et al.] that give computers the ability of showing and communicate emotions can be important in many application domains (e.g. e-tutoring, games, telemedicine etc.)
Basing on Scherer's [Scherer 1982 - 2006] component process theory (CPT) of emotions we generated new facial expressions [Paleari & Lisetti 2006, 2006b and Grizard & Lisetti 2006, 2006b] on two different platforms: (1) a robot, Cleo and (2) an avatar, Cherry.
Robotic Platform: Cleo
iCat is a social robotic platform developed by Philips for studying human-robot interaction. It communicates with interlocutors via simulated social patterns, for example the expression of some facial anf vocal emotions.
iCat is composed of a mechanical head and it is able to move its lips, lids, eyes, eyebrows and neck. It has a webcam in its nose, speaker and microphones in theirs paws, and touch sensors and lights (blue, red, green) in theirs paw and ears.
We have created 9 emotional facial expression [Grizard & Lisetti 2006] based on Scherer's [Scherer 2001] psychological theory. During the implementation, we have noticed that the created animations on iCat lack believability in comparison with Philips animations. Scherer's theory has been developed for human and not for a toy robot.
Furthermore, we noticed, Philips intensifies and exaggerates the iCat emotive expressions inspired by cartoon animations. We have designed our nine final emotional facial expressions on iCat robot (developed by Philips) basing on the combination of Scherer's psychological theory and cartoon animations (exaggeration in movements and use of lights in paws and ears).
Video of Cleo (iCat) showing some new facial expressionsWe have conducted user studies that confirm the believability and the recognition of our new designed expressions.
Top |
Software Platform: Cherry Avatar
Cherry is an embodied conversational agent we developed [Lisetti 2002] from the commercial haptek tool. It is animated through haptek hypertext, a scripting language designed to control the avatar by defining their positions and morphs (appereance) as well as their facial expressions and speech.
Through hypertext users can also trigger switches which are collection of states or facial expressions.
We have created in this way five different facial expressions [Paleari & Lisetti 2006] basing on Scherer's [Scherer 2001] psychological theory of emotions. During the development of these new natural and psychologically grounded expressions we encountered several problems and we have listed some open research questions. In particular we found different open issue regarding timing and intensities of the different Ekman's [Ekman & Friesel 1971] action units (AU).
We refined our animations by adapting the theory and in particular by finding possible answers to the open research questions that arised during the development. To do that we referred to a database containing videos of actors showing different facial expressions.
Video of Cherry showing some new facial expressionsWe have conducted user studies that confirmed the believability and the recocognizability of our new designed expressions. The facial expressions were recognized in over the 90% of the cases. Believability was rated in a similar way for the expressions we designed and for the existing facial expressions developed by haptek.
Top |
Emotion-Based Architectures for Socially Intelligent Agents
An Affective Architecture on Three Levels [Lisetti 2006]
We are studying mechanisms to model computationally affective social cognitive intelligence to enhance both agents’ autonomy, and human-computer or human-robot interaction. Our approach will involve the design of an affective-cognitive hybrid architecture which, building upon the PI’s previous research, will synthesize affective-cognitive states for artificial agents. We will study ways to combine the reactive behavioural approach with the deliberative planning BDI approach (of modelling cognition in terms of Beliefs, Desires, and Intentions).
Top |
Social Robots: Cherry
We consider the notion of social expertise in terms of (1) internal motivational goal-based abilities and (2) external communicative behavior. Because of the important functional role that emotions play in human decision-making and in human-human communication, we propose a paradigm for modeling some of the functions of emotions in intelligent autonomous artificial agents to enhance both (a) robots’ autonomy and (b) human-robot interaction.
To this end, we have developed an autonomous service robot whose functionality has been designed so that it could socially interact with humans on a daily basis in the context of an office suite environment and studied and evaluated the design in vivo.
The social robot is furthermore evaluated from a social informatics approach, using workplace ethnography to guide its design while it is being developed. From our perspective, an interesting modeling issue therefore becomes that of social relations. In particular, we have chosen to focus our contribution to the field in addressing the technical goals of (1) understanding how to embody affective social intelligence and (2) determining when embodied affective social intelligence is useful (or not).
he videos below show Cherry, our service robot performing various activities
|
|
|
| Cherry delivering papers to the front office of Computer Science | Cherry offering LifeSavers to students | Cherry greeting a professor appropriately |
Top |
E-FERET: An Online Database for Facial Expression Recognition
E-FERET is an online Facial Expression Image Database. In this database we stored facial images of many different individuals from different genders, ethnicities, age groups, etc. with different facial expressions on their faces.
When doing research on face/facial expression recognition it is crucial for the researchers to have a database of a lot of images to train and test their systems. So, we thought E-FERET might help the researchers who need images of different facial expressions. Please follow the link below to dowload images from E-FERET. You can also upload your images to our system to help us have a bigger database.
Top |
Web page created by Marco Paleari - Last updated 18th of January 2007