Visuo-Auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction
Project Type: PhD Project
Research Field: Bayesian Multimodal Perception
Sponsors: Fundação para a Ciência e a Tecnologia PhD Scholarship – SFRH/BD/60954/2009
Time span: 03/2010-11/2012

This project intends to research Bayesian models of emotions that deal with fusion, multimodality (auditory and visual sensing) and emotional imitation learning and also to provide an application that allows the robot to interact with the human with emotions.

Recently, we proposed, implemented and show results about two novel Bayesian classifiers. These classifiers have both the purpose to classify human emotional state among the scope {anger, fear, sad, neutral and happy}. The first difference between them is clearly the channel of communication, in other words, one of them uses the auditory channel (listen to human voice) and the other uses the visual channel (camera looking to the human face). The second difference is of course the model, since the channels are different, the detection and also the variables are diverse. Thus, each classifier analysis it's respective input signal during a certain period of time and gives a result among the mentioned scope. Since the objective is to use the fusion of these classifiers during a story-board conversation between human and machine; for the auditory classifier this certain period of time is defined by the duration of each spoken phrase. Considering that is difficult to speak and perform recognizable facial expression at the same time, after the speech some seconds are given for the human to perform the desired facial expression that will be related to that phrase.

Then we have a spoken phrase and two classifications among the same scope. In the previous work [reference to the journal not submitted yet] we proposed a fusion using a third Bayesian network where the two classifications outputs entered as likelihoods of this third network with the same weight, and also a personality random variable was added to define 3 different types of response learning possibilities {Empathic, Antipathetic and Humoristic}.

Future work is to explore other possibilities of fusion, like the Bayesian Mixture Models fusion where the confidence of each model can be given by the results of the hit-hate tests already implemented and achieved.

Moreover, here we are interested on improving the learning scope of the reactions of the robot, specially exploring more the humoristic personality; also to perform more experiments and to define an assessment for measuring the humor personality of the robot.

Related Projects

BACS - Bayesian Approach to Cognitive Systems

Related People

Jorge Dias
José Prado