A computer system that can recognise the emotional state of the
person communicating with it, and could then be taught to respond
accordingly, has been developed.
The system is hoped will make the communication between
computers and humans more intuitive by allowing the machine to
respond to the emotional cues it is picking up from the user and
then tailoring its responses to them. For example, during a
tutorial, if the user is getting frustrated or confused, the
computer can simplify its responses.
The project was as a result of a collaboration between computer
scientists at the Universidad
Carlos III de Madrid (UC3M) and the Universidad de Granada. They
created the system to recognise negative emotions such as anger, boredom and doubt. To "read"
these, the computer system studied the tone of voice of the person
talking to it, the speed of their speech, the energy of the voice
signal and the duration of pauses, among a total of 60 "acoustic
The system continues to monitor speech to see if the user's
emotional state changed. As the team explains in a release marking the publication of the paper in the Journal on Advances in
Signal Processing: "…If the system did not correctly
recognise what the interlocutor wanted to say several times, or if
it asked the user to repeat information that s/he had already
given, these factors could anger or bore the user when s/he was
interacting with the system."
The system takes its analysis of these dialogues and will use
these to predict the likely next actions of the user, says David
Grill, a professor in UC3M's Computer Science Department. "To that
end, we have developed a statistical method that uses earlier
dialogues to learn what actions the user is most likely to take at
any given moment," the researchers explain.
The next step is for the responses of the computer system to be
tailored to the mood and intentions of the user. As the team explains: "…if
they have doubts, more detailed help can be offered, whereas if
they are bored, such an offer could be counterproductive." The
success of these responses was in objective terms, namely whether
the computer's response produced shorter and more successful
dialogues between man and machine.