Context in Human Emotion Perception for Automatic Affect Detection: A Survey of Audiovisual Databases

An important aspect of human emotion perception is the use of contextual information to understand others' feelings even in situations where their behavior is not very expressive or has an emotionally ambiguous meaning. For technology to successfully detect affect, it must mimic this human ability when analyzing audiovisual input. Databases upon...

Arousal and Valence prediction in spontaneous emotional speech: Felt versus perceived emotion

In this paper, we describe emotion recognition experiments carried out for spontaneous affective speech with the aim to compare the added value of annotation of felt emotion versus annotation of perceived emotion. Using speech material available in the TNO-GAMING corpus (a corpus containing audiovisual recordings of people playing videogames),...

Attuning speech-enabled interfaces to user and context for inclusive design: Technology, methodology and practice

This paper presents a methodology to apply speech technology for compensating sensory, motor, cognitive and affective usage difficulties. It distinguishes (1) an analysis of accessibility and technological issues for the identification of context-dependent user needs and corresponding opportunities to include speech in multimodal user interfaces...

How does real affect affect affect recognition in speech?

Multimodal subjectivity analysis of multiparty conversation

We investigate the combination of several sources of information for the purpose of subjectivity recognition and polarity classification in meetings. We focus on features from two modalities, transcribed words and acoustics, and we compare the performance of three different textual representations: words, characters, and phonemes. Our...

Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data

We investigated inter-observer agreement and the reliability of self-reported emotion ratings (i.e., self-raters judging their own emotions) in spontaneous multimodal emotion data. During a multiplayer video game, vocal and facial expressions were recorded (including the game content itself) and were annotated by the players themselves on...

Automatic recognition of spontaneous emotions in speech using acoustic and lexical features

We developed acoustic and lexical classifiers, based on a boosting algorithm, to assess the separability on arousal and valence dimensions in spontaneous emotional speech. The spontaneous emotional speech data was acquired by inviting subjects to play a first-person shooter video game. Our acoustic classifiers performed significantly better than...

An open-set detection evaluation methodology for automatic emotion recognition in speech

In this paper, we present a detection approach and an ‘open-set’ detection evaluation methodology for automatic emotion recognition in speech. The traditional classification approach does not seem to be suitable and flexible enough for typical emotion recognition tasks. For example, classification does not have an appropriate way to cope with ...

Affective multimodal mirror: Sensing and eliciting laughter

In this paper, we present a multimodal affective mirror that senses and elicits laughter. Currently, the mirror contains a vocal and a facial affect-sensing module, a component that fuses the output of these two modules to achieve a user-state assessment, a user state transition model, and a component to present audiovisual affective feedback...

An open-set detection evaluation methodology applied to language and emotion recognition

This paper introduces a detection methodology for recognition technologies in speech for which it is dif cult to obtain an abundance of non-target classes. An example is language recognition, where we would like to be able to measure the detection capability of a single target language without confounding with the modeling capability of non...

Inducing and measuring emotion through a multiplayer first-person shooter computer game

To develop an annotated database of spontaneous, multimodal, emotional expressions, recordings were made of facial and vocal expressions of emotions while participants were playing a multiplayer first-person shooter (fps) computer game. During a replay of the session, participants scored their own emotions by assigning values to them on an...

Visualizing acoustic similarities between emotions in speech: An acoustic map of emotions

In this paper, we introduce a visual analysis method to assess the discriminability and confusiability between emotions according to automatic emotion classifiers. The degree of acoustic similarities between emotions can be defined in terms of distances that are based on pair-wise emotion discrimination experiments. By employing Multidimensional...

Measuring cognitive task load on a naval ship: Implications of a real world environment

Application of more and more automation in process control shifts the operator’s task from manual to supervisory control. Increasing system autonomy, complexity and information fluctuations make it extremely difficult to develop static support concepts that cover all critical situations after implementing the system. Therefore, support systems...

Unobtrusive multimodal emotion detection in adaptive interfaces: Speech and facial expressions

Two unobtrusive modalities for automatic emotion recognition are discussed: speech and facial expressions. First, an overview is given of emotion recognition studies based on a combination of speech and facial expressions. We will identify difficulties concerning data collection, data fusion, system evaluation and emotion annotation that one is...

Evaluating automatic laughter segmentation in meetings using acoustic and acoustic-phonetic features

In this study, we investigated automatic laughter segmentation in meetings. We first performed laughterspeech discrimination experiments with traditional spectral features and subsequently used acousticphonetic features. In segmentation, we used Gaussian Mixture Models that were trained with spectral features. For the evaluation of the laughter...

Automatic detection of frequent pronunciation errors made by L2-learners

In this paper, we present an acoustic-phonetic approach to automatic pronunciation error detection. Classifiers using techniques such as Linear Discriminant Analysis and Decision Trees were developed for three sounds that are frequently pronounced incorrectly by L2-learners of Dutch: /a/, /y/ and /x/. This paper will focus mainly on the problems...

Automatic detection of laughter

In the context of detecting ‘paralinguistic events’ with the aim to make classification of the speaker’s emotional state possible, a detector was developed for one of the most obvious ‘paralinguistic events’, namely laughter. Gaussian Mixture Models were trained with Perceptual Linear Prediction features, pitch&energy, pitch&voicing and...