Description:
This invention introduces BP4D+, a large scale multimodal spontaneous emotion dataset integrating synchronized 3D facial dynamics, 2D video, thermal imaging, and physiological signals. Designed for real world affective computing, it enables accurate training and benchmarking of AI systems that recognize authentic human emotion.
Background:
Emotion recognition research has been constrained by the reliance on posed expressions and limited single modality datasets. Such datasets fail to capture the complexity, spontaneity, and subtlety of genuine human emotions, as well as the physiological and thermal cues integral to affective states. Existing corpora are often small, lack demographic diversity, and provide incomplete or inconsistent annotations, impeding the development of robust, generalizable emotion analysis algorithms. A large scale, multimodal dataset capturing natural emotional responses with detailed expert annotations is essential to enable accurate emotion recognition in healthcare, security, human computer interaction, and behavioral analysis applications.
Technology Overview:
The BP4D+ dataset is a multimodal spontaneous emotion corpus comprising synchronized high resolution 3D dynamic facial geometry, 2D texture video, thermal facial imaging, and physiological signals including heart rate, blood pressure, respiration rate, and skin conductivity collected from 140 ethnically diverse participants. Ten structured emotion elicitation tasks were used to provoke genuine affective expressions across multiple emotional states. Each dataset instance includes expert Facial Action Coding System (FACS) annotations for 34 action units occurrence and intensity, tracked 3D, 2D, and thermal facial landmarks, and head pose data. This corpus provides a unique, ecologically valid foundation for developing and benchmarking algorithms in emotion recognition, affective computing, and behavioral science.
Advantages:
• Synchronized multimodal data integrating 3D facial dynamics, thermal imaging, and physiological signals
• Authentic spontaneous emotion capture enhances ecological validity over posed datasets
• Comprehensive FACS coding of 34 action units with occurrence and intensity measures
• Large and ethnically diverse participant base enables robust, generalizable AI model training
• High resolution 3D and thermal tracking provides detailed spatiotemporal emotion dynamics
• Enables benchmarking of multimodal emotion recognition algorithms under realistic conditions
Applications:
• Development of multimodal AI and emotion recognition systems
• Mental health and neurological assessment tools
• Driver and operator monitoring for safety critical systems
• Behavioral research and consumer emotion analytics
• Emotionally intelligent virtual assistants, robots, and avatars
• Security and surveillance systems detecting stress or aggression
Intellectual Property Summary:
• Z. Zhang et al. Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis 2016 IEEE Conference on Computer Vision and Pattern Recognition CVPR Las Vegas NV USA 2016 pp. 3438 to 3446 doi 10.1109/CVPR.2016.374
Stage of Development:
Algorithm and Dataset
Licensing Status:
This technology is available for licensing.
Licensing Potential:
Highly relevant to AI developers, digital health companies, emotion aware computing platforms, and safety monitoring system providers seeking validated multimodal datasets for robust emotion recognition.
Additional Information:
Information available upon request.
Inventors:
Lijun Yin