IRO Journals
Journal of Innovative Image Processing

Interpretable Deep Multimodal Fusion of ECG and Facial Expressions for Robust Emotion Detection

Open Access
Published: 17 October, 2026
Pages: 1532-1556
  • Sankar Ganesh Karuppasamy , Sankar Ganesh Karuppasamy
    Assistant Professor, Department of Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai, India
    Assistant Professor, Department of Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai, India
  • Rajeev Kumar M. Rajeev Kumar M.
    Professor, Department of Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai, India
    Professor, Department of Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai, India
view PDF
view PDF

How to Cite

Karuppasamy, Sankar Ganesh, and Rajeev Kumar M. 2026. “Interpretable Deep Multimodal Fusion of ECG and Facial Expressions for Robust Emotion Detection”. Journal of Innovative Image Processing 8 (4): 1532-56. https://doi.org/10.36548/jiip.2026.4.010.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

Emotion Recognition
Electrocardiogram (ECG)
Facial Expression Analysis
Cross-Attention Mechanism
Multimodal Fusion
SHAP
Grad-CAM

Abstract

In healthcare, emotional computing, and dynamic human-computer interaction, emotion recognition is a key component of human-centric Artificial Intelligence (AI). However, existing approaches depend on either physiological or visual signals and often show limited robustness or accuracy in noisy or inadequate data environments. This study proposes a task-specific bidirectional cross-modal fusion framework in which physiological ECG representations and facial-expression representations mutually attend to each other, while an auxiliary embedding-alignment objective constrains the two modalities to a shared latent space. While the facial images are processed using EfficientNet-B0 to extract discriminative visual features, the ECG signals are transformed using a one-dimensional Convolutional Neural Network (1D-CNN) in conjunction with a Bidirectional Long Short-Term Memory (Bi-LSTM) network to extract local morphological features and long-range temporal inference. A cross-attention fusion technique is then used to fuse the trained ECG models with facial expression data. The inter-modal dependencies between physiological and behavioral signals are represented by a cross-attention fusion module, which improves the model's dependability and broadens its capacity to identify a variety of emotional classes, specifically focusing on six fundamental emotions: happy, sad, angry, fearful, surprised, and neutral. The acquired representation is classified using a fully connected attention-based decoder. Model interpretability is attained through SHAP analysis of ECG embeddings and Grad-CAM visualization of the facial areas. When compared to current and baseline fusion techniques, the evaluation conducted using the MAHNOB-HCI and AMIGOS datasets demonstrated that the proposed model achieved higher performance, including an accuracy of 97%, precision of 96%, recall of 97%, F1-score of 96%, and improved adaptability.

References

  1. Setyawan, Hendy, Rifki Wijaya, and Gamma Kosala. ”Development of an Emotion Detection System Based on Heartrate Using Support Vector Machine.” In 2024 12th International Conference on Information and Communication Technology (ICoICT), IEEE, 2024, 79-85.
  2. Hasna, T., N. Sindhu, and K. Deepika. ”Enhancing Anxiety Detection Through ECG-Derived HRV Features and Machine Learning.” In 2025 6th International Conference on Control, Communication and Computing (ICCC), IEEE, 2025, 1-6.
  3. Mokhtari, Bilal, Abdelhak Merizig, and Hadjer Zerdoumi. “Enhanced Emotion Recognition in an IoMT Platform: Leveraging Data Augmentation and the Random Forest Algorithm for ECG-based E-health.” International Journal of Information Technology 18, no. 1 (2026): 599-609.
  4. Kumar, Puneet, Shreshtha Misra, Zhuhong Shao, Bin Zhu, Balasubramanian Raman, and Xiaobai Li. “Multimodal Interpretable Depression Analysis Using Visual, Physiological, Audio and Textual Data.” In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), IEEE, 2025, 5305-5315.
  5. Kim, Ji Woong, Aisha Urooj Khan, and Imon Banerjee. “Systematic Review of Hybrid Vision Transformer Architectures for Radiological Image Analysis.” Journal of Imaging Informatics in Medicine 38, no. 5 (2025): 3248-3262.
  6. Wang, Xinyu, Le Sun, Chuhan Lu, and Baozhu Li. “A Novel Transformer Network with a CNN-enhanced Cross-Attention Mechanism for Hyperspectral Image Classification.” Remote Sensing 16, no. 7 (2024): 1180.
  7. Tsai, Chih Mao, and Jiann-Der Lee. “Dynamic Ensemble Learning with Gradient-Weighted Class Activation Mapping for Enhanced Gastrointestinal Disease Classification.” Electronics 14, no. 2 (2025): 305.
  8. Wang, Ling, Jiayu Hao, and Tie Hua Zhou. “ECG Multi-Emotion Recognition Based on Heart Rate Variability Signal Features Mining.” Sensors 23, no. 20 (2023): 8636.
  9. Fang, Ancheng, Fan Pan, Weichuang Yu, Linkun Yang, and Peiyu He. “ECG-based Emotion Recognition Using Random Convolutional Kernel Method.” Biomedical Signal Processing and Control 91 (2024): 105907.
  10. Sutar, Manisha Balkrishna, and Asha Ambhaikar. “A Comparative Study on Deep Facial Expression Recognition.” In 2023 7th International Conference on Intelligent Computing and Control Systems (ICICCS), IEEE, 2023, 903-911.
  11. Karnati, Mohan, Ayan Seal, Debotosh Bhattacharjee, Anis Yazidi, and Ondrej Krejcar. “Understanding Deep Learning Techniques for Recognition of Human Emotions Using Facial Expressions: A Comprehensive Survey.” IEEE Transactions on Instrumentation and Measurement 72 (2023): 1-31.
  12. Bhatlawande, Shripad, Swati Shilaskar, Sourjadip Pramanik, and Swarali Sole. “Multimodal Emotion Recognition Based on the Fusion of Vision, EEG, ECG, and EMG Signals.” International journal of electrical and computer engineering systems 15, no. 1 (2024): 41-58.
  13. Jia, M., and Z. Sun. “A Survey of Multi-Modal Emotion Recognition Based on Deep Learning.” Highlights in Science, Engineering and Technology 119 (2024): 533-540.
  14. Li, Jixiang, and Jianxin Peng. “End-to-End Multimodal Emotion Recognition Based on Facial Expressions and Remote Photoplethysmography Signals.” IEEE Journal of Biomedical and Health Informatics 28, no. 10 (2024): 6054-6063.
  15. Fan, Tianqi, Sen Qiu, Zhelong Wang, Hongyu Zhao, Junhan Jiang, Yongzhen Wang, Junnan Xu, Tao Sun, and Nan Jiang. “A New Deep Convolutional Neural Network Incorporating Attentional Mechanisms for ECG Emotion Recognition.” Computers in Biology and Medicine 159 (2023): 106938.
  16. Soleymani, Mohammad, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. “A Multimodal Database for Affect Recognition and Implicit Tagging.” IEEE transactions on affective computing 3, no. 1 (2011): 42-55.
  17. Miranda-Correa, Juan Abdon, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. “Amigos: A Dataset for Affect, Personality and Mood Research on Individuals and Groups.” IEEE transactions on affective computing 12, no. 2 (2018): 479-493.
  18. Septian, Belen. “Noise suppression of ECG Signal Using Optimized Digital Butterworth Bandpass Filter.” The Indonesian Journal of Computer Science 13, no. 4 (2024).
  19. Ahmed, Anas Fouad. “A Novel Improved Pan-Tompkins Method for QRS Complex Detection of the ECG Signal.” Signal, Image and Video Processing 19, no. 12 (2025): 960.
  20. Yun, Vincent-Daniel. “Sharpness-Aware Minimization with Z-Score Gradient Filtering.” In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2026, 2961-2965.
  21. Kumar, Gaurav, and Neeraj Varshney. “Hybrid deep-CNN and Bi-LSTM Model with Attention Mechanism for Enhanced ECG-Based Heart Disease Diagnosis.” Physical and engineering sciences in medicine 48, no. 4 (2025): 2075-2085.
  22. Härtinger, Philipp, and Carsten Steger. “Adaptive Histogram Equalization in Constant Time.” Journal of Real-Time Image Processing 21, no. 3 (2024): 93.
  23. Son, Gye-Jin, Hee-Cheol Jung, and Young-Dae Kim. “Temporal-Quality Ensemble Technique for Handling Image Blur in Packaging Defect Inspection.” Sensors, vol. 24, no. 14, 2024, Art. no. 4438.