IRO Journals
Journal of Electronics and Informatics

A Multimodal Framework for Real-time Cognitive Load and Stress Detection Using Facial Expressions and GSR Signals

Open Access
Published: 29 September, 2026
Pages: 330-344
  • Sakthivel V. , Sakthivel V.
    Assistant Professor, Computer Science and Engineering, Chennai Institute of Technology, Chennai, India
    Assistant Professor, Computer Science and Engineering, Chennai Institute of Technology, Chennai, India
  • Suresh Joseph K. Suresh Joseph K.
    Professor, Computer Science and Engineering, Pondicherry University, Pondicherry, India
    Professor, Computer Science and Engineering, Pondicherry University, Pondicherry, India
view PDF
view PDF

How to Cite

V., Sakthivel, and Suresh Joseph K. 2026. “A Multimodal Framework for Real-Time Cognitive Load and Stress Detection Using Facial Expressions and GSR Signals”. Journal of Electronics and Informatics 8 (4): 330-44. https://doi.org/10.36548/jei.2026.4.002.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

Electrodermal Activity
Facial Behavioural Analysis
MediaPipe Facial Landmarks
Temporal Feature Synchronization
Multimodal Feature Fusion
Psychophysiological Stress Classification
Real-time Stress Assessment

Abstract

Stress detection in real-time requires the combination of both physiological and behavioral modalities because the use of a single modality provides limited information on the psychophysiological state of individuals. In this study, a multimodal model is proposed, wherein the physiological modality uses EDA, while the behavioral modality uses facial expression analysis. SCL and SCR features are extracted from GSR signals recorded with a sensing module developed using the ESP32 board. Simultaneously, EAR, blink rate, gaze, and head pose features are extracted from webcam video using the MediaPipe library. The features from the two modalities are time-stamped for alignment and extracted within 30 seconds' window. Then, the features are combined in the feature-level fusion process. The classification of stress levels is done using RF and XGBoost classifiers as low, medium, and high levels of stress. The results of this experiment show that multimodal fusion gives an accuracy of 91.6% compared to features from either one of the modalities.

References

  1. Koelstra, Sander, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. "Deap: A Database for Emotion Analysis; using Physiological Signals." IEEE Transactions on Affective Computing, 2011, vol. 3, no. 1: 18-31.
  2. Soleymani, Mohammad, Sadjad Asghari-Esfeden, Yun Fu, and Maja Pantic. "Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection." IEEE Transactions on Affective Computing, 2015, vol. 7, no. 1: 17-28.
  3. Baltrušaitis, Tadas, Peter Robinson, and Louis-Philippe Morency. "Openface: an Open Source Facial Behavior Analysis Toolkit." In Winter Conference on Applications of Computer Vision (WACV), IEEE, 2016, 1-10.
  4. Kahou, Samira Ebrahimi, Christopher Pal, Xavier Bouthillier, Pierre Froumenty, Çaglar Gülçehre, Roland Memisevic, Pascal Vincent et al. "Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video." In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 2013, 543-550.
  5. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep Learning." Nature, 2015, vol. 521, no. 7553: 436-444.
  6. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "ImageNet Classification with Deep Convolutional Neural Networks." Communications of the ACM, 2017, vol. 60, no. 6: 84-90.
  7. Graves, Alex. "Supervised Sequence Labelling." In Supervised Sequence Labelling with Recurrent Neural Networks, Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, 5-13.
  8. McDuff, Daniel, Rana Kaliouby, Thibaud Senechal, May Amr, Jeffrey Cohn, and Rosalind Picard. "Affectiva-mit Facial Expression Dataset (am-fed): Naturalistic and Spontaneous Facial Expressions Collected." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2013, 881-888.
  9. Gunes, Hatice, and Björn Schuller. "Categorical and Dimensional Affect Analysis in Continuous Input: Current Trends and Future Directions." Image and Vision Computing, 2013, vol. 31, no. 2: 120-136.
  10. Li, Shan, and Weihong Deng. "Deep Facial Expression Recognition: A Survey." IEEE Transactions on Affective Computing, 2020, vol. 13, no. 3: 1195-1215.
  11. Gjoreski, Martin, Hristijan Gjoreski, Mitja Luštrek, and Matjaž Gams. "Continuous Stress Detection using a Wrist Device: in Laboratory and Real life." In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, 2016, 1185-1193.
  12. Sharma, Nandita, and Tom Gedeon. "Objective Measures, Sensors and Computational Techniques for Stress Recognition and Classification: A Survey." Computer Methods and Programs in Biomedicine, 2012, vol. 108, no. 3: 1287-1301.
  13. Yin, Zhong, Mengyuan Zhao, Yongxiong Wang, Jingdong Yang, and Jianhua Zhang. "Recognition of Emotions using Multimodal Physiological Signals and an Ensemble Deep Learning Model." Computer Methods and Programs in Biomedicine, 2017, vol. 140: 93-110.
  14. Ashish, Vaswani. "Attention is All You Need." Advances in Neural Information Processing Systems, 2017.
  15. Sabour, Rita Meziati, Yannick Benezeth, Pierre De Oliveira, Julien Chappe, and Fan Yang. "Ubfc-Phys: A Multimodal Database for Psychophysiological Studies of Social Stress." IEEE Transactions on Affective Computing 2021, vol. 14, no. 1: 622-636.
  16. WESAD (Wearable Stress and Affect Detection) Dataset: https://archive.ics.uci.edu/dataset/465/wesad