Subject-Adaptive Multimodal Fusion for Robust Affect Detection: Bridging the Gap Between Laboratory Baselines and Real-World Physiological Noise Using CNN-LSTM With Attention Mechanism

Ikegwuonu-vic Uzoamaka Hope; Nlerum Promise Anebo

doi:10.56201/ijcsmt.vol.12.no3.2026.pg139.159

📄 Download Paper

Subject-Adaptive Multimodal Fusion for Robust Affect Detection: Bridging the Gap Between Laboratory Baselines and Real-World Physiological Noise Using CNN-LSTM With Attention Mechanism

Ikegwuonu-vic Uzoamaka Hope, Nlerum Promise Anebo

Abstract

Wearable physiological signal analysis offers transformative potential for mental health monitoring, yet existing affect detection systems demonstrate limited generalization across individuals due to inter-subject variability and environmental noise. This research addresses the critical gap between controlled laboratory performance and real-world applicability by developing a subject-adaptive multimodal fusion framework. We propose a novel hybrid CNN-LSTM architecture with integrated cross-modal attention mechanisms that dynamically weights contributions from six physiological modalities: electrocardiogram, electrodermal activity, respiratory, electromyogram, temperature, and accelerometer signals. Employing rigorous Leave-One-Subject-Out validation on the WESAD dataset (n=15), our framework achieves 96.2% global accuracy and 98.6% AUC-ROC in binary stress detection, representing a 3-7% improvement over existing benchmarks. Notably, eight subjects achieved perfect classification (AUC=1.000), while all others exceeded 90.5% accuracy. The attention mechanism provides interpretable feature weighting, identifying cardiac (42%) and respiratory (28%) signals as primary stress indicators while adapting to individual response patterns. These findings establish a new state-of-the-art for multimodal affect detection with enhanced generalization capabilities. The framework's practical significance lies in its potential for developing personalized mental health monitoring systems that maintain accuracy across diverse populations while offering explainable insights into physiological stress responses, thereby advancing toward clinically deployable wearable technologies.

Keywords

Affective ComputingMultimodal FusionSubject AdaptationAttention MechanismsCNN-LSTM ArchitecturePhysiological Signal ProcessingWearable Stress DetectionPersonalized HealthcareExplainabl

References

Aqajari, S. A. H., Labbaf, S., Tran, P. H., Nguyen, B., Mehrabadi, M. A., Levorato, M., Dutt, N., & Rahmani, A. M. (2023). Context-aware stress monitoring using wearable and mobile technologies in everyday settings. medRxiv. https://doi.org/10.1101/2023.04.20.23288181 Awais, M. A., Ward, T., Redmond, P., & Healy, G. (2024). From lab to life: Assessing the impact of real-world interactions on the operation of rapid serial visual presentation-based brain-computer interfaces. Journal of Neural Engineering, 21(4), Article 046011. https://doi.org/10.1088/1741-2552/ad5d17 Barker, D., Tippireddy, M. K. R., Farhan, A., & Ahmed, B. (2025). Ethical Considerations in Emotion Recognition Research. Psychology International, 7(2), 43. https://doi.org/10.3390/psycholint7020043 Can, Y. S., & Ersoy, C. (2021). Privacy-preserving federated deep learning for wearable IoT-based biomedical monitoring. ACM Transactions on Internet Technology, 21(1), Article 21. https://doi.org/10.1145/3428152 Chen, C., Fang, H., Yang, Y., & Zhou, Y. (2025). Model-agnostic meta-learning for EEG- based inter-subject emotion recognition. Journal of Neural Engineering, 22(1), Article 016008. https://doi.org/10.1088/1741-2552/ad9956 Cheng, Y., Wu, C., Cai, P., Zhong, Z., Huang, H., & Jiang, Y. (2023). Tiny-PPG: A lightweight deep neural network for real-time detection of motion artifacts in photoplethysmogram signals on edge devices (arXiv:2305.03308). arXiv. https://doi.org/10.48550/arXiv.2305.03308 D. Yang et al., 'Asynchronous Multimodal Video Sequence Fusion via Learning Modality- Exclusive and -Agnostic Representations,' in IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 12, pp. 12360-12375, Dec. 2024, doi: 10.1109/TCSVT.2024.3435561. Ding, C., Yao, T., Wu, C., & Ni, J. (2024). Advances in deep learning for personalized ECG diagnostics: A systematic review addressing inter-patient variability and generalization constraints. Biosensors and Bioelectronics, 266, Article 117073. https://doi.org/10.1016/j.bios.2024.117073 https://www.ihealthcareanalyst.com/report/wearable-medical-devices-market/ J. Li, Y. Xia, J. Lei, R. J. Cooper and H. Zhao, 'Real-Time Motion Artifact Removal in fNIRS with Denoising Autoencoder at the Edge,' 2025 IEEE International Symposium on Circuits and Systems , London, United Kingdom, 2025, pp. 1-5, doi: 10.1109/ISCAS56072.2025.11044035. L. Bondin and A. Dingli, 'A Federated Affective Computing Framework To Learn From Small Data,' 2021 International Conference on Computational Science and Computational Intelligence , Las Vegas, NV, USA, 2021, pp. 14- 19, doi: 10.1109/CSCI54926.2021.00009. Lian, Z., Liu, R., Xu, K., Liu, B., Liu, X., Zhang, Y., Liu, X., Li, Y., Cheng, Z., Zuo, H., Ma, M., Peng, X., Chen, X., Li, Y., Cambria, E., Zhao, G., Schuller, B. W., & Tao, J. (2025). MER 2025: When affective computing meets large language models (arXiv:2504.19423). arXiv. https://doi.org/10.48550/arXiv.2504.19423 Pei, G., Li, H., Lu, Y., Wang, Y., & Li, T. (2024). Affective computing: Recent advances, challenges, and future trends. Intelligent Computing, 3, Article 0076. https://doi.org/10.34133/icomputing.0076 R. W. Picard, E. Vyzas and J. Healey, 'Toward machine emotional intelligence: analysis of affective physiological state,' in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001, doi: 10.1109/34.954607. Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., & Van Laerhoven, K. (2018). Introducing WESAD, a multimodal dataset for wearable stress and affect detection. Proceedings of the 2018 ACM International Conference on Multimodal Interaction, 400-408. https://doi.org/10.1145/3242969.3242985 Stahlschmidt, S., Fußer, G., & Schwenker, F. (2022). Deep learning fusion strategies for emotion recognition based on physiological signals: A review. Information Fusion, 79, 1–21. https://doi.org/10.1016/j.inffus.2021.09.018 Sun, Z., Zhang, W., Zhou, Y., Geng, S., Zhang, D., Wang, J., Liu, B., Fu, Z., Zheng, L., Jiang, C., Zhang, G., & Hong, S. (2025). A lightweight deep neural network for personalized detecting ventricular arrhythmias from a single-lead ECG device. PLOS Digital Health, 4(10), Article e0001037. https://doi.org/10.1371/journal.pdig.0001037 Thakur, P., Kaur, N., Aggarwal, N., & Singh, S. (2025). A Comprehensive Review of Unimodal and Multimodal Emotion Detection: Datasets, Approaches, and Limitations. Expert Systems, 42. Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., & Zhang, W. (2022). A systematic review on affective computing: Emotion models, databases, and recent advances (arXiv:2203.06935). arXiv. https://doi.org/10.48550/arXiv.2203.06935 Xie, W., Fang, Y., Yang, G., Yu, K., & Li, W. (2023). Transformer-Based Multi- Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification. Biomolecules, 13(9), 1391. https://doi.org/10.3390/biom13091391 Yang, D., Kuang, H., Huang, S., & Zhang, L. (2022). Learning modality-specific and - agnostic representations for asynchronous multimodal language sequences. In Proceedings of the 30th ACM International Conference on Multimedia (MM '22) (pp. 1708–1717). Association for Computing Machinery. https://doi.org/10.1145/3503161.3547755 Zhang, T., Lian, C., Xu, B., Su, Y., & Zeng, Z. (2024). Cardiac signals classification via optional multimodal multiscale receptive fields CNN-enhanced Transformer. Knowledge-Based Systems, 299, Article 112175. https://doi.org/10.1016/j.knosys.2024.112175

Subject-Adaptive Multimodal Fusion for Robust Affect Detection: Bridging the Gap Between Laboratory Baselines and Real-World Physiological Noise Using CNN-LSTM With Attention Mechanism

Abstract

Keywords

References

More Articles from INTERNATIONAL JOURNAL OF COMPUTER SCIENCE AND MATHEMATICAL THEORY