Journal: Volume 30, No. 2, 2025
Pages: 22 – 32
DOI: https://doi.org/10.62660/bcstu/2.2025.22
957 Views

Integration of multimodal biometric data into an information system for comprehensive analysis of emotional states based on electroencephalograms

Vladyslav Holoborodko, Volodymyr Lyfar
Received 09.02.2025
Revised 08.05.2025
Accepted 16.06.2025

Abstract

The purpose of this study was to establish quantitative parameters for emotional state detection through comprehensive analysis of electroencephalography (EEG) signal characteristics integrated with synchronised biometric indicators, enhancing recognition accuracy. The study examined the incorporation of multimodal biometric data into an information system for an extensive analysis of emotional states derived from EEG. A unique methodology integrating input from various sensory systems was offered to improve the precision of emotion recognition. The analysis focused on EEG processing techniques and the integration of data from additional biometric channels, including facial expressions, heart rate, and galvanic skin response. The algorithmic and technological facets of system creation were examined, alongside experimental study findings that validated its efficacy. Special emphasis was placed on the system’s flexibility to diverse operational situations. The algorithmic and technological aspects of system development were analysed alongside the results of experimental studies that confirmed the efficiency of such systems. Multimodal emotion recognition systems held significant potential for application in various fields, particularly in the evaluation of mental health, the diagnosis of emotional states, adaptive education, and the creation of systems for advanced human-machine interaction. Special attention was given to the adaptability of these systems to diverse operational contexts and their capacity to integrate diverse biometric modalities. These integrations enhanced the robustness and reliability of emotion detection. Additionally, advancements in machine learning, particularly deep neural networks, facilitated the comprehensive analysis and synchronisation of multimodal data, enabling these systems to capture the nuances of emotional states with high precision

Keywords

affective computing; physiological signals; neural network analysis; biosignal processing; human-computer interaction

References

  1. Alarcão, S.M., & Fonseca, M.J. (2019). Emotions recognition using EEG signals: A survey. IEEE Transactions on Affective Computing, 10(3), 374-393. doi: 10.1109/TAFFC.2017.2714671.
  2. Alharbi, B., & Alshanbari, H.S. (2023). Face-voice based multimodal biometric authentication system via FaceNet and GMM. PeerJ Computer Science, 9, article number e1468. doi: 10.7717/peerj-cs.1468.
  3. Balogun, M.O., Odeniyi, L.A., Omidiora, E.O., Olabiyisi, S.O., & Falohun, A.S. (2023). Optimized negative selection algorithm for image classification in multimodal biometric system. Acta Informatica Pragensia, 12(1), 3-18. doi: 10.18267/j.aip.186.
  4. Coelho, K.K., Tristao, E.T., Nogueira, M., Vieira, A.B., & Nacif, J.A.M. (2023). Multimodal biometric authentication method by federated learning. Biomedical Signal Processing and Control, 85, article number 105022. doi: 10.1016/j.bspc.2023.105022.
  5. Declaration of Helsinki. (2013). Retrieved from https://www.wma.net/policies-post/wma-declaration-ofhelsinki/.
  6. Halim, N., Fuad, N., Marwan, M., & Nasir, E. (2022). Emotion state recognition using band power of EEG signals. In Proceedings of the 6th international conference on electrical, control and computer engineering: InECCE2021 (pp. 939-950). Singapore: Springer. doi: 10.1007/978-981-16-8690-0_82.
  7. Ipeayeda, F.W., Oyediran, M.O., Ajagbe, S.A., Jooda, J.O., & Adigun, M.O. (2023). Optimized gravitational search algorithm for feature fusion in a multimodal biometric system. Results in Engineering, 20, article number 101572. doi: 10.1016/j.rineng.2023.101572.
  8. Kazi, M., Kale, K., Mehsen, R.S., Mane, A., Humbe, V., Rode, Y., Dabhade, S., Bansod, N., Razvi, A., & Deshmukh, P. (2023). Face, fingerprint, and signature based multimodal biometric system using score level and decision level fusion approaches. IETE Journal of Research, 70(4), 3703-3722. doi: 10.1080/03772063.2023.2217784.
  9. Kumar, P., Saini, R., Kaur, B., Roy, P.P., & Scheme, E. (2019). Fusion of neuro-signals and dynamic signatures for person authentication. Sensors, 19(21), article number 4641. doi: 10.3390/s19214641.
  10. Li, W., Zhang, Z., & Song, A. (2021). EEG-based emotion recognition: An Odyssey from methodology to philosophy. Measurement, 172, article number 108747. doi: 10.1016/j.measurement.2020.108747.
  11. Mishra, A.R., et al. (2023). SignEEG v1.0: Multimodal electroencephalography and signature database for biometric systems. Scientific Data, 11, article number 718. doi: 10.1038/s41597-024-03546-z.
  12. Rodrigues, J.D.C., Filho, P.P.R., Damaeviius, R., & Albuquerque, V.H.C. (2020). EEG-based biometric systems. In Neurotechnology: Methods, advances and applications (pp. 97-153). London: The Institution of Engineering and Technology. doi: 10.1049/PBHE019E_ch5.
  13. Safavipour, M.H., Doostari, M.A., & Sadjedi, H. (2023). Deep hybrid multimodal biometric recognition system based on features-level deep fusion of five biometric traits. Computational Intelligence and Neuroscience, 2023, article number 6443786. doi: 10.1155/2023/6443786.
  14. Saini, R., Kaur, B., & Arora, P. (2018). Don’t just sign, use brain too: A novel multimodal approach for user identification and verification. Information Sciences, 430-431, 163-178. doi: 10.1016/j.ins.2017.11.045.
  15. Salama, G.M., El-Gazar, S., Omar, B., & Hassan, A. (2023). Multimodal cancelable biometric authentication system based on EEG signal for IoT applications. Journal of Optics, 53, 1839-1853. doi: 10.1007/s12596-02301302-x.
  16. Singh, S.P., & Tiwari, S. (2023). A dual multimodal biometric authentication system based on WOA-ANN and SSA-DBN techniques. Sci, 5(1), article number 10. doi: 10.3390/sci5010010.
  17. Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C.F. (2021). A multimodal emotion recognition method based on facial expressions and electroencephalography. Biomedical Signal Processing and Control, 70, article number 103029. doi: 10.1016/j.bspc.2021.103029.
  18. Tang, H., Liu, W., Zheng, W.-L., & Lu, B.-L. (2017). Multimodal emotion recognition using deep neural networks. In International conference on neural information processing (pp. 811-819). Cham: Springer. doi: 10.1007/978-3-319-70093-9_86.
  19. Toa, C.K., Sim, K.S., & Tan, S.C. (2021). Emotiv insight with convolutional neural network: Visual attention test classification. In Advances in computational collective intelligence: 13th international conference, ICCCI 2021 (pp. 348-357). Cham: Springer. doi: 10.1007/978-3-030-88113-9_28.
  20. Wang, Y., Yang, X., Li, J., Yang, C., Zhao, H., & Yin, Z. (2022). A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion, 83-84, 19-52. doi: 10.1016/j.inffus.2022.03.009.
  21. Yu, J., Li, C., Lou, K., Wei, C., & Liu, Q. (2022). Embedding decomposition for artifacts removal in EEG signals. Journal of Neural Engineering, 19, article number 026052. doi: 10.1088/1741-2552/ac63eb.
  22. Zabcikova, M. (2019). Visual and auditory stimuli response, measured by Emotiv Insight headset. MATEC Web of Conferences, 292, article number 01024. doi: 10.1051/matecconf/2019292010.
  23. Zhang, H., Zhao, M., Wei, C., Mantini, D., Li, Z., & Liu, Q. (2021). EEGdenoisenet: A benchmark dataset for deep learning solutions of EEG denoising. Journal of Neural Engineering, 18, article number 056057. doi: 10.1088/17412552/ac2bf8.

Suggested citation

Holoborodko, V., & Lyfar, V. (2025). Integration of multimodal biometric data into an information system for comprehensive analysis of emotional states based on electroencephalograms. Bulletin of Cherkasy State Technological University, 30(2), 22-32. https://doi.org/10.62660/bcstu/2.2025.22