Negative Emotions Recognition While Driving Using Electroencephalogram Signal

Document Type : Research Paper

Authors

1 Industrial Engineering School, K. N. Toosi University of Technology, Tehran, Iran

2 Industrial Engineering school, K. N. Toosi University of Technology, Tehran, Iran

Abstract

The role of the human factor has been confirmed as the number one cause of driving crashes and emotions are known as one of the most important factors of driver distraction. Although biological signals have a great potential for detecting emotions, so far few studies have been conducted to use these signals to develop emotion recognition systems while driving. Therefore, in this paper an electroencephalography (EEG) based classification model presents for recognizing low-valence high-arousal (LVHA) emotions (known as negative emotions) of drivers. For this purpose, two driving tests were designed in a driving simulator, one for driving under normal conditions and the other for driving in the negative emotional state. 18 people participated in these tests and the activity of four channels of their brain signals was recorded during the tests. The energies of delta, theta, alpha, beta, and gamma frequency bands, and the total signal energy along with gender were employed as inputs for classification models and emotional state was considered as output. Different models were used for subject-independent classification, among which the neural network classifier with an accuracy of 95% had the best performance. The results of the analysis showed that all channels are effective in increasing the accuracy of classifiers; also, gender has a relative impact on the accuracy of classification models. Assessing the effects of different frequency bands revealed that alpha and gamma bands have a greater effect on the accuracy of models than do other bands. At the end, different combinations of EEG channels were used to recognize negative emotions while driving, and the results indicated that using only two channels can help recognize these emotions with an accuracy of 89%. 

Keywords

References

- Abou Elassad, Z. E., Mousannif, H., Al Moatassime, H. and Karkouch, A. (2020) “The application of machine learning techniques for driving behavior analysis: A conceptual framework and a systematic literature review”, Engineering Applications of Artificial Intelligence, Vol.  87, No., pp. 103312.
 
- Agrafioti, F., Hatzinakos, D. and Anderson, A. K. (2012) “ECG pattern analysis for emotion detection”, IEEE Transactions on Affective Computing, Vol.  3, No. 1, pp. 102-15.
 
- Al Machot, F., Mosa, A. H., Dabbour, K., Fasih, A., Schwarzlmüller, C., Ali, M. and Kyamakya, K. (2011) “A novel real-time emotion detection system from audio streams based on bayesian quadratic discriminate classifier for adas”, Paper presented at the Proceedings of the Joint INDS'11 & ISTET'11.
 
- Ali, M., Al Machot, F., Mosa, A. H. and Kyamakya, K. (2016). ”Cnn based subject-independent driver emotion recognition system involving physiological signals for adas” Advanced Microsystems for Automotive Applications 2016 (pp. 125-38): Springer.
 
- Ali, M., Al Machot, F., Haj Mosa, A., Jdeed, M., Al Machot, E. and Kyamakya, K. (2018) “A globally generalized emotion recognition system involving different physiological signals”, Sensors, Vol.  18, No. 6, pp. 1905.
 
- Ali, Y., Sharma, A., Haque, M. M., Zheng, Z. and Saifuzzaman, M. (2020) “The impact of the connected environment on driving behavior and safety: A driving simulator study”, Accident analysis & prevention, Vol.  144, No., pp. 105643.
- Alkinani, M. H., Khan, W. Z. and Arshad, Q. (2020) “Detecting human driver inattentive and aggressive driving behavior using deep learning: Recent advances, requirements and open challenges”, IEEE Access, Vol.  8, No., pp. 105008-30.
 
- Asadamraji, M., Saffarzadeh, M. and Mirzaee Tayeghani, M. (2017) “Modeling Driver’s Hazard Perception using Driver's Personality Characteristics”, International Journal of Transportation Engineering, Vol.  5, No. 2, pp. 167-82. doi:10.22119/ijte.2017.46520
 
- Asadamraji, M., Saffarzadeh, M., Ross, V., Borujerdian, A., Ferdosi, T. and Sheikholeslami, S. (2019) “A novel driver hazard perception sensitivity model based on drivers’ characteristics: A simulator study”, Traffic injury prevention, Vol.  20, No. 5, pp. 492-7.
 
- Atkinson, J., & Campos, D. (2016) “Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers”, Expert Systems with Applications, Vol.  47, No., pp. 35-41.
 
- Azman, A., Raman, K. J., Mhlanga, I. A. J., Ibrahim, S. Z., Yogarayan, S., Abdullah, M. F. A., Razak, S. F. A., Amin, A. H. M. and Muthu, K. S. (2018) “Real Time Driver Anger Detection”, Paper presented at the International Conference on Information Science and Applications.
 
- Basu, S., Jana, N., Bag, A., Mahadevappa, M., Mukherjee, J., Kumar, S. and Guha, R. (2015) “Emotion recognition based on physiological signals using valence-arousal model”, Paper presented at the 2015 Third International Conference on Image Information Processing (ICIIP).
 
- Bradley, M. M., & Lang, P. J. Affective norms for English words (ANEW): Instruction manual and affective ratings. Citeseer; 1999.
 
- Braun, M., Schubert, J., Pfleging, B. and Alt, F. (2019) “Improving driver emotions with affective strategies”, Multimodal Technologies and Interaction, Vol.  3, No. 1, pp. 21.
 
- Bustamante, E. A., & Spain, R. D. (2008) “Measurement invariance of the Nasa TLX”, Paper presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting.
 
- Cai, H., & Lin, Y. (2011) “Modeling of operators' emotion and task performance in a virtual driving environment”, International Journal of Human-Computer Studies, Vol.  69, No. 9, pp. 571-86.
 
- Cao, A., Chintamani, K. K., Pandya, A. K. and Ellis, R. D. (2009) “NASA TLX: Software for assessing subjective mental workload”, Behavior research methods, Vol.  41, No. 1, pp. 113-7.
- Casson, A. J., Yates, D. C., Smith, S. J., Duncan, J. S. and Rodriguez-Villegas, E. (2010) “Wearable electroencephalography”, IEEE engineering in medicine and biology magazine, Vol.  29, No. 3, pp. 44-56.
 
- Chen, L.-l., Zhang, A. and Lou, X.-g. (2019) “Cross-subject driver status detection from physiological signals based on hybrid feature selection and transfer learning”, Expert Systems with Applications, Vol.  137, No., pp. 266-80.
 
- Cheng, B., & Liu, G. (2008) “Emotion recognition from surface EMG signal using wavelet transform and neural network”, Paper presented at the Proceedings of the 2nd international conference on bioinformatics and biomedical engineering (ICBBE).
 
- Cunningham, M. L., & Regan, M. A. (2016) “The impact of emotion, life stress and mental health issues on driving performance and safety”, Road & Transport Research: A Journal of Australian and New Zealand Research and Practice, Vol.  25, No. 3, pp. 40.
 
- Davidson, R. J. (1992) “Anterior cerebral asymmetry and the nature of emotion”, Brain and cognition, Vol.  20, No. 1, pp. 125-51.
 
- Davidson, R. J., & Fox, N. A. (1982) “Asymmetrical brain activity discriminates between positive and negative affective stimuli in human infants”, Science, Vol.  218, No. 4578, pp. 1235-7.
 
- Ekman, P. (1992) “An argument for basic emotions”, Cognition & emotion, Vol.  6, No. 3-4, pp. 169-200.
 
- Fuller, R. (2000) “The task-capability interface model of the driving process”, Recherche-Transports-Sécurité, Vol.  66, No. 1, pp. 47-57.
 
- Fuller, R. (2005) “Towards a general theory of driver behaviour”, Accident analysis & prevention, Vol.  37, No. 3, pp. 461-72.
 
- Ganesh, S., Chinchani, A. M., Bhushan, A., Kanchan, D. and Kubakaddi, S. (2017) “Participant-dependent and participant-independent classification of emotions using EEG signals”, Paper presented at the 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET).
 
- García, H. F., Álvarez, M. A. and Orozco, Á. A. (2016) “Gaussian process dynamical models for multimodal affect recognition”, Paper presented at the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).
 
- Gómez-Herrero, G., De Clercq, W., Anwar, H., Kara, O., Egiazarian, K., Van Huffel, S. and Van Paesschen, W. (2006) “Automatic removal of ocular artifacts in the EEG without an EOG reference channel”, Paper presented at the Proceedings of the 7th Nordic Signal Processing Symposium-NORSIG 2006.
 
- Gong, P., Ma, H. T. and Wang, Y. (2016) “Emotion recognition based on the multiple physiological signals”, Paper presented at the 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR).
 
- Gorunescu, F. (2011). ”Data Mining: Concepts, models and techniques”, Springer Science & Business Media.
 
- Grier, R. A. (2015) “How high is high? A meta-analysis of NASA-TLX global workload scores”, Paper presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting.
 
- Guendil, Z., Lachiri, Z., Maaoui, C. and Pruski, A. (2015) “Emotion recognition from physiological signals using fusion of wavelet based features”, Paper presented at the 2015 7th International Conference on Modelling, Identification and Control (ICMIC).
 
- Guo, H.-W., Huang, Y.-S., Lin, C.-H., Chien, J.-C., Haraikawa, K. and Shieh, J.-S. (2016) “Heart rate variability signal features for emotion recognition by using principal component analysis and support vectors machine”, Paper presented at the 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE).
- Han, J., Pei, J. and Kamber, M. (2011). ”Data mining: concepts and techniques”, Elsevier.
 
- Hart, S. (1988). Development of NASA-TLX: Results of empirical and theoretical research.” inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload: North-Holland.
 
- Hart, S. G. (2006) “NASA-task load index (NASA-TLX); 20 years later”, Paper presented at the Proceedings of the human factors and ergonomics society annual meeting.
 
- Hayley, A. C., de Ridder, B., Stough, C., Ford, T. C. and Downey, L. A. (2017) “Emotional intelligence and risky driving behaviour in adults”, Transportation research part F: traffic psychology and behaviour, Vol.  49, No. 1, pp. 124-31.
 
- Ille, N., Berg, P. and Scherg, M. (2002) “Artifact correction of the ongoing EEG using spatial filters based on artifact and brain signal topographies”, Journal of clinical neurophysiology, Vol.  19, No. 2, pp. 113-24.
 
- Izquierdo-Reyes, J., Ramirez-Mendoza, R. A., Bustamante-Bello, M. R., Navarro-Tuch, S. and Avila-Vazquez, R. (2018) “Advanced driver monitoring for assistance system (ADMAS)”, International Journal on Interactive Design and Manufacturing (IJIDeM), Vol.  12, No. 1, pp. 187-97.
 
- Jeon, M. (2016) “Don’t cry while you’re driving: sad driving is as bad as angry driving”, International Journal of Human–Computer Interaction, Vol.  32, No. 10, pp. 777-90.
 
- Jeon, M., & Walker, B. N. (2011) “What to detect? Analyzing factor structures of affect in driving contexts for an emotion detection and regulation system”, Paper presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting.
 
- Jeon, M., Yim, J.-B. and Walker, B. N. (2011) “An angry driver is not the same as a fearful driver: effects of specific negative emotions on risk perception, driving performance, and workload”, Paper presented at the Proceedings of the 3rd international conference on automotive user interfaces and interactive vehicular applications.
 
- Jeon, M., Walker, B. N. and Yim, J.-B. (2014) “Effects of specific emotions on subjective judgment, driving performance, and perceived workload”, Transportation research part F: traffic psychology and behaviour, Vol.  24, No. 1, pp. 197-209.
 
- Jerritta, S., Murugappan, M., Nagarajan, R. and Wan, K. (2011) “Physiological signals based human emotion recognition: a review”, Paper presented at the 2011 IEEE 7th International Colloquium on Signal Processing and its Applications.
 
- Karimi, S., Aghabayk, K., Abrari Vajari, M. and Stephens, A. (2021) “Aggressive Driving: Self-Reported Anger Expression and Its Relationship with Driver Personality”, International Journal of Transportation Engineering, Vol.  8, No. 3, pp. 299-316. doi:10.22119/ijte.2021.254078.1541
 
- Katsis, C. D., Katertsidis, N., Ganiatsas, G. and Fotiadis, D. I. (2008) “Toward emotion recognition in car-racing drivers: A biosignal processing approach”, IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, Vol.  38, No. 3, pp. 502-12.
- Kaye, S.-A., Lewis, I. and Freeman, J. (2018) “Comparison of self-report and objective measures of driving behavior and road safety: A systematic review”, Journal of safety research, Vol.  65, No., pp. 141-51.
 
- khalifeh, v., Nadimi, N., Khoshdel Sangdeh, A. and Mohammadian Amiri, A. (2021) “Evaluation of the effect of driving education and training programs on modification of driver's dangerous behaviors”, International Journal of Transportation Engineering, Vol., No., pp. -. doi:10.22119/ijte.2021.237613.1523
 
- Kim, J. (2007). ”Bimodal emotion recognition using speech and physiological changes” Robust speech recognition and understanding: IntechOpen.
 
- Kim, J., & André, E. (2008) “Emotion recognition based on physiological changes in music listening”, IEEE transactions on pattern analysis and machine intelligence, Vol.  30, No. 12, pp. 2067-83.
 
- Kolodyazhniy, V., Kreibig, S. D., Gross, J. J., Roth, W. T. and Wilhelm, F. H. (2011) “An affective computing approach to physiological emotion specificity: Toward subject‐independent and stimulus‐independent classification of film‐induced emotions”, Psychophysiology, Vol.  48, No. 7, pp. 908-22.
 
- Koornstra, M. J. (2009) “Risk-adaptation theory”, Transportation research part F: traffic psychology and behaviour, Vol.  12, No. 1, pp. 77-90.
 
- Lan, Z., Sourina, O., Wang, L. and Liu, Y. (2016) “Real-time EEG-based emotion monitoring using stable features”, The Visual Computer, Vol.  32, No. 3, pp. 347-58.
 
- Lang, P. J., Bradley, M. M. and Cuthbert, B. N. (1997) “International affective picture system (IAPS): Technical manual and affective ratings”, NIMH Center for the Study of Emotion and Attention, Vol.  1, No. 1, pp. 39-58.
 
- Li, X., Song, D., Zhang, P., Yu, G., Hou, Y. and Hu, B. (2016) “Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network”, Paper presented at the 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).
 
- Lin, Y.-P., Wang, C.-H., Jung, T.-P., Wu, T.-L., Jeng, S.-K., Duann, J.-R. and Chen, J.-H. (2010) “EEG-based emotion recognition in music listening”, IEEE Transactions on Biomedical Engineering, Vol.  57, No. 7, pp. 1798-806.
 
- Lisetti, C. L., & Nasoz, F. (2004) “Using noninvasive wearable computers to recognize human emotions from physiological signals”, EURASIP journal on applied signal processing, Vol.  11, No., pp. 1672-87.
 
- Liu, J., Meng, H., Nandi, A. and Li, M. (2016) “Emotion detection from EEG recordings”, Paper presented at the 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).
 
- Maaoui, C., & Pruski, A. (2010). ”Emotion recognition through physiological signals for human-machine communication” Cutting Edge Robotics 2010: IntechOpen.
 
- Mantouka, E., Barmpounakis, E., Vlahogianni, E. and Golias, J. (2020) “Smartphone sensing for understanding driving behavior: Current practice and challenges”, International Journal of Transportation Science and Technology, Vol., No.
 
- Masood, N., & Farooq, H. (2019) “Investigating EEG patterns for dual-stimuli induced human fear emotional state”, Sensors, Vol.  19, No. 3, pp. 522.
 
- Mihajlović, V., Grundlehner, B., Vullers, R. and Penders, J. (2015) “Wearable, wireless EEG solutions in daily life applications: what are we missing?”, IEEE journal of biomedical and health informatics, Vol.  19, No. 1, pp. 6-21.
 
- Minhad, K. N., Ali, S. H. M. and Reaz, M. B. I. (2017) “Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness”, Journal of Transport & Health, Vol.  7, No. 1, pp. 75-89.
 
- Mirmohamadsadeghi, L., Yazdani, A. and Vesin, J.-M. (2016) “Using cardio-respiratory signals to recognize emotions elicited by watching music video clips”, Paper presented at the 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP).
 
- Monajati, M., Abbasi, S. H., Shabaninia, F. and Shamekhi, S. (2012) “Emotions states recognition based on physiological parameters by employing of fuzzy-adaptive resonance theory”, International Journal of Intelligence Science, Vol.  2, No. 4, pp. 166.
 
- Murugappan, M. (2014) “Frequency band localization on multiple physiological signals for human emotion classification using DWT”, Emotion Recognition: A Pattern Analysis Approach, Vol., No., pp. 295-313.
 
- Petrantonakis, P. C., & Hadjileontiadis, L. J. (2010) “Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis”, IEEE Transactions on Affective Computing, Vol.  1, No. 2, pp. 81-97.
 
- Samara, A., Menezes, M. L. R. and Galway, L. (2016) “Feature extraction for emotion recognition and modelling using neurophysiological data”, Paper presented at the 2016 15th International Conference on Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS).
- Schmidt-Daffy, M. (2013) “Fear and anxiety while driving: Differential impact of task demands, speed and motivation”, Transportation research part F: traffic psychology and behaviour, Vol.  16, No. 1, pp. 14-28.
 
- Sheikholeslami, S., Boroujerdian, A. M. and Asadamraji, M. (2020) “A rural road accident probability model based on single-vehicle hazard properties including hazard color and mobility: a driving simulator study”, Journal of advanced transportation, Vol.  2020, No.
 
- Shin, D., Shin, D. and Shin, D. (2017) “Development of emotion recognition interface using complex EEG/ECG bio-signal for interactive contents”, Multimedia Tools and Applications, Vol.  76, No. 9, pp. 11449-70.
 
- Soleymani, M., Villaro-Dixon, F., Pun, T. and Chanel, G. (2017) “Toolbox for Emotional feAture extraction from Physiological signals (TEAP)”, Frontiers in ICT, Vol.  4, No. 1, pp. 1.
 
- Taylor, D. (1964) “DRIVERS'GALVANIC SKIN RESPONSE AND THE RISK OF ACCIDENT”, Ergonomics, Vol.  7, No. 4, pp. 439-51.
 
- Valenza, G., Lanata, A. and Scilingo, E. P. (2012) “The role of nonlinear dynamics in affective valence and arousal recognition”, IEEE Transactions on Affective Computing, Vol.  3, No. 2, pp. 237-49.
- Valenza, G., Citi, L., Lanatá, A., Scilingo, E. P. and Barbieri, R. (2014) “Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics”, Scientific reports, Vol.  4, No. 1, pp. 4998.
 
- Vilaca, A., Cunha, P. and Ferreira, A. L. (2017) “Systematic literature review on driving behavior”, Paper presented at the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).
- Vlakveld, W., Doumen, M. and van der Kint, S. (2021) “Driving and gaze behavior while texting when the smartphone is placed in a mount: A simulator study”, Transportation research part F: traffic psychology and behaviour, Vol.  76, No., pp. 26-37.
 
- Wan, P., Wu, C. and Ma, X. (2014) “A study of Chinese professional drivers' electroencephalogram characteristics under angry driving based on field experiments”, Paper presented at the 14th COTA International Conference of Transportation Professionals, Changsha, July.
 
- Wang, F., Zhong, S.-h., Peng, J., Jiang, J. and Liu, Y. (2018) “Data augmentation for eeg-based emotion recognition with deep convolutional neural networks”, Paper presented at the International Conference on Multimedia Modeling.
 
- Wen, W., Liu, G., Cheng, N., Wei, J., Shangguan, P. and Huang, W. (2014) “Emotion recognition based on multi-variant correlation of physiological signals”, IEEE Transactions on Affective Computing, Vol.  5, No. 2, pp. 126-40.
 
- Wong, W. M., Tan, A. W., Loo, C. K. and Liew, W. S. (2010) “PSO optimization of synergetic neural classifier for multichannel emotion recognition”, Paper presented at the 2010 Second World Congress on Nature and Biologically Inspired Computing (NaBIC).
 
- Wu, C.-K., Chung, P.-C. and Wang, C.-J. (2012) “Representative segment-based emotion analysis and classification with automatic respiration signal segmentation”, IEEE Transactions on Affective Computing, Vol.  3, No. 4, pp. 482-95.
 
- Yerkes, R. M., & Dodson, J. D. (1908) “The relation of strength of stimulus to rapidity of habit‐formation”, Journal of comparative neurology and psychology, Vol.  18, No. 5, pp. 459-82.
- Zhang, J., Chen, M., Hu, S., Cao, Y. and Kozma, R. (2016) “PNN for EEG-based Emotion Recognition”, Paper presented at the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).
 
- Zheng, W.-L., & Lu, B.-L. (2015) “Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks”, IEEE Transactions on Autonomous Mental Development, Vol.  7, No. 3, pp. 162-75.
 
- Zheng, W.-L., Zhu, J.-Y. and Lu, B.-L. (2017) “Identifying stable patterns over time for emotion recognition from EEG”, IEEE Transactions on Affective Computing, Vol.  10, No. 3, pp. 417-29.
 
- Zheng, W. (2016) “Multichannel EEG-based emotion recognition via group sparse canonical correlation analysis”, IEEE Transactions on Cognitive and Developmental Systems, Vol.  9, No. 3, pp. 281-90.
 
- Zong, C., & Chetouani, M. (2009) “Hilbert-Huang transform based physiological signals analysis for emotion recognition”, Paper presented at the 2009 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT).
 
- Zuckerman, M. (2007). ”Sensation seeking and risky behavior”, American Psychological Association.

Files

Share

How to cite

Statistics