Facial expression recognition of masked faces using transfer learning
Denezhkina Lidia
Tongji University
Luo Ye
Tongji University
DOI: https://doi.org/10.59429/esta.v12i3.11362
Keywords: facial emotion recognition; occluded facial emotion recognition; face masks, transfer learning; convolutional neural network
Abstract
Human emotions, reflected through facial expressions, provide valuable insight into an individual's state of mind. Automatic facial emotion recognition (FER) has made significant progress. However, the recognizing emotions on occluded faces remains an area that requires further exploration. We propose a method to address the challenges of partial occlusion, that is, to recognize emotions on faces where the lower part is obscured by a mask. We tackle the classical FER task by employing a transfer learning approach: first, training a teacher model on a dataset of unmasked faces, and then fine-tuning a student model with the learned weights on a dataset of masked faces. Our model integrates a combination of convolutional layers, Inception blocks, Residual blocks, and Squeeze-and-Excitation (SE) networks. We evaluate the proposed approach on three datasets: JAFFE, KDEF, and Oulu-CASIA, achieving accuracy rates of 80.00%, 78.57%, and 89.38%, respectively.
References
[1] P. Ekman, Pictures of facial affect, Consulting Psychologists Press, 1976.
[2] C. Szegedy, W. Liu, Y. Jia, et al, Going deeper with convolutions, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, pp. 1-9, 2015.
[3] K. He, X. Zhang, S. Ren, et al, Deep residual learning for image recognition, Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
[4] M. J. Lyons, "Excavating AI"re-excavated: debunking a fallacious ccount of the JAFFE dataset, 2021.
[5] E. Goeleven, R. De, L. Rudi, et al, The Karolinska Directed Emotional Faces: A validation study, Cognition and emotion, vol. 22, pp. 1094-1118, 2008.
[6] G. Zhao, X. Huang, M. Taini, et al, Facialexpression recognition from near-infrared videos, Image and Vision Computing, vol. 29, issue 9, pp. 607-619, 2011.
[7] A. Anwar, A. Raychowdhury, Masked face recognition for secure authentication, 2020.
[8] C. Jiang, M. R. Hasan, T. Gedeon, et al, MaskTheFER: mask-aware facial expression recognition using convolutional neural network, 2023 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Port Macquarie, Australia, pp. 456-463, 2023.
[9] S. Tegani, T. Abdelmoutia, Using COVID-19 masks dataset to implement deep convolutional neural networks for facial emotion recognition, 2021 4th International Symposium on Advanced Electrical and Communication Technologies (ISAECT), Alkhobar, Saudi Arabia, pp. 1-5, 2021.