Convolutional Neural Network-Based Facial Expression Recognition: Enhanced by Data Augmentation and Transfer Learning

Authors

  • HMLS Kumari Computer center, Faculty of Engineering, University of Peradeniya, Sri Lanka

Keywords:

Convolutional Neural Network (CNN), Data Augmentation, Facial Expression Recognition (FER), fine tuning, Transfer learning model

Abstract

Facial expression recognition has emerged as a dynamic field within computer vision and human-computer interaction, finding diverse applications such as animation, social robots, personalized banking, and more. Current studies employ transfer learning models in facial expression recognition through the application of convolutional neural networks. The proposed model combines data augmentation with fine-tunned transfer learning models to get a better FER model. A comprehensive collection of training images is crucial as input to effectively train a convolutional neural network (CNN) for accurate facial expression recognition. Hence, the presented research employed data augmentation to enhance the quantity of input images derived from a pre-existing dataset. Manually employing CNN is outdated. Therefore, fine-tuned transfer learning models are used in the proposed study. Activating the final 8 layers of the transfer learning model by freezing the whole transfer learning model is the novel methodology of the proposed model. Then we vary the values of dense layers and dropout layers of the activated 8 layers, which results the fine-tuning of the transfer learning model. The CK+, JAFFE and FER2013 datasets are used in the proposed model. Subsequently, conduct a stratified 5-fold cross-validation to assess the model's performance on previously unseen data and avoid overfitting the proposed model. The method under consideration utilized transfer learning models, namely DenseNet121, DenseNet201, DenseNet169, and InceptionV3, along with fine-tuned transfer learning models applied to augmented datasets CK+, JAFFE and FER2013 datasets. The outcomes indicate an achievement of 99.36% accuracy for the CK+ dataset, 95.14% for the facial recognition dataset (Human).

References

“University of Glasgow,” The expression

oftheemotionsinmanandanimals,https://www.gla.ac.uk/myglasgow/library/files/special/exhibns/month/nov2009.html (accessed Nov. 24, 2023).

Li, S., & Deng, W. (2022). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3), 1195–1215. https://doi.org/10.1109/taffc.2020.2981446

Fathallah, A., Abdi, L., & Douik, A. (2017). Facial expression recognition via deep learning. 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA). https://doi.org/10.1109/aiccsa.2017.124

Jia, S., Wang, S., Hu, C., Webster, P. J., & Li, X. (2021). Detection of genuine and posed facial expressions of emotion: Databases and methods. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.580287

Handbook of Face Recognition.(2011) https://doi.org/10.1007/978-0-85729-932-1

M. A. Akhand, S. Roy, N. Siddique, M. A. Kamal, and T. Shimamura, “Facial emotion recognition using transfer learning in the deep CNN,” Electronics, vol. 10, no. 9, p. 1036, 2021. doi:10.3390/electronics10091036

T. U. Ahmed, S. Hossain, M. S. Hossain, R. ul Islam, and K. Andersson, “Facial expression recognition using convolutional neural network with data augmentation,” 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR), 2019. doi:10.1109/iciev.2019.8858529

S. Alizadeh and A. Fazel, “Convolutional neural networks for facial expression recognition,” [1704.06756] Convolutional Neural Networks for Facial Expression Recognition, http://export.arxiv.org/abs/1704.06756 (accessed Nov. 24, 2023).

S. Porcu, A. Floris, and L. Atzori, “Evaluation of data augmentation techniques for facial expression recognition systems,” Electronics, vol. 9, no. 11, p. 1892, 2020. doi:10.3390/electronics9111892

A. Ravi, “Pre-trained convolutional neural network features for facial expression recognition,” arXiv.org, https://arxiv.org/abs/1812.06387 (accessed Nov. 27, 2023).

S. Porcu, A. Floris, and L. Atzori, “Evaluation of data augmentation techniques for facial expression recognition systems,” Electronics, vol. 9, no. 11, p. 1892, 2020. doi:10.3390/electronics9111892.

N. Darapaneni et al., “Facial expression recognition and recommendations using deep neural network with transfer learning,” 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), 2020. doi:10.1109/uemcon51285.2020.9298082

Md. Z. Uddin, W. Khaksar, and J. Torresen, “Facial expression recognition using salient features and convolutional neural network,” IEEE Access, vol. 5, pp. 26146–26161, 2017. doi:10.1109/access.2017.2777003

A. T. Lopes, E. de Aguiar, A. F. De Souza, and T. Oliveira-Santos, “Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order,” Pattern Recognition, vol. 61, pp. 610–628, 2017. doi:10.1016/j.patcog.2016.07.026

C. Pramerdorfer and M. Kampel, “Facial expression recognition using convolutional neural networks: State of the art,” arXiv.org, https://arxiv.org/abs/1612.02903v1 (accessed Nov. 24, 2023).

I. Talegaonkar, K. Joshi, S. Valunj, R. Kohok, and A. Kulkarni, “Real time facial expression recognition using deep learning,” SSRN Electronic Journal, 2019. doi:10.2139/ssrn.3421486

Papers with code - CK+ dataset. CK+ Dataset | Papers With Code. (n.d.). https://paperswithcode.com/dataset/ck

Khan, Z. (2023, November 24). Facial recognition dataset (human). Kaggle. https://www.kaggle.com/datasets/zawarkhan69/human-facial-expression-dataset

Awati, R. (2023, April 24). What are convolutional neural networks?: Definition from TechTarget. Enterprise AI. https://www.techtarget.com/searchenterpriseai/definition/convolutional-neural-network

X. Wang, K. Wang, and S. Lian, “A survey on Face data augmentation for the training of Deep Neural Networks,” Neural Computing and Applications, vol. 32, no. 19, pp. 15503–15531, 2020. doi:10.1007/s00521-020-04748-3

Stanford University CS231N: Deep Learning for Computer Vision, http://cs231n.stanford.edu/reports/2016/pdfs/023_Report.pdf (accessed Nov. 24, 2023).

C. Pramerdorfer and M. Kampel, “Facial expression recognition using convolutional neural networks: State of the art,” arXiv.org, https://arxiv.org/abs/1612.02903v1 (accessed Nov. 24, 2023).

Accuracy vs. precision vs. recall in machine learning: What’s the difference? Evidently AI - Open-Source ML Monitoring and Observability. (n.d.). https://www.evidentlyai.com/classification-metrics/accuracyprecisionrecall#:~:text=Accuracy%20shows%20how%20often%20a,objects%20of%20the%20target%20class.

Mage.ai. (n.d.). https://www.mage.ai/blog/definitive-guide-to-accuracy-precision-recall-for-product-developers

A. Ahmed, “Architecture of densenet-121,” OpenGenus IQ: Computing Expertise & Legacy, https://iq.opengenus.org/architecture-of-densenet121/ (accessed Nov. 24, 2023).

G. Singhal, “Gaurav Singhal,” Pluralsight, https://www.pluralsight.com/guides/introduction-to-densenet-with-tensorflow (accessed Nov. 20, 2023).

V. Kurama, “A guide to resnet, inception V3, and squeezenet,” Paperspace Blog, https://blog.paperspace.com/popular-deep-learning-architectures-resnet-inceptionv3-squeezenet/ (accessed Nov. 24, 2023)

Ramalingam, S., & Garzia, F. (2018). Facial expression recognition using transfer learning. 2018 International Carnahan Conference on Security Technology (ICCST). https://doi.org/10.1109/ccst.2018.8585504

Randellini, E., Rigutini, L., & Saccà, C. (2021). Data Augmentation and transfer learning approaches applied to facial expressions recognition. NLP Techniques and Applications. https://doi.org/10.5121/csit.2021.111912

Darapaneni, N., Choubey, R., Salvi, P., Pathak, A., Suryavanshi, S., & Paduri, A. R. (2020). Facial expression recognition and recommendations using deep neural network with transfer learning. 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON). https://doi.org/10.1109/uemcon51285.2020.9298082

Ahmed, T. U., Hossain, S., Hossain, M. S., ul Islam, R., & Andersson, K. (2019). Facial expression recognition using convolutional neural network with data augmentation. 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). https://doi.org/10.1109/iciev.2019.8858529

Hrga, I., & Ivasic-Kos, M. (2022). Effect of data augmentation methods on face image classification results. Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods. https://doi.org/10.5220/0010883800003122

Published

01/17/2024

How to Cite

Kumari, H. (2024). Convolutional Neural Network-Based Facial Expression Recognition: Enhanced by Data Augmentation and Transfer Learning. International Journal of Research in Computing, 2(2), 42–49. Retrieved from http://ijrcom.org/index.php/ijrc/article/view/127