International Research Journal of Innovations in Engineering and Technology - IRJIET International Open Access, Monthly, Peer Reviewed, Reputed Journal ISSN (online): 2581-3048

Impact Factor (2025): 6.9

DOI Prefix: 10.47001/IRJIET

An In-Depth Review of Deep Facial Expression Recognition: Obstacles, Utilizations, and Prospective Directions

Abstract

Facial expression recognition (FER) is emerging as an emerging and multifaceted field of study. The use of FER in areas such as healthcare, security, and safe driving has not only enhanced the credibility of these technologies, but also their integration into human computer interaction to achieve intelligent outcomes. Computational FER seeks to replicate the skill of humans in decoding facial expressions, providing important cues that complement spoken language and aid listeners in understanding. Likewise, FER's deep learning (DL) and Artificial Intelligence (AI) methodologies are meticulously designed, incorporating advanced modules for efficiency and real time processing. In light of this background, many investigations have looked at different aspects of FER. Although current surveys focus primarily on traditional technologies and generic methodologies for on premises servers, they overlook the large field of deep learning inspired by edge vision and AI assisted FER technologies. To fill this gap, the current study conducts a comprehensive and thorough analysis of the prevailing FER literature. It carefully surveys the operational framework of FER technologies, highlighting their basic and intermediate phases, as well as the underlying pattern structures. Furthermore, the study addresses the limitations inherent in current FER surveys. The exploration extends to the FER datasets, subjecting them to thorough examination, thus revealing the attendant challenges and pitfalls. In addition, it provides a comprehensive discussion of the various metrics used to measure the effectiveness of FER methods.

Country : Iraq

1 Raed Ibrahim Khaleel Almsari2 Abbas Hussein Miry3 Tariq M. Salman

  1. MSC Student, Electrical Engineering Department, Al-Mustansiriyah University, Baghdad, Iraq
  2. Assistant Professor Dr., Electrical Engineering Department, Al-Mustansiriyah University, Baghdad, Iraq
  3. Assistant Professor Dr., Electrical Engineering Department, Al-Mustansiriyah University, Baghdad, Iraq

IRJIET, Volume 7, Issue 12, December 2023 pp. 96-103

doi.org/10.47001/IRJIET/2023.712014

Full Paper
Download

References

  1. J. Whitehill, Z. Serpell, Y. C. Lin, A. Foster, and J. R. Movellan, “The faces of engagement: Automatic recognition of student engagement from facial expressions,” IEEE Trans. Affect. Comput., vol. 5, no. 1, pp. 86–98, 2014, doi: 10.1109/TAFFC.2014.2316163.
  2. C. L. Kim and B. G. Kim, “Few-shot learning for facial expression recognition: a comprehensive survey,” J. Real-Time Image Process., vol. 20, no. 3, pp. 1–18, 2023, doi: 10.1007/s11554-023-01310-x.
  3. O. S. Ekundayo and S. Viriri, “Facial Expression Recognition: A Review of Trends and Techniques,” IEEE Access, vol. 9, pp. 136944–136973, 2021, doi: 10.1109/ACCESS.2021.3113464.
  4. S. Deshmukh, M. Patwardhan, and A. Mahajan, “Survey on real-time facial expression recognition techniques,” IET Biometrics, vol. 5, no. 3, pp. 155–163, 2016, doi: 10.1049/iet-bmt.2014.0104.
  5. A.Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,” IEEE Trans. Affect. Comput., vol. 10, no. 1, pp. 18–31, 2019, doi: 10.1109/TAFFC.2017.2740923.
  6. A.Dhall, O. V. Ramana Murthy, R. Goecke, J. Joshi, and T. Gedeon, “Video and image based Emotion recognition challenges in the wild: EmotiW 2015,” ICMI 2015 - Proc. 2015 ACM Int. Conf. Multimodal Interact., no. December, pp. 423–426, 2015, doi: 10.1145/2818346.2829994.
  7. A.Dhall, R. Goecke, S. Ghosh, J. Joshi, J. Hoey, and T. Gedeon, “From individual to group-level emotion recognition: Emoti W 5.0,” ICMI 2017 - Proc. 19th ACM Int. Conf. Multimodal Interact., vol. 2017-Janua, pp. 524–528, 2017, doi: 10.1145/3136755.3143004.
  8. I.J. Goodfellow et al., “Challenges in representation learning: A report on three machine learning contests,” Neural Networks, vol. 64, pp. 59–63, 2015, doi: 10.1016/j.neunet.2014.09.005.
  9. E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” ICMI 2016 - Proc. 18th ACM Int. Conf. Multimodal Interact., pp. 279–283, 2016, doi: 10.1145/2993148.2993165.
  10. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression,” 2010 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. - Work. CVPRW 2010, no. May 2014, pp. 94–101, 2010, doi: 10.1109/CVPRW.2010.5543262.
  11. S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” Proc. Natl. Acad. Sci. U. S. A., vol. 111, no. 15, pp. 1–9, 2014, doi: 10.1073/pnas.1322355111.
  12. C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “EmotioNet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-Decem, no. June 2016, pp. 5562–5570, 2016, doi: 10.1109/CVPR.2016.600.
  13. S. Li and W. Deng, “Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition,” IEEE Trans. Image Process., vol. 28, no. 1, pp. 356–370, 2019, doi: 10.1109/TIP.2018.2868382.
  14. A.Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Collecting large, richly annotated facial-expression databases from movies,” IEEE Multimed., vol. 19, no. 3, pp. 34–41, 2012, doi: 10.1109/MMUL.2012.26.
  15. P. Naga, S. Das Marri, and R. Borreo, “Facial emotion recognition methods, datasets and technologies: A literature survey,” Mater. Today Proc., vol. 80, no. xxxx, pp. 2824–2828, 2023, doi: 10.1016/j.matpr.2021.07.046.
  16. A.S. Aljoloud, H. Ullah, and A. A. Alanazi, “Facial emotion recognition using neighborhood features,” Int. J. Adv. Comput. Sci. Appl., vol. 11, no. 1, pp. 299–306, 2020, doi: 10.14569/ijacsa.2020.0110137.
  17. D. Yang, A. Alsadoon, P. W. C. Prasad, A. K. Singh, and A. Elchouemi, “An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment,” Procedia Comput. Sci., vol. 125, no. November 2022, pp. 2–10, 2018, doi: 10.1016/j.procs.2017.12.003.
  18. E. Dandıl and R. Özdemir, “Real-time Facial Emotion Classification Using Deep Learning,” Data Sci. Appl., vol. 2, no. 1, pp. 13–17, 2019.
  19. G. Tonguç and B. Ozaydın Ozkara, “Automatic recognition of student emotions from facial expressions during a lecture,” Comput. Educ., vol. 148, no. January, p. 103797, 2020, doi: 10.1016/j.compedu.2019.103797.
  20. M. A. Ozdemir, B. Elagoz, A. Alaybeyoglu, R. Sadighzadeh, and A. Akan, “Real time emotion recognition from facial expressions using CNN architecture,” TIPTEKNO 2019 - Tip Teknol. Kongresi, no. October, 2019, doi: 10.1109/TIPTEKNO.2019.8895215.
  21. A.Mahmood, S. Hussain, K. Iqbal, and W. S. Elkilani, “Recognition of Facial Expressions under Varying Conditions Using Dual-Feature Fusion,” Math. Probl. Eng., vol. 2019, p. 9185481, 2019, doi: 10.1155/2019/9185481.
  22. M. R. M. Shahane*, R. Sharma .K, and M. S. Siddeeq, “Emotion Recognition using Feed Forward Neural Network & Naïve Bayes,” Int. J. Innov. Technol. Explor. Eng., vol. 9, no. 2, pp. 2487–2491, 2019, doi: 10.35940/ijitee.b7070.129219.
  23. S. Jyoti, G. Sharma, and A. Dhall, “Expression empowered residen network for facial action unit detection,” Proc. - 14th IEEE Int. Conf. Autom. Face Gesture Recognition, FG 2019, no. 1, pp. 1–8, 2019, doi: 10.1109/FG.2019.8756580.
  24. M. Malygina, M. Artemyev, A. Belyaev, and O. Perepelkina, Overview of the Advancements in Automatic Emotion Recognition: Comparative Performance of Commercial Algorithms. 2019. doi: 10.31234/osf.io/x7bvd.
  25. S. H. Wang, P. Phillips, Z. C. Dong, and Y. D. Zhang, “Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm,” Neurocomputing, vol. 272, no. September 2020, pp. 668–676, 2018, doi: 10.1016/j.neucom.2017.08.015.
  26. S. Palaniswamy and S. Tripathi, “Emotion recognition from facial expressions using images with pose, illumination and age variation for human-computer/robot interaction,” J. ICT Res. Appl., vol. 12, no. 1, pp. 14–34, 2018, doi: 10.5614/itbj.ict.res.appl.2018.12.1.2.
  27. S. Tripathi, S. Tripathi, and H. Beigi, “Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning,” arXiv1804.05788v3 [cs.AI] 6 Nov 2019 MULTI-MODAL, 2018.
  28. A.Pellacani, M. Graziano, and M. Suatoni, “Design , Development , Validation and Verification of GNC technologies,” Eucass2019, no. July, p. 28760, 2019, doi: 10.13009/EUCASS2019-38.
  29. B. Li, “Facial expression recognition via transfer learning,” EAI Endorsed Trans. e-Learning, no. December, p. 169180, 2018, doi: 10.4108/eai.8-4-2021.169180.
  30. S. Khandelwal, “Research Paper on Facial Emotion Recognition Using Transfer Learning,” pp. 1–6.
  31. S. Sidharth, A. A. Samuel, H. Ranjana, J. T. Panachakel, S. P. K, and S. P. Jun, “Emotion Detection from EEG using Transfer Learning,” arXiv:2306.05680v1, 2023.
  32. Y. Piparsaniyan, V. Sharma, and K. Mahapatra, Robust facial expression recognition using Gabor feature and Bayesian discriminating classifier. 2014. doi: 10.1109/ICCSP.2014.6949900.
  33. E. Owusu, Y. Zhan, and Q. R. Mao, “A neural-AdaBoost based facial expression recognition system,” Expert Syst. Appl., vol. 41, no. 7, pp. 3383–3390, 2014, doi: 10.1016/j.eswa.2013.11.041.
  34. M. Abdulrahman, T. R. Gwadabe, F. J. Abdu, and A. Eleyan, “Gabor wavelet transform based facial expression recognition using PCA and LBP,” 2014 22nd Signal Process. Commun. Appl. Conf. SIU 2014 - Proc., no. August 2015, pp. 2265–2268, 2014, doi: 10.1109/SIU.2014.6830717.
  35. D. K. Hu, A. S. Ye, L. Li, and L. Zhang, “Recognition of facial expression via kernel PCA network,” Appl. Mech. Mater., vol. 631–632, pp. 498–501, 2014, doi: 10.4028/www.scientific.net/AMM.631-632.498.
  36. A.De, A. Saha, and M. C. Pal, “A human facial expression recognition model based on eigen face approach,” Procedia Comput. Sci., vol. 45, no. C, pp. 282–289, 2015, doi: 10.1016/j.procs.2015.03.142.
  37. M. Suk and B. Prabhakaran, “Real-time facial expression recognition on smartphones,” Proc. - 2015 IEEE Winter Conf. Appl. Comput. Vision, WACV 2015, no. November, pp. 1054–1059, 2015, doi: 10.1109/WACV.2015.145.
  38. Y. D. Zhang et al., “Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation,” IEEE Access, vol. 4, no. c, pp. 8375–8385, 2016, doi: 10.1109/ACCESS.2016.2628407.
  39. K. Slimani, M. Kas, Y. El Merabet, R. Messoussi, and Y. Ruichek, “Facial emotion recognition,” Int. J. Adv. Res. Comput. Eng. Technol., vol. 7, no. 11, pp. 88–94, 2018, doi: 10.1145/3177148.3180092.
  40. Y. Sun and J. Yu, “Facial expression recognition by fusing gabor and local binary pattern features,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 10133 LNCS, no. January 2017, pp. 209–220, 2017, doi: 10.1007/978-3-319-51814-5_18.
  41. A.K. Katsaggelos, S. Bahaadini, and R. Molina, “Audiovisual Fusion: Challenges and New Approaches,” Proc. IEEE, vol. 103, no. 9, pp. 1635–1653, 2015, doi: 10.1109/JPROC.2015.2459017.
  42. A.T. Lopes, E. de Aguiar, A. F. De Souza, and T. Oliveira-Santos, “Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order,” Pattern Recognit., vol. 61, no. October 2017, pp. 610–628, 2017, doi: 10.1016/j.patcog.2016.07.026.
  43. S.-H. Wang, W. Yang, Z. Dong, P. Phillips, and Y. Zhang, “Facial Emotion Recognition via Discrete Wavelet Transform, Principal Component Analysis, and Cat Swarm Optimization,” Lect. Notes Comput. Sci., vol. 10559, pp. 203–214, Sep. 2017, doi: 10.1007/978-3-319-67777-4_18.
  44. E. BATTINI SÖNMEZ, “An Automatic Multilevel Facial Expression Recognition System,” Süleyman Demirel Üniversitesi Fen Bilim. Enstitüsü Derg., vol. 0, no. 0, p. 10, 2018, doi: 10.19113/sdufbed.50007.

Copyright @ 2026-2017 IRJIET. All Right Reserved.

Developed by Byzero Technologies