Research Progress and Application Prospect of Facial Micro-Expression Analysis in Forensic Psychiatry

doi:10.12116/j.issn.1004-5619.2022.120104

Abstract

Abstract:

Research on facial micro-expression analysis has been going on for decades. Micro-expression can reflect the true emotions of individuals, and it has important application value in assisting auxiliary diagnosis and disease monitoring of mental disorders. In recent years, the development of artificial intelligence and big data technology has made the automatic recognition of micro-expressions possible, which will make micro-expression analysis more convenient and more widely used. This paper reviews the development of facial micro-expression analysis and its application in forensic psychiatry, to look into further application prospects and development direction.

Key words: forensic psychiatry, facial micro-expression, mental disorder, violence risk, malingering, review

CLC Number:

Wen LI, Hao-zhe LI, Chen CHEN, Wei-xiong CAI. Research Progress and Application Prospect of Facial Micro-Expression Analysis in Forensic Psychiatry[J]. Journal of Forensic Medicine, 2023, 39(5): 493-500.

References 68

1	YAN W J， WU Q， LIANG J， et al. How fast are the leaked facial expressions： The duration of micro-expressions[J]. J Nonverbal Behav，2013，37（4）：217-230. doi：10.1007/s10919-013-0159-8 .
2	EKMAN P. Darwin， deception， and facial expression[J]. Ann N Y Acad Sci，2003，1000：205-221. doi：10.1196/annals.1280.010 .
3	HAYNIE D L， LAMB M E. Positive and negative facial expressiveness in 7-，10-， and 13-month-old infants[J]. Infant Behav Dev，1995，18（2）：257-259. doi：10.1016/0163-6383（95）90055-1 .
4	FRANK M G， SVETIEVA E. Microexpressions and deception[M]//Understanding facial expressions in communication. New Delhi： Springer India，2014：227-242. doi：10.1007/978-81-322-1934-7_11 .
5	GROSS J J. Emotion regulation： Affective， cognitive， and social consequences[J]. Psychophysiology，2002，39（3）：281-291. doi：10.1017/s00485772013 93198 .
6	王甦菁，邹博超，刘瑞，等. 隐藏情绪分析与识别方法[J].心理科学进展，2020，28（9）：1426-1436.
	WANG S J， ZOU B C， LIU R， et al. Concealed emotion analysis and recognition method[J]. Xinlikexue Jinzhan，2020，28（9）：1426-1436.
7	HAGGARD E A， ISAACS K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy[J]. Meth Res Psychother，1966：154-165. doi：10.1007/978-1-4684-6045-2_14 .
8	EKMAN P， FRIESEN W V. Nonverbal leakage and clues to deception[J]. Psychiatry，1969，32（1）：88-106. doi：10.1080/00332747.1969.11023575 .
9	张雨铭. 微表情分析技术在侦查讯问中的应用研究[J].河南警察学院学报，2021，30（4）：123-128. doi：10.16231/j.cnki.jhpc.2021.04.014 .
	ZHANG Y M. Research on the application of micro-expression analysis in investigation and interrogation[J]. Henan Jingcha Xueyuan Xuebao，2021，30（4）：123-128.
10	魏昭质. 非接触式审讯技术在侦查讯问中的应用[J].北京警察学院学报，2021（1）：93-98. doi：10.16478/j.cnki.jbjpc.20201123.001 .
	WEI Z Z. The application of the non-contact interrogation technology in interrogation[J]. Beijing Jingcha Xueyuan Xuebao，2021（1）：93-98.
11	MATSUMOTO D， HWANG H C. Microexpressions differentiate truths from lies about future malicious intent[J]. Front Psychol，2018，9：2545. doi：10.3389/fpsyg.2018.02545 .
12	汤瑞丽，蔡运荃. 微表情在课堂教学中的识别与运用[J].基础教育研究，2016（15）：72-74. doi： 10.3969/j.issn.1002-3275.2016.15.021 .
	TANG R L， CAI Y Q. Recognition and application of microexpressions in classroom teaching[J]. Jichu Jiaoyu Yanjiu，2016（15）：72-74.
13	韩丽，李洋，周子佳，等. 课堂环境中基于面部表情的教学效果分析[J].现代远程教育研究，2017（4）：97-103，112. doi：10.3969/j.issn.1009-5195.2017.04.011 .
	HAN L， LI Y， ZHOU Z J， et al. Teaching effect analysis based on the facial expression recognition in classroom[J]. Xiandai Yuancheng Jiaoyu Yanjiu，2017（4）：97-103，112.
14	向莉，薛红，黄岿，等. 基于微表情识别技术的病情预警系统设计[J].电子技术与软件工程，2021（12）：122-123.
	XIANG L， XUE H， HUANG K， et al. Design of disease early warning system based on micro-expression recognition technology[J]. Dianzi Jishu Yu Ruanjian Gongcheng，2021（12）：122-123.
15	王心如，司建伟，张思梦，等. 基于视频识别的独居老人突发失能检测算法研究[J].数码设计（上），2020，9（6）：52.
	WANG X R， SI J W， ZHANG S M， et al. Research on detection algorithm of sudden disability in elderly people living alone based on video recognition[J]. Shuma Sheji （Volume Ⅰ），2020，9（6）：52.
16	COHN J F， KRUEZ T S， MATTHEWS I， et al. Detecting depression from facial actions and vocal prosody[C]//2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. Amsterdam： IEEE，2009：1-7. doi：10.1109/ACII.2009.5349358 .
17	ALVARI G， FURLANELLO C， VENUTI P. Is smiling the key? Machine learning analytics detect subtle patterns in micro-expressions of infants with ASD[J]. J Clin Med，2021，10（8）：1776. doi：10. 3390/jcm10081776 .
18	FUJIWARA T， MIZUKI R， MIKI T， et al. Association between facial expression and PTSD symptoms among young children exposed to the Great East Japan Earthquake： A pilot study[J]. Front Psychol，2015，6：1534. doi：10.3389/fpsyg.2015.01534 .
19	ELLGRING H， GAEBEL W. Facial expression in schizophrenic patients[C]//Past， Present and Future of Psychiatry. Rio De Janiero： World Scientific，1994：435-9. doi：10.1142/9789814440912_0090 .
20	EKMAN P， FRIESEN W. What the face reveals： Basic and applied studies of spontaneous expression using the facial action coding system （FACS）[M]. 2nd ed. Palo Alto： Oxford University Press，1978：371-372.
21	HWANG H C， MATSUMOTO D. Introduction to measuring emotions in the face[M]//MEISELMAN H L. Emotion measurement. 2nd ed. Sawston： Woodhead Publishing，2021：225-249. doi：10.1016/B978-0-12-821124-3.00008-9 .
22	LOTZIN A， HAACK-DEES B， RESCH F， et al. Facial emotional expression in schizophrenia adolescents during verbal interaction with a parent[J]. Eur Arch Psychiatry Clin Neurosci，2013，263（6）：529-536. doi：10.1007/s00406-012-0386-8 .
23	FRANK M G， HERBASZ M， SINUK K， et al. I see how you feel： Training laypeople and professionals to recognize fleeting emotions; proceedings of the 2005[C]. Annual Meeting of the International Communication， New York：2009.
24	CAMPBELL K， CARPENTER K L， HASHEMI J， et al. Computer vision analysis captures atypical attention in toddlers with autism[J]. Autism，2019，23（3）：619-628. doi：10.1177/1362361318766247 .
25	ALI M R， MYERS T， WAGNER E， et al. Facial expressions can detect Parkinson’s disease： Preliminary evidence from videos collected online[J]. NPJ Digit Med，2021，4：129. doi：10.1038/s41746-021-00502-8 .
26	SAPIRO G， HASHEMI J， DAWSON G. Computer vision and behavioral phenotyping： An autism case study[J]. Curr Opin Biomed Eng，2019，9：14-20. doi：10.1016/j.cobme.2018.12.002 .
27	OWAYJAN M， KASHOUR A， HADDAD N AL， et al. The design and development of a Lie Detection System using facial micro-expressions[C]//2012 2nd International Conference on Advances in Computational Tools for Engineering Applications （ACTEA）. Beirut： IEEE，2012：33-38. doi：10.1109/ICTEA.2012.6462897 .
28	徐峰，张军平. 人脸微表情识别综述[J].自动化学报，2017，43（3）：333-348. doi：10.16383/j.aas.2017.c160398 .
	XU F， ZHANG J P. Facial microexpression recognition： A survey[J]. Zidonghua Xuebao，2017，43（3）：333-348.
29	HASHEMI J， DAWSON G， CARPENTER K L H， et al. Computer vision analysis for quantification of autism risk behaviors[J]. IEEE Trans Affect Comput，2021，12（1）：215-226. doi：10.1109/TAFFC.2018.286 8196 .
30	PFISTER T， LI X B， ZHAO G Y， et al. Recognising spontaneous facial micro-expressions[C]//2011 International Conference on Computer Vision. Barcelona： IEEE，2011：1449-1456. doi：10.1109/ICCV.2011.612 6401 .
31	YAN W J， QI W， LIU Y J， et al. CASME database： A dataset of spontaneous micro-expressions collected from neutralized faces[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai： IEEE，2013：1-7. doi：10.1109/FG.2013.6553799 .
32	POLIKOVSKY S， KAMEDA Y， OHTA Y. Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[C]//3rd International Conference on Imaging for Crime Detection and Prevention （ICDP 2009）. London： IET，2009：1-6.
33	SHREVE M， GODAVARTHY S， GOLDGOF D， et al. Macro- and micro-expression spotting in long videos using spatio-temporal strain[C]//2011 IEEE Int Conf Autom Face Gesture Recognit （FG），Santa Barbara： IEEE，2011：51-56. doi：10.1109/FG. 2011.5771451 .
34	OTHMANI A， KADOCH D， BENTOUNES K， et al. Towards robust deep neural networks for affect and depression recognition from speech[EB/OL]. （2020-09-18）[2022-01-20]. doi:10.48550/arXiv.1911. 00310 . .
35	LIU Z Y， WANG D Y， DING Z J， et al. A novel bimodal fusion-based model for depression recognition[C]//2020 IEEE International Conference on E-health Networking， Application & Services. Shenzhen：IEEE，2020：1-4. doi：10.1109/HEALTH COM49281.2021.9399033 .
36	REJAIBI E， KOMATY A， MERIAUDEAU F， et al. MFCC-based recurrent neural network for automatic clinical depression recognition and assessment from speech[J]. Biomed Signal Process Control，2022，71：103107. doi：10.1016/j.bspc.2021.103107 .
37	HE L， CHAN J C W， WANG Z M. Automatic depression recognition using CNN with attention mechanism from videos[J]. Neurocomputing，2021，422：165-175. doi：10.1016/j.neucom.2020.10.015 .
38	CHEN Q， CHATURVEDI I， JI S X， et al. Sequential fusion of facial appearance and dynamics for depression recognition[J]. Pattern Recognit Lett，2021，150：115-121. doi：10.1016/j.patrec.2021.07.005 .
39	XU L J， HOU J J， GAO J. A novel smart depression recognition method using human-computer interaction system[J]. Wirel Commun Mob Comput，2021，2021：5565967. doi：10.1155/2021/5565967 .
40	DEN UYL M J， VAN KUILENBURG H. The FaceReader：Online facial expression recognition[C]//IN L P J. J， NOLDUS L P J J， GRIECO F， et al. Proceedings of Measuring Behaviour 2005，5th International Conference on Methods and Techniques in Behavioral Research. Wageningen： Noldus Information Technology，2005：589-590.
41	LITTLEWORT G， WHITEHILL J， WU T F， et al. The computer expression recognition toolbox （CERT）[C]//2011 IEEE International Conference on Automatic Face & Gesture Recognition. Santa Barbara： IEEE，2011：298-305. doi：10.1109/FG.2011. 5771414 .
42	BALTRUSAITIS T， ZADEH A， LIM Y C， et al. OpenFace 2.0： Facial behavior analysis toolkit[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition. Xi’an： IEEE，2018：59-66. doi：10.1109/FG.2018.00019 .
43	ALGHOWINEM S， GOECKE R， WAGNER M， et al. Multimodal depression detection： Fusion analysis of paralinguistic， head pose and eye gaze behaviors[J]. IEEE Trans Affect Comput，2018，9（4）：478-490. doi：10.1109/TAFFC.2016.2634527 .
44	PAMPOUCHIDOU A， PEDIADITIS M， KAZANT-ZAKI E， et al. Automated facial video-based recognition of depression and anxiety symptom severity： Cross-corpus validation[J]. Mach Vis Appl，2020，31（4）：1-19. doi：10.1007/s00138-020-01080-7 .
45	WEI H. Elderly depression recognition based on facial micro-expression extraction[J]. Traitement Du Signal，2021，38（4）：1123-1130. doi：10.18280/ts.380423 .
46	EGGER H L， DAWSON G， HASHEMI J， et al. Automatic emotion and attention analysis of young children at home： A ResearchKit autism feasibility study[J]. NPJ Digit Med，2018，1：20. doi：10.1038/s41746-018-0024-6 .
47	FALKENBERG I， BARTELS M， WILD B. Keep smiling! Facial reactions to emotional stimuli and their relationship to emotional contagion in patients with schizophrenia[J]. Eur Arch Psychiatry Clin Neurosci，2008，258（4）：245-253. doi：10.1007/s00406-007-0792-5 .
48	GAEBEL W， WÖLWER W. Facial expressivity in the course of schizophrenia and depression[J]. Eur Arch Psychiatry Clin Neurosci，2004，254（5）：335-342. doi：10.1007/s00406-004-0510-5 .
49	TRÉMEAU F， MALASPINA D， DUVAL F， et al. Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects[J]. Am J Psychiatry，2005，162（1）：92-101. doi：10.1176/appi.ajp.162.1.92 .
50	AGHEVLI M A， BLANCHARD J J， HORAN W P. The expression and experience of emotion in schizophrenia： A study of social interactions[J]. Psychiatry Res，2003，119（3）：261-270. doi：10.1016/S0165-1781（03）00133-1 .
51	王国强. 《全国精神卫生工作规划（2015—2020年）》解读[EB/OL].（2015-06-18）[2022-01-22]. .
	WANG G Q. Interpretation of National Mental Health Work Plan （2015—2020）[EB/OL].（2015-06-18）[2022-01-22]. .
52	DUPUY M， ABDALLAH M， SWENDSEN J， et al. Real-time cognitive performance and positive symptom expression in schizophrenia[J]. Eur Arch Psychiatry Clin Neurosci，2022，272（3）：415-425. doi：10.1007/s00406-021-01296-2 .
53	DE MELO W C， GRANGER E， HADID A. Depression detection based on deep distribution learning[C]//2019 IEEE International Conference on Image Processing. Chinese Taipei：IEEE，2019：4544-4548. doi：10.1109/ICIP.2019.8803467 .
54	COMER J S. Introduction to the special series： Applying new technologies to extend the scope and accessibility of mental health care[J]. Cogn Behav Pract，2015，22（3）：253-257. doi：10.1016/j.cbpra.2015.04.002 .
55	FALAGAS M E， VARDAKAS K Z， VERGIDIS P I. Under-diagnosis of common chronic diseases： Prevalence and impact on human health[J]. Int J Clin Pract，2007，61（9）：1569-1579. doi：10.1111/j.1742-1241.2007.01423.x .
56	FJELLVANG M， GRØNING L， HAUKVIK U K. Imaging violence in schizophrenia： A systematic review and critical discussion of the MRI literature[J]. Front Psychiatry，2018，9：333. doi：10.3389/fpsyt.201 8.00333 .
57	WHITING D， FAZEL S. Epidemiology and risk factors for violence in people with mental disorders [M]//CARPINIELLO B， VITA A， MENCACCI C. Violence and mental disorders. Cham： Springer International Publishing，2020：49-62.
58	POLDRACK R A， MONAHAN J， IMREY P B， et al. Predicting violent behavior： What can neuroscience add?[J]. Trends Cogn Sci，2018，22（2）：111-123. doi：10.1016/j.tics.2017.11.003 .
59	WANG K Z， BANI-FATEMI A， ADANTY C， et al. Prediction of physical violence in schizophrenia with machine learning algorithms[J]. Psychiatry Res，2020，289：112960. doi：10.1016/j.psychres.2020.112960 .
60	SANSEGUNDO M S， FERRER-CASCALES R， BELLIDO J H， et al. Prediction of violence， suicide behaviors and suicide ideation in a sample of institutionalized offenders with schizophrenia and other psychosis[J]. Front Psychol，2018，9：1385. doi：10.3389/fpsyg.2018.01385 .
61	LECLERC M P， REGENBOGEN C， HAMILTON R H， et al. Some neuroanatomical insights to impulsive aggression in schizophrenia[J]. Schizophr Res，2018，201：27-34. doi：10.1016/j.schres.2018.06.016 .
62	EKMAN P， DAVIDSON R J， FRIESEN W V. The Duchenne smile： Emotional expression and brain physiology. II[J]. J Pers Soc Psychol，1990，58（2）：342-353. doi：10.1037//0022-3514.58.2.330 .
63	PORTER S， BRINKE L TEN. Reading between the lies： Identifying concealed and falsified emotions in universal facial expressions[J]. Psychol Sci，2008，19（5）：508-514. doi：10.1111/j.1467-9280.2008. 02116.x .
64	BURGOON J K. Microexpressions are not the best way to catch a liar[J]. Front Psychol，2018，9：1672. doi：10.3389/fpsyg.2018.01672 .
65	庄东哲. 侦查讯问中的行为科学技术方法[J].中国人民公安大学学报（社会科学版），2014，30（3）：95-101.
	ZHUANG D Z. Technological methods of behavioral science in investigation and interrogation[J]. Zhongguo Renmin Gongan Daxue Xuebao （Social sciences edition），2014，30（3）：95-101.
66	王鹏，李雅楠. 人工智能识别微反应技术在侦查讯问中的应用[J].河北公安警察职业学院学报，2021，21（1）：13-16.
	WANG P， LI Y N. Application of the technology of microreaction recognition by artificial intelligence in investigation and interrogation[J]. Hebei Gongan Jingcha Zhiye Xueyuan Xuebao，2021，21（1）：13-16.
67	OLDERBAK S， HILDEBRANDT A， PINKPANK T， et al. Psychometric challenges and proposed solutions when scoring facial emotion expression codes[J]. Behav Res Methods，2014，46（4）：992-1006. doi：10.3758/s13428-013-0421-3 .
68	GORDON I， TANAKA J W， PIERCE M， et al. Facial expression production and training[J]. J Vis，2011，11（11）：565. doi：10.1167/11.11.565 .

[1]	Zhong-hua WANG, Shu-jin LI. Research Progress on Molecular Biology of Human Height Estimation [J]. Journal of Forensic Medicine, 2023, 39(5): 487-492.
[2]	Lu CHEN, Zhe ZHOU, Sheng-qi WANG. Process of Forensic Medicine in DNA Identification of Aged Human Remains [J]. Journal of Forensic Medicine, 2023, 39(5): 478-486.
[3]	Yong ZENG, Dong-hua ZOU, Ying FAN, Qing XU, Lu-yang TAO, Yi-jiu CHEN, Zheng-dong LI. Research Progress and Forensic Application of Human Vascular Finite Element Modeling and Biomechanics [J]. Journal of Forensic Medicine, 2023, 39(5): 471-477.
[4]	Fei FAN, Juan WU, Zhen-hua DENG. Application Progress of Objective Audiological Detection Techniques in Forensic Clinical Medicine [J]. Journal of Forensic Medicine, 2023, 39(4): 360-366.
[5]	Qing-qing XIANG, Li-fang CHEN, Qin SU, Yu-kun DU, Pei-yan LIANG, Xiao-dong KANG, He SHI, Qu-yi XU, Jian ZHAO, Chao LIU, Xiao-hui CHEN. Research Progress on Microbial Community Succession in the Postmortem Interval Estimation [J]. Journal of Forensic Medicine, 2023, 39(4): 399-405.
[6]	Yu-qi CAO, Yan SHI, Ping XIANG, Yin-long GUO. Research Progress on Machine Learning Assisted Non-Targeted Screening Strategy for Identification of Fentanyl Analogs [J]. Journal of Forensic Medicine, 2023, 39(4): 406-416.
[7]	Ran LI, Hong-yu SUN. Methods and Research Hotspots of Forensic Kinship Testing [J]. Journal of Forensic Medicine, 2023, 39(3): 231-239.
[8]	Xiao-yan MA, Hong-yu SUN, Qing LI. Research Progresses of Tri-Allelic Patterns in Autosomal STR in Forensic DNA Analysis [J]. Journal of Forensic Medicine, 2023, 39(3): 240-246.
[9]	Hang CHEN, Jing HU, Zheng QIAO, Hong-xiao DENG, Min LÜ, Wei LIU. Research Progress on Biological Matrix Reference Materials in Forensic Toxicology [J]. Journal of Forensic Medicine, 2023, 39(2): 176-185.
[10]	Hong-yan GAO, Guang-fu LIU, Jian WU, Peng-yu CHEN. Animal DNA Typing and Its Research Progress in Forensic Medicine [J]. Journal of Forensic Medicine, 2023, 39(2): 161-167.
[11]	Zhong-ping CHENG, Yan-fei LIU, Xing-min XU, Yao-nan MO. Progress in the Application of Magnetic Nanoparticles in Forensic Trace Analysis [J]. Journal of Forensic Medicine, 2023, 39(2): 168-175.
[12]	Hong-xia HAO, Ya-hui WANG, Zhi-lu ZHOU, Tai-ang LIU, Jin CHEN, Yu-heng HE, Lei WAN, Wen-tao XIA. Research Progress of Age Estimation in the Living by Knee Joint MRI [J]. Journal of Forensic Medicine, 2023, 39(1): 66-71.
[13]	Zi-wei WANG, Qian-nan XU, Cheng-tao LI, Xi-ling LIU. Age Estimation Based on DNA Methylation and Its Application Prospects in Forensic Medicine [J]. Journal of Forensic Medicine, 2023, 39(1): 72-82.
[14]	Hui WU, Fang-fang LIU, Jun-da WU, Ying XIE. Research Progress on Estimation of Postmortem Interval Based on Ocular Tissues Structure [J]. Journal of Forensic Medicine, 2023, 39(1): 50-56.
[15]	Tian-pu WU, Jian-long MA, Xin-biao LIAO, Dong-chuan ZHANG, Kai-jun MA, Yan-geng YU, Long CHEN. Research Progress on Molecular Changes in Pulmonary Hypoxia and Cause of Death Identification in Mechanical Asphyxia [J]. Journal of Forensic Medicine, 2023, 39(1): 57-65.