Numerical Analysis of Facial Movement by USB Camera and Development of Numerical Processing Teaching Materials

2023 Fiscal Year Final Research Report

• Project/Area Number: 19K03178
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Review Section: Basic Section 09080:Science education-related
• Research Institution: Gifu Shotoku Gakuen University
• Principal Investigator: Itou Satoshi 岐阜聖徳学園大学, 経済情報学部, 名誉教授 (80130946)
• Co-Investigator(Kenkyū-buntansha): 鷲野 嘉映 愛知みずほ短期大学, その他部局等, 教授(移行) (90220855)
• Project Period (FY): 2019-04-01 – 2024-03-31
• Keywords: 顔の動き / 動きのラベリング / 動きの分類

Outline of Final Research Achievements

With a USB camera in front of the face, the basic facial movements a) left/right b) up/down c) tilt were performed to detect the face and obtain the feature point coordinates of the face. A 6-axis inertial sensor was attached to the top of the head, and simultaneous measurements confirmed that correct detection was possible.
Deep learning was used to label the movements of facial feature points in the recorded video. Among the obtained coordinates of facial organs, three coordinate points at the head of the nose and the corners of both eyes were recorded and used for analysis. Three types of “movements” were selected for classification and estimation. The trajectories of the three feature points for these three movements were subjected to deep learning, and classification was attempted.
In all cases, the estimation rate was over 80%. This is presumably due to the effective construction of the model.

Free Research Field

教育工学

Academic Significance and Societal Importance of the Research Achievements

科学的理解を深める方法の一つとして、現象をモデル化（抽象化）して数値処理を経て現象の理解にフィードバックする方法がある。学習者は自身の顔の動きに関心を持つことが可能であり、鏡などで確認が可能である。しかし、その動きを数値データとして処理することは経験がないであろう。顔の動きをUSBカメラで取得し動きが数値処理された結果（グラフなど）を見て、顔の動きを制御するなど内容を理解しながら学習が可能になる。
深層学習を用いて、録画された動画での顔の特徴点の動きをラベリングできることを示し、AIの威力を示すことができた。