Development of Facial Expression Recognition Technology in Immersive Virtual Environments

2021 Fiscal Year Final Research Report

• Project/Area Number: 16H05870
• Research Category: Grant-in-Aid for Young Scientists (A)
• Allocation Type: Single-year Grants
• Research Field: Human interface and interaction
• Research Institution: Keio University
• Principal Investigator: Maki Sugimoto 慶應義塾大学, 理工学部(矢上), 教授 (50517399)
• Project Period (FY): 2016-04-01 – 2020-03-31
• Keywords: 表情認識 / ユビキタス光センシング / HMD

Outline of Final Research Achievements

The aim of this project was to develop facial expression recognition technology using an embedded optical sensor array and machine learning, which enables facial expression recognition in a head-mounted display (HMD). Facial expression recognition in an HMD with an ordinal camera system is challenging due to occlusions. We developed a method for detecting basic facial expressions and extended it to estimate the intensity of each facial expression. We have shown that synthesized expressions can be rendered on avatars in a virtual environment. In addition to identifying the basic expression classes, we confirmed that it is possible to reconstruct high-dimensional 3D positions of feature points on a facial surface from the low-dimensional sensor values. Furthermore, we demonstrated that facial expression recognition accuracy was improved by making a training dataset that incorporates gaze and head direction as a machine learning method robust to the diversity of facial expressions.

Free Research Field

バーチャルリアリティ

Academic Significance and Societal Importance of the Research Achievements

本研究では学習データの多様性を考慮した機械学習を行なうことで，没入型バーチャル環境における表情認識精度を頑健にできることを示した．こうした，没入型バーチャル環境に対してユーザの身体情報を反映する技術基盤は，我々が没入型バーチャル環境においてソーシャルコミュニケーションを行なう上で必要となるものであり，実環境とバーチャル環境をシームレスに接続することに貢献するものである．また，本研究課題で開発を行なった組み込み型光センサと機械学習を用いる手法は，低次元のセンサ情報から高次元のユーザの状態を推定することにも活用できるため，様々な身体情報の認識を行なっていく上で重要な技術になると期待される．