2020 Fiscal Year Research-status Report
Vocabulary acquisition and 3D avatar approach for Japanese sign language communication
| Project/Area Number |
19K12023
|
|---|
| Research Institution | Osaka Prefecture University |
|---|
Principal Investigator |
ロイ パルサプラティム 大阪府立大学, 研究推進機構, 客員研究員 (10837222)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
岩村 雅一 大阪府立大学, 工学(系)研究科(研究院), 准教授 (80361129)
井上 勝文 大阪府立大学, 工学(系)研究科(研究院), 准教授 (50733804)
|
|---|
| Project Period (FY) |
2019-04-01 – 2022-03-31
|
| Keywords | Sign Lang. Recognition / 3D Conv. Network / Multi-stream network / Optical flow / Skelton / Face / Hand |
|---|
| Outline of Annual Research Achievements |
The goal of this research project is to obtain the vocabulary of sign language. To do that, we need a highly accurate sign language recognition method. Hence, in this fiscal year, we have explored existing word-level sign language recognition methods and found that their recognition performance is not sufficient for our goal.
The state-of-the-art recognition method of American sign language was initially developed for action recognition and applied to sign language recognition.
However, due to the difference between action recognition and sign language recognition, methods developed for action recognition cannot grasp the detailed features required for distinguishing sign languages. Thus, we proposed a new multi-stream neural network focusing on local regions. Our experiments revealed that the proposed method significantly improved the recognition accuracy by about 15% in the Top-1 accuracy compared with the previous state-of-the-art method.
We presented our research work at IEICE Technical Committee on Pattern Recognition, and Media Understanding (PRMU) in March 2021, and the first author won the Best Presentation Award of the month.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
In the second year, we could propose the state-of-the-art word-level sign language recognition method. Using this method can accelerate the development of a method to obtain the vocabulary of sign language.
|
| Strategy for Future Research Activity |
As written above, we plan to develop a method to obtain sign language vocabulary using the proposed multi-stream neural network. In the first year, we have collected a sufficient amount of videos of American sign languages. We will analyze the video data to acquire phoneme-like elements for sign languages. Regarding them as the alphabet, we challenge to describe sign language vocabulary by the phoneme-like elements.
|
| Causes of Carryover |
Due to the pandemic of Covid'19, we could not spend the research budget as planned.
|
