Vocabulary acquisition and 3D avatar approach for Japanese sign language communication

2019 Fiscal Year Research-status Report

• Project/Area Number: 19K12023
• Research Institution: Osaka Prefecture University
• Principal Investigator: ロイ パルサプラティム 大阪府立大学, 研究推進機構, 客員研究員 (10837222)
• Co-Investigator(Kenkyū-buntansha): 岩村 雅一 大阪府立大学, 工学(系)研究科(研究院), 准教授 (80361129) ; 井上 勝文 大阪府立大学, 工学(系)研究科(研究院), 准教授 (50733804)
• Project Period (FY): 2019-04-01 – 2022-03-31
• Keywords: Sign Lang. Recognition / 3D Avatar Model / Machine Learning / Natural Lang. Processing / SyntheticData Generation

Outline of Annual Research Achievements

We proposed the project for an automatic translation system from sign gesture to speech and vice versa so that normal people and hearing and speech impaired people can communicate with each other. To achieve this, a number of steps which are very crucial for the success of the project are already performed. We have studied the literature work extensively. A number of research work from top international conference and journals are implemented and tested. Some preprocessing steps like extraction of finger and hand movement from videos and images are done. We have tested some feature extraction algorithms for classification. More than 500 videos along with subtitles are collected to make a benchmark dataset and to test our experiment. Along with the recognition component, a 3D avatar (animated) model is being designed. This model will consider sign gesture and corresponding labels (obtained from subtitle/background speech) and will generate synthetic sign gestures. For this purpose, the Natural Language Processing technique is used to convert the input sentence to sign language. Next, the motion of the avatar is defined based on the sign language. The generation of sign movements is accomplished with the help of an animation tool called Blender.

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

The development of the modules were being done smoothly in the beginning. However, due to the pandemic of Covid'19, presently, the city is undergoing prolonged lockdown. The research labs are closed for last 3 months and we could not do much progress in this period. However, we have been working on collecting dataset and literature review. Once the situation improves, pending stuffs will be completed soon.

Strategy for Future Research Activity

We are going to implement a deep learning architecture for sign language recognition. It will consider image/video data along with finger/hand movement trajectory data. For this purpose, the sign language videos which contain sub-titles (text) and background speech will be separated using text extraction or speech separation module. The text image/speech will be converted to natural language for data annotation. Thereafter, deep learning framework will be tested. We plan to write a paper using this result and submit in an international conference. In parallel, our 3D avatar model will be used by Deep Learning Architecture to generate many synthetic data which will help in training a robust sign gesture recognition model. Following this, an end to end system with GUI support for easy learning and understanding for Sign Gesture Recognition will be developed.

Causes of Carryover

Due to the pandemic of Covid'19, we could not spend the research budget as planned.