| Project/Area Number |
19K12023
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (C)
|
| Allocation Type | Multi-year Fund |
|---|
| Section | 一般 |
|---|
| Review Section |
Basic Section 61010:Perceptual information processing-related
|
|---|
| Research Institution | Osaka Prefecture University |
|---|
Principal Investigator |
ロイ パルサプラティム 大阪府立大学, 研究推進機構, 客員研究員 (10837222)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
岩村 雅一 大阪府立大学, 工学(系)研究科(研究院), 准教授 (80361129)
井上 勝文 大阪府立大学, 工学(系)研究科(研究院), 准教授 (50733804)
|
|---|
| Project Period (FY) |
2019-04-01 – 2022-03-31
|
| Project Status |
Granted (Fiscal Year 2019)
|
| Budget Amount *help |
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2021: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Fiscal Year 2020: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Fiscal Year 2019: ¥3,120,000 (Direct Cost: ¥2,400,000、Indirect Cost: ¥720,000)
|
|---|
| Keywords | Sign Lang. Recognition / 3D Avatar Model / Machine Learning / Natural Lang. Processing / SyntheticData Generation / Computer Vision / Deep Learning / Sign Language / Data Synthesis |
|---|
| Outline of Research at the Start |
Hearing impaired people use Sign language (SL) as the primary way of communication which is performed through the use of hand gestures, movements of arm/body, expressions, etc. To understand the SL by common people, many approaches were proposed for gesture recognition. A limitation of these approaches is that they require a large dataset for preparing the machine learning models which require manual annotation of millions of gestures. To solve this, we propose to develop a 3D avatar model to mimic SL which will be used to generate synthetic data. It will be a robust system for SL recognition.
|
| Outline of Annual Research Achievements |
We proposed the project for an automatic translation system from sign gesture to speech and vice versa so that normal people and hearing and speech impaired people can communicate with each other. To achieve this, a number of steps which are very crucial for the success of the project are already performed. We have studied the literature work extensively. A number of research work from top international conference and journals are implemented and tested. Some preprocessing steps like extraction of finger and hand movement from videos and images are done. We have tested some feature extraction algorithms for classification. More than 500 videos along with subtitles are collected to make a benchmark dataset and to test our experiment. Along with the recognition component, a 3D avatar (animated) model is being designed. This model will consider sign gesture and corresponding labels (obtained from subtitle/background speech) and will generate synthetic sign gestures. For this purpose, the Natural Language Processing technique is used to convert the input sentence to sign language. Next, the motion of the avatar is defined based on the sign language. The generation of sign movements is accomplished with the help of an animation tool called Blender.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
The development of the modules were being done smoothly in the beginning. However, due to the pandemic of Covid'19, presently, the city is undergoing prolonged lockdown. The research labs are closed for last 3 months and we could not do much progress in this period. However, we have been working on collecting dataset and literature review. Once the situation improves, pending stuffs will be completed soon.
|
| Strategy for Future Research Activity |
We are going to implement a deep learning architecture for sign language recognition. It will consider image/video data along with finger/hand movement trajectory data. For this purpose, the sign language videos which contain sub-titles (text) and background speech will be separated using text extraction or speech separation module. The text image/speech will be converted to natural language for data annotation. Thereafter, deep learning framework will be tested. We plan to write a paper using this result and submit in an international conference. In parallel, our 3D avatar model will be used by Deep Learning Architecture to generate many synthetic data which will help in training a robust sign gesture recognition model. Following this, an end to end system with GUI support for easy learning and understanding for Sign Gesture Recognition will be developed.
|
