| 研究課題/領域番号 |
19K12023
|
|---|
| 研究機関 | 大阪府立大学 |
|---|
研究代表者 |
ロイ パルサプラティム 大阪府立大学, 研究推進機構, 客員研究員 (10837222)
|
|---|
| 研究分担者 |
岩村 雅一 大阪府立大学, 工学(系)研究科(研究院), 准教授 (80361129)
井上 勝文 大阪府立大学, 工学(系)研究科(研究院), 准教授 (50733804)
|
|---|
| 研究期間 (年度) |
2019-04-01 – 2022-03-31
|
| キーワード | Sign Lang. Recognition / 3D Avatar Model / Machine Learning / Natural Lang. Processing / SyntheticData Generation |
|---|
| 研究実績の概要 |
We proposed the project for an automatic translation system from sign gesture to speech and vice versa so that normal people and hearing and speech impaired people can communicate with each other. To achieve this, a number of steps which are very crucial for the success of the project are already performed. We have studied the literature work extensively. A number of research work from top international conference and journals are implemented and tested. Some preprocessing steps like extraction of finger and hand movement from videos and images are done. We have tested some feature extraction algorithms for classification. More than 500 videos along with subtitles are collected to make a benchmark dataset and to test our experiment. Along with the recognition component, a 3D avatar (animated) model is being designed. This model will consider sign gesture and corresponding labels (obtained from subtitle/background speech) and will generate synthetic sign gestures. For this purpose, the Natural Language Processing technique is used to convert the input sentence to sign language. Next, the motion of the avatar is defined based on the sign language. The generation of sign movements is accomplished with the help of an animation tool called Blender.
|
| 現在までの達成度 (区分) |
現在までの達成度 (区分)
2: おおむね順調に進展している
理由
The development of the modules were being done smoothly in the beginning. However, due to the pandemic of Covid'19, presently, the city is undergoing prolonged lockdown. The research labs are closed for last 3 months and we could not do much progress in this period. However, we have been working on collecting dataset and literature review. Once the situation improves, pending stuffs will be completed soon.
|
| 今後の研究の推進方策 |
We are going to implement a deep learning architecture for sign language recognition. It will consider image/video data along with finger/hand movement trajectory data. For this purpose, the sign language videos which contain sub-titles (text) and background speech will be separated using text extraction or speech separation module. The text image/speech will be converted to natural language for data annotation. Thereafter, deep learning framework will be tested. We plan to write a paper using this result and submit in an international conference. In parallel, our 3D avatar model will be used by Deep Learning Architecture to generate many synthetic data which will help in training a robust sign gesture recognition model. Following this, an end to end system with GUI support for easy learning and understanding for Sign Gesture Recognition will be developed.
|
| 次年度使用額が生じた理由 |
Due to the pandemic of Covid'19, we could not spend the research budget as planned.
|
