1997 Fiscal Year Final Research Report Summary
Studies on Multimodal Communication by Integrating Speech and Diagram
| Project/Area Number |
08458078
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (B)
|
| Allocation Type | Single-year Grants |
|---|
| Section | 一般 |
|---|
| Research Field |
Intelligent informatics
|
|---|
| Research Institution | KYOTO UNIVERSITY |
|---|
Principal Investigator |
DOSHITA Shuji Kyoto University, Graduate School of Engineering, Professor, 工学研究科, 教授 (00025925)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
ARAKI Masahiro Kyoto University.Center for for Information and Multimedia Studies.Lecturer, 総合情報メディアセンター, 講師 (50252490)
KAWAHARA Tatsuya Kyoto University.Graduate School of Engineering.Assistant Professor, 工学研究科, 助教授 (00234104)
|
|---|
| Project Period (FY) |
1996 – 1997
|
| Keywords | Multimodal Communication / Dialogue / Speech Recognition / Gesture / Drawing System / Information Integration |
|---|
| Research Abstract |
The aim of this research is to realize multimodal communication between human and machines through implementing multimodal drawing system with speech input. The results of this research are below :
1. Integration of speech, gesture and diagram
We developed multi-mode integration method without assuming the synchronization of each input elements. The validity of this integration method was examined by evaluating the improvement of accuracy of information input in multimodal dictation system.
2. Implementation of multimodal interface using speech and pointing gesture
We developed the cooperation mechanism which uses the result of pointing gesture recognition, which is reliable channel, for the recognition of speech, which is relatively unreliable channel. By using the result of pointing gesture recognition, the accuracy of speech recognition raised about 6%. Using this integration method, we implemented a multimodal drawing system with speech input.
3. Interpretation of utterance using diagrammatic and contextual information in multimodal drawing system We realized utterance understanding mechanism which uses context management mechanism by stack in order to deal with spontaneous speech (e. g. ellipsis, reference expression, etc.). We integrated this utterance understanding mechanism to multimodal drawing system.
|
