2000 Fiscal Year Final Research Report Summary
An Investigation of Cooperative Understanding of Utterances and Gestures Based on Interaction in Semantics Level
| Project/Area Number |
10680388
|
|---|
| Research Category |
Grant-in-Aid for Scientific Research (C)
|
| Allocation Type | Single-year Grants |
|---|
| Section | 一般 |
|---|
| Research Field |
Intelligent informatics
|
|---|
| Research Institution | Kyushu Institute of Technology (2000)
Oita University (1998-1999) |
|---|
Principal Investigator |
ENDO Tsutomu Kyushu Institute of Technology, Department of Artificial Intelligence, Professor, 情報工学部, 教授 (10112294)
|
|---|
| Co-Investigator(Kenkyū-buntansha) |
KAGAWA Tsuneo Oita University, Department of Computer Science and Intelligent Systems, Research Associate, 工学部, 助手 (90253773)
|
|---|
| Project Period (FY) |
1998 – 2000
|
| Keywords | Semantic analysis / multimedia / multimodal / Human interface / Cooperative understanding / Information Integration / Gesture recognition / Natural language understanding |
|---|
| Research Abstract |
We are developing a problem solving and knowledge acquisition system based on co-reference between drill texts and dialogue with a teacher, focusing on first-grade mathematics. This research proposed a method of cooperative understanding of utterances and gestures.
(1) Contextual information processing.
We defined the context of dialogue, which consists of surface and case structure of utterances, intention and attention of the speaker, situation of dialogue, and world knowledge. We then presented the algorithms of generating utterances from the system as well as interpreting responses from the teacher using contextual information.
(2) Analysis of gestures and utterances.
Our point of interest is the movement of the tip of teacher's pen. We developed a simple input device to detect the three-dimensional coordinates of the tip of pen, and presented the algorithms to extract features from moving points. A feature-based approach is used for gesture recognition. We then proposed a method of parsing word candidates given from speech recognition program.
(3) Cooperative understanding of utterances and gestures.
We defined a multi-modal semantic representation to describe the meaning of utterances and gestures, and showed how to integrate our algorithms for utterance and gesture analysis. We concluded with an evaluation of the understanding system against the design principles, which provide the basis for the integration of multi-modal information during a dialogue.
(4) Generation of gestures in cooperation with utterances.
Gestures such as pointing of objects on a drill text or drawing of pictures, are represented by movement of a pen, and are displayed as three-dimensional graphical data. We defined a gesture frame and gesture element as an intermediate representation, and presented algorithms of generating them from the semantic representation with the synchronized phrase..
|
Research Products
(16 results)
