• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Japanese Machine Lip-reading System Based on Human Lip-reading Method

Research Project

Project/Area Number 23700672
Research Category

Grant-in-Aid for Young Scientists (B)

Allocation TypeMulti-year Fund
Research Field Rehabilitation science/Welfare engineering
Research InstitutionKanagawa Institute of Technology

Principal Investigator

MIYAZAKI Tsuyoshi  神奈川工科大学, 情報学部, 准教授 (20329634)

Co-Investigator(Renkei-kenkyūsha) NAKASHIMA Toyoshiro  椙山女学園大学, 文化情報学部・文化情報学科, 教授 (90247601)
Project Period (FY) 2011 – 2013
Project Status Completed (Fiscal Year 2013)
Budget Amount *help
¥2,990,000 (Direct Cost: ¥2,300,000、Indirect Cost: ¥690,000)
Fiscal Year 2013: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2012: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2011: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Keywords機械読唇 / 障害者支援 / 画像処理 / 画像認識 / パターン認識 / 聴覚障害者支援 / 発話映像 / 口形画像 / 口形検出 / 動画像認識 / 聴覚障がい者支援
Research Abstract

I have improved the detection rate of mouth shapes from Japanese utterance images, and proposed a recognition method of Japanese words. I aimed for more than 90% at the detection rate of mouth shapes, but it was finally with 82.7% by using the optical flow and the template matching. This shows that the detection rate was improved by 8.6% from the grant start. By the recognition of the Japanese utterance word, I proposed the calculation method as reliability in consideration of the length of utterance word, and 36 prefectures were recognized from 47 prefectures utterance images by the experiment. The recognition rate became approximately 76.6%. In future, I consider a method to improve accuracy to extract the period that the mouth shapes are formed.

Report

(4 results)
  • 2013 Annual Research Report   Final Research Report ( PDF )
  • 2012 Research-status Report
  • 2011 Research-status Report
  • Research Products

    (9 results)

All 2014 2013 2012

All Journal Article (1 results) (of which Peer Reviewed: 1 results) Presentation (8 results)

  • [Journal Article] Mouth Shape Detection Based on Template Matching and Optical Flow for Machine Lip Reading2013

    • Author(s)
      Tsuyoshi Miyazaki, Toyoshiro Nakashima, Naohiro Ishii
    • Journal Title

      International Journal of Software Innovation

      Volume: Vol.1, No.1 Pages: 14-25

    • Related Report
      2013 Final Research Report 2012 Research-status Report
    • Peer Reviewed
  • [Presentation] Analysis of Mouth Shape Deformation Rate for Generation of Japanese Utterance Images Automatically2014

    • Author(s)
      Tsuyoshi Miyazaki and Toyoshiro Nakashima
    • Organizer
      International Conference on Software Engineering Research, Management and Applications (SERA 2014) (掲載決定, 頁未定)
    • Related Report
      2013 Final Research Report
  • [Presentation] 口形ベースの機械読唇における単語認識手法の提案と評価2014

    • Author(s)
      宮崎剛, 中島豊四郎
    • Organizer
      マルチメディア,分散,協調とモバイル(DICOMO2014)シンポジウム, 896-902
    • Related Report
      2013 Final Research Report
  • [Presentation] 口形ベースの機械読唇における単語認識手法の提案と評価2014

    • Author(s)
      宮崎剛, 中島豊四郎
    • Organizer
      マルチメディア,分散,協調とモバイル(DICOMO2014)シンポジウム
    • Place of Presentation
      新潟県月岡温泉 ホテル泉慶
    • Related Report
      2013 Annual Research Report
  • [Presentation] 日本語発話時の口形変化量の分析と発話映像自動生成への適用2013

    • Author(s)
      宮崎剛, 中島豊四郎
    • Organizer
      マルチメディア,分散,協調とモバイル(DICOMO2013)シンポジウム
    • Place of Presentation
      十勝川温泉 ホテル大平原
    • Related Report
      2013 Annual Research Report
  • [Presentation] A Detection Method of Basic Mouth Shapes from Japanese Utterance Images by Template Matching and Optical Flow2012

    • Author(s)
      Tsuyoshi Miyazaki, Toyoshiro Nakashima and Naohiro Ishii
    • Organizer
      13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2012, SCI 443, 83-94
    • Related Report
      2013 Final Research Report
  • [Presentation] Producing text and speech from video images of lips movement photographed in speaking Japanese by using mouth shape sequence code -An experimental system to communicate with hearing impaired persons -2012

    • Author(s)
      Shiori Kawahata, Eiko Koyama, Tsuyoshi Miyazaki and Fujio Yamamoto
    • Organizer
      The Seventeenth International Symposium on Artificial Life and Robotics 2012 (AROB 17th '12), 867-870
    • Related Report
      2013 Final Research Report
  • [Presentation] A Detection Method of Basic Mouth Shapes from Japanese Utterance Images by Template Matching and Optical Flow2012

    • Author(s)
      Tsuyoshi Miyazaki, Toyoshiro Nakashima and Naohiro Ishii
    • Organizer
      13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2012, SCI 443
    • Place of Presentation
      京都府京都市 キャンパスプラザ京都
    • Related Report
      2012 Research-status Report
  • [Presentation] Producing text and speech from video images of lips movement photographed in speaking Japanese by using mouth shape sequence code - An experimental system to communicate with hearing impaired persons -2012

    • Author(s)
      Shiori Kawahata, Eiko Koyama, Tsuyoshi Miyazaki and Fujio Yamamoto
    • Organizer
      The Seventeenth International Symposium on Artificial Life and Robotics 2012 (AROB 17th '12)
    • Place of Presentation
      大分県別府市 別府国際コンベンションセンター
    • Related Report
      2011 Research-status Report

URL: 

Published: 2011-08-05   Modified: 2019-07-29  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi