• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Integration of Event Related Brain Potentials into Speech Recognition Framework

Research Project

Project/Area Number 26870371
Research Category

Grant-in-Aid for Young Scientists (B)

Allocation TypeMulti-year Fund
Research Field Cognitive science
Perceptual information processing
Research InstitutionNara Institute of Science and Technology

Principal Investigator

SAKRIANI SAKTI  奈良先端科学技術大学院大学, 情報科学研究科, 助教 (00395005)

Project Period (FY) 2014-04-01 – 2017-03-31
Project Status Completed (Fiscal Year 2016)
Budget Amount *help
¥3,900,000 (Direct Cost: ¥3,000,000、Indirect Cost: ¥900,000)
Fiscal Year 2015: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Fiscal Year 2014: ¥2,210,000 (Direct Cost: ¥1,700,000、Indirect Cost: ¥510,000)
Keywords音声情報処理 / 脳認知科学 / 音声認識 / 事象関連電位 / 脳波
Outline of Final Research Achievements

Most of ASR systems exist today are still tuned by minimizing word error rate, in which all errors from any words like functional words or fillers are treated in a uniform manner. In fact, the impact of all errors is not the same, but how big and differences are the impact are still unknown. To align with human perception, in this research I propose to utilize the event-related brain potential (ERP) studies into ASR framework, in which the human cognitive process during language comprehension is directly analysis and the possibilities to incorporate within ASR framework are investigated. We were successfully perform the ERP experiments given the ASR results, and we can confirmed that a difference occurs in the EEG/ERP signal when perceiving the communication failure factor. Furthermore, we can learn also how to incorporate different knowledge in ASR framework. We also have opportunities to collaborate with some research institutes from Indonesia, Vietnam, and France.

Report

(4 results)
  • 2016 Annual Research Report   Final Research Report ( PDF )
  • 2015 Research-status Report
  • 2014 Research-status Report
  • Research Products

    (48 results)

All 2017 2016 2015 2014 Other

All Int'l Joint Research (3 results) Journal Article (20 results) (of which Int'l Joint Research: 8 results,  Peer Reviewed: 13 results,  Acknowledgement Compliant: 20 results,  Open Access: 1 results) Presentation (24 results) (of which Int'l Joint Research: 8 results) Funded Workshop (1 results)

  • [Int'l Joint Research] University of Indonesia (UI)/Bandung Institute of Technology (ITB)(Indonesia)

    • Related Report
      2016 Annual Research Report
  • [Int'l Joint Research] International Research Institute MICA(ベトナム)

    • Related Report
      2016 Annual Research Report
  • [Int'l Joint Research] Lab d'Informatique de Grenoble (LIG)/Laboratoire Informatique d'Avignon (LIA)(France)

    • Related Report
      2016 Annual Research Report
  • [Journal Article] Compressing Recurrent Neural Network with Tensor Train2017

    • Author(s)
      Andros Tjandra, Sakriani Sakti, Satoshi Nakamura
    • Journal Title

      Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN 2017)

      Volume: 印刷中

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Gated Recurrent Neural Tensor Network2016

    • Author(s)
      Andros Tjandra, Sakriani Sakti, Ruli Manurung, Mirna Adriani, Satoshi Nakamura
    • Journal Title

      Proceedings of The 2016 International Joint Conference on Neural Networks (IJCNN 2016)

      Volume: Vol. 1 Pages: 448-456

    • DOI

      10.1109/ijcnn.2016.7727233

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Removing Noise from Event-Related Potentials using a Probabilistic Generative Model with Grouped Covariance Matrices2016

    • Author(s)
      Hayato Maki, Tomoki Toda, Sakriani Sakti, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society

      Volume: Vol. 1 Pages: 3728-3731

    • DOI

      10.1109/embc.2016.7591538

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Unsupervised Joint Estimation of Grapheme-to-Phoneme Conversion Systems and Acoustic Model Adaptation for Non-Native Speech Recognition2016

    • Author(s)
      Satoshi Tsujioka, Sakriani Sakti, Koichiro Yoshino, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of Interspeech 2016

      Volume: Vol. 1 Pages: 3091-3095

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Supervised Learning of Acoustic Models in a Zero Resource Setting to Improve DPGMM Clustering2016

    • Author(s)
      Michael Heck, Sakriani Sakti, Satoshi Nakamura
    • Journal Title

      Proceedings of Interspeech 2016

      Volume: Vol. 1 Pages: 1310-1314

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models2016

    • Author(s)
      Quoc Truong Do, Sakriani Sakti, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of Interspeech 2016

      Volume: Vol. 1 Pages: 2533-2537

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Iterative Training of A DPGMM-HMM Acoustic Unit Recognizer in A Zero Resource Scenario2016

    • Author(s)
      Michael Heck, Sakriani Sakti, Satoshi Nakamura
    • Journal Title

      Proceedings of IEEE Spoken Language Technology Workshop (SLT 2016)

      Volume: Vol. 1 Pages: 57-63

    • DOI

      10.1109/slt.2016.7846245

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Deep Bottleneck Features And Sound-Dependent i-Vectors for Simultaneous Recognition of Speech and Environmental Sounds2016

    • Author(s)
      Sakriani Sakti, Seiji Kawanishi, Graham Neubig, Koichiro Yoshino, Satoshi Nakamura
    • Journal Title

      Proceedings of IEEE Spoken Language Technology Workshop (SLT 2016)

      Volume: Vol. 1 Pages: 35-42

    • DOI

      10.1109/slt.2016.7846242

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Int'l Joint Research / Acknowledgement Compliant
  • [Journal Article] Incongruity Detection on ASR Outputs based on EEG Signals2016

    • Author(s)
      Sakriani Sakti, Yu Odagaki, Takafumi Sasakura, Graham Neubig, Tomoki Toda, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 83-84

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] A Noise Reduction Method Using Spatial Prior of Event-Related Potentials2016

    • Author(s)
      Hayato Maki, Tomoki Toda, Sakriani Sakti, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 627-628

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] The NAIST ASR for IWSLT: A Multi-architecture DNN System Combination Approach2016

    • Author(s)
      Michael Heck, Quoc Truong Do, Sakriani Sakti, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 337-338

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] Multi-Task Deep Neural Networks for Speech and Environmental Sound Recognition2016

    • Author(s)
      Seiji Kawanishi, Sakriani Sakti, Koichiro Yoshino, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 163-164

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] Non-native Automatic Speech Recognition Utilizing Acoustic Data-driven Pronunciation Learning and Acoustic Model Adaptation2016

    • Author(s)
      Satoshi Tsujioka, Sakriani Sakti, Koichiro Yoshino, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 75-76

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] Exploring Bottleneck Features for Emotional Speech Recognition2016

    • Author(s)
      Kohei Mukaihara, Sakriani Sakti, Koichiro Yoshino, Graham Neubig, Satoshi Nakamura
    • Journal Title

      Proceedings of ASJ 2016

      Volume: Vol. 1 Pages: 161-162

    • Related Report
      2016 Annual Research Report
    • Acknowledgement Compliant
  • [Journal Article] A Study On Natural Expressive Speech: Automatic Memorable Spoken Quote Detection2015

    • Author(s)
      Fajri Koto, Sakriani Sakti, Graham Neubig, Tomoki Toda, Mirna Adriani, Satoshi Nakamura
    • Journal Title

      Springer Lecture Notes

      Volume: Vol. 1

    • Related Report
      2014 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Unknown Word Detection based on Event-Related Brain Desynchronization Responses2015

    • Author(s)
      Takafumi Sasakura, Sakriani Sakti, Graham Neubig, Tomoki Toda, Satoshi Nakamura
    • Journal Title

      Springer Lecture Notes

      Volume: Vol. 1

    • Related Report
      2014 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Combination of Two-dimensional Cochleogram and Spectrogram Features for Deep Learning- based ASR2015

    • Author(s)
      Andros Tjandra, Sakriani Sakti, Graham Neubig, Tomoki Toda, Mirna Adriani, Satoshi Nakamura
    • Journal Title

      Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2015)

      Volume: Vol. 1 Pages: 4525-4529

    • Related Report
      2014 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] COLLECTION AND ANALYSIS OF A JAPANESE-ENGLISH EMPHASIZED SPEECH CORPORA2014

    • Author(s)
      Do Quoc Truong, Sakriani Sakti, Graham Neubig, Tomoki Toda and Satoshi Nakamura
    • Journal Title

      Proc. of Oriental-COCOSDA 17t

      Volume: - Pages: 77-82

    • DOI

      10.1109/icsda.2014.7051435

    • Related Report
      2014 Research-status Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] An Event-Related Brain Potential Study on the Impact of Speech Recognition Errors2014

    • Author(s)
      Sakriani Sakti, Yu Odagaki, Takafumi Sasakura, Graham Neubig, Tomoki Toda, Satoshi Nakamura
    • Journal Title

      Proceedings of Asia Pacific Signal and Information Processing Association (APSIPA)

      Volume: Vol. 1 Pages: 1-4

    • DOI

      10.1109/apsipa.2014.7041620

    • Related Report
      2014 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] 単語視認時の脳波信号を用いた未知語知覚検出2014

    • Author(s)
      Takafumi Sasakura, Sakti Sakriani, Neubig Graham, Tomoki Toda, Satoshi Nakamura
    • Journal Title

      SIG-SLUD-B402

      Volume: Vol. 1 Pages: 57-62

    • NAID

      130008057614

    • Related Report
      2014 Research-status Report
    • Acknowledgement Compliant
  • [Presentation] Compressing Recurrent Neural Network with Tensor Train2017

    • Author(s)
      Andros Tjandra
    • Organizer
      The 2017 International Joint Conference on Neural Networks (IJCNN 2017)
    • Place of Presentation
      Anchorage, Alaska, USA
    • Year and Date
      2017-05-14
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Iterative Training of A DPGMM-HMM Acoustic Unit Recognizer in A Zero Resource Scenario2016

    • Author(s)
      Michael Heck
    • Organizer
      IEEE Spoken Language Technology Workshop (SLT 2016)
    • Place of Presentation
      San Diego, USA
    • Year and Date
      2016-12-13
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Deep Bottleneck Features And Sound-Dependent i-Vectors for Simultaneous Recognition of Speech and Environmental Sounds2016

    • Author(s)
      Sakriani Sakti
    • Organizer
      IEEE Spoken Language Technology Workshop (SLT 2016)
    • Place of Presentation
      San Diego, USA
    • Year and Date
      2016-12-13
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Unsupervised Joint Estimation of Grapheme-to-Phoneme Conversion Systems and Acoustic Model Adaptation for Non-Native Speech Recognition2016

    • Author(s)
      Satoshi Tsujioka
    • Organizer
      Interspeech
    • Place of Presentation
      San Fransisco, USA
    • Year and Date
      2016-09-08
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Supervised Learning of Acoustic Models in a Zero Resource Setting to Improve DPGMM Clustering2016

    • Author(s)
      Michael Heck
    • Organizer
      Interspeech
    • Place of Presentation
      San Fransisco, USA
    • Year and Date
      2016-09-08
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models2016

    • Author(s)
      Quoc Truong Do
    • Organizer
      Interspeech
    • Place of Presentation
      San Fransisco, USA
    • Year and Date
      2016-09-08
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Removing Noise from Event-Related Potentials using a Probabilistic Generative Model with Grouped Covariance Matrices2016

    • Author(s)
      Hayato Maki
    • Organizer
      International Conference of the IEEE Engineering in Medicine and Biology Society
    • Place of Presentation
      Orlando, Florida, USA
    • Year and Date
      2016-08-16
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Gated Recurrent Neural Tensor Network2016

    • Author(s)
      Sakriani Sakti
    • Organizer
      The 2016 International Joint Conference on Neural Networks (IJCNN 2016)
    • Place of Presentation
      Vancouver, Canada
    • Year and Date
      2016-07-24
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Deep Neural Networkを用いた音声と環境音のマルチタスク学習2016

    • Author(s)
      川西 誠司
    • Organizer
      日本音響学会 春季研究発表会
    • Place of Presentation
      桐蔭横浜大学(神奈川県横浜市)
    • Year and Date
      2016-03-09
    • Related Report
      2015 Research-status Report
  • [Presentation] 英語習熟度を考慮した発音辞書と音響モデル逐次適応による非母語音声認識2016

    • Author(s)
      辻岡 聡
    • Organizer
      日本音響学会 春季研究発表会
    • Place of Presentation
      桐蔭横浜大学(神奈川県横浜市)
    • Year and Date
      2016-03-09
    • Related Report
      2015 Research-status Report
  • [Presentation] ボトルネック特徴量を用いた感情音声認識の検討2016

    • Author(s)
      向原 康平
    • Organizer
      日本音響学会 春季研究発表会
    • Place of Presentation
      桐蔭横浜大学(神奈川県横浜市)
    • Year and Date
      2016-03-09
    • Related Report
      2015 Research-status Report
  • [Presentation] A Study of Social-Affective Communication: Automatic Prediction of Emotion Triggers and Responses in Television Talk Shows2015

    • Author(s)
      Nurul Lubis
    • Organizer
      2015 IEEE Automatic Speech Recognition and Understanding
    • Place of Presentation
      Scottsdale(米国)
    • Year and Date
      2015-12-13
    • Related Report
      2015 Research-status Report
  • [Presentation] The NAIST English Speech Recognition System for IWSLT 20152015

    • Author(s)
      Michael Heck
    • Organizer
      12th International Workshop on Spoken Language Translation (IWSLT)
    • Place of Presentation
      Da Nang (ベトナム)
    • Year and Date
      2015-12-03
    • Related Report
      2015 Research-status Report
  • [Presentation] 感情音声認識におけるCNNおよびDNNボトルネック特徴量の検討2015

    • Author(s)
      向原 康平
    • Organizer
      第109回音声言語情報処理研究会 (SIG-SLP)
    • Place of Presentation
      名古屋工業大学(愛知県名古屋市)
    • Year and Date
      2015-12-02
    • Related Report
      2015 Research-status Report
  • [Presentation] Construction and Analysis of Social-Affective Interaction Corpus in English and Indonesian2015

    • Author(s)
      Nurul Lubis
    • Organizer
      Oriental COCOSDA 2016
    • Place of Presentation
      Shanghai(中国)
    • Year and Date
      2015-10-28
    • Related Report
      2015 Research-status Report
  • [Presentation] 非母語音声の認識のための実音声を用いた発音辞書獲得2015

    • Author(s)
      辻岡 聡
    • Organizer
      第107回音声言語情報処理研究会 (SIG-SLP)
    • Place of Presentation
      上諏訪温泉 かたくら諏訪湖ホテル(長野県諏訪市)
    • Year and Date
      2015-07-16
    • Related Report
      2015 Research-status Report
  • [Presentation] ボトルネック特徴量を用いた感情音声の認識2015

    • Author(s)
      向原 康平
    • Organizer
      第107回音声言語情報処理研究会 (SIG-SLP)
    • Place of Presentation
      上諏訪温泉 かたくら諏訪湖ホテル(長野県諏訪市)
    • Year and Date
      2015-07-16
    • Related Report
      2015 Research-status Report
  • [Presentation] Combination of Two-dimensional Cochleogram and Spectrogram Features for Deep Learning-based ASR2015

    • Author(s)
      Andros Tjandra
    • Organizer
      the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2015)
    • Place of Presentation
      Brisbane, Australia
    • Year and Date
      2015-04-19 – 2015-04-25
    • Related Report
      2014 Research-status Report
  • [Presentation] A Study On Natural Expressive Speech: Automatic Memorable Spoken Quote Detection2015

    • Author(s)
      Graham Neubig
    • Organizer
      the 6th International Workshop on Spoken Dialog Systems (IWSDS)
    • Place of Presentation
      Busan, Korea
    • Year and Date
      2015-01-11 – 2015-01-13
    • Related Report
      2014 Research-status Report
  • [Presentation] Unknown Word Detection based on Event-Related Brain Desynchronization Responses2015

    • Author(s)
      Takafumi Sasakura
    • Organizer
      the 6th International Workshop on Spoken Dialog Systems (IWSDS)
    • Place of Presentation
      Busan, Korea
    • Year and Date
      2015-01-11 – 2015-01-13
    • Related Report
      2014 Research-status Report
  • [Presentation] 単語視認時の脳波信号を用いた未知語知覚検出2014

    • Author(s)
      Takafumi Sasakura
    • Organizer
      SIG-SLUD
    • Place of Presentation
      東京工業大学すずかけ台キャンパス(神奈川県横浜市)
    • Year and Date
      2014-12-15
    • Related Report
      2014 Research-status Report
  • [Presentation] An Event-Related Brain Potential Study on the Impact of Speech Recognition Errors2014

    • Author(s)
      Sakriani Sakti
    • Organizer
      Asia Pacific Signal and Information Processing Association (APSIPA)
    • Place of Presentation
      Siem Reap, Cambodia
    • Year and Date
      2014-12-09 – 2014-12-12
    • Related Report
      2014 Research-status Report
  • [Presentation] 違和感が事象関連電位に与える影響について2014

    • Author(s)
      Yu Odagaki
    • Organizer
      Japan Neuroscience
    • Place of Presentation
      パシフィコ横浜(神奈川県横浜市)
    • Year and Date
      2014-09-11 – 2014-09-13
    • Related Report
      2014 Research-status Report
  • [Presentation] Memorable Spoken Quote Corpora of TED Public Speaking2014

    • Author(s)
      Fajri Koto
    • Organizer
      the 17th Oriental COCOSDA
    • Place of Presentation
      Phuket, Thailand
    • Year and Date
      2014-09-09 – 2014-09-12
    • Related Report
      2014 Research-status Report
  • [Funded Workshop] The 5th International Workshop on Spoken Language Technologies for Under-resourced Languages (SLTU'16)2016

    • Place of Presentation
      Yogyakarta, Indonesia
    • Year and Date
      2016-05-09
    • Related Report
      2016 Annual Research Report

URL: 

Published: 2014-04-04   Modified: 2022-02-16  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi