• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

From "Exploration" To "Thinking" - Development of Chaos Dynamics through Reinforcement Learning

Research Project

Project/Area Number 15K00360
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Research Field Intelligent robotics
Research InstitutionOita University

Principal Investigator

SHIBATA Katsunari  大分大学, 理工学部, 教授 (10260522)

Project Period (FY) 2015-04-01 – 2020-03-31
Project Status Completed (Fiscal Year 2019)
Budget Amount *help
¥4,680,000 (Direct Cost: ¥3,600,000、Indirect Cost: ¥1,080,000)
Fiscal Year 2019: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Fiscal Year 2018: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Fiscal Year 2017: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2016: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2015: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Keywordsダイナミック強化学習 / 感度 / 感度調整学習 / カオスニューラルネット / 思考 / 探索 / ダイナミクス / 汎用人工知能 / ニューラルネット / カオスダイナミクス / リアプノフ指数 / 勾配消失問題 / 深層学習 / 強化学習 / 決定論的探索 / 内部ダイナミクス / カオス生成・維持学習 / 因果トレース / 誤差逆伝播法 / 不応性 / 障害物回避 / 視覚センサ / カオスニューロン / リザバー / BPWT / 符号付き因果トレース / 高次探索
Outline of Final Research Achievements

I could not reach the initial goal that is to establish the algorithm of reinforcement learning using a chaos neural network (NN), which I have proposed, and then the emergence of “primitive thinking” on the basis of the hypothesis that “exploration” grows into “thinking” through reinforcement learning.
On the other hand, I have proposed an index “sensitivity” in each neuron to control the chaoticity of the network globally, and also “sensitivity adjustment learning” to learn it. It can be used as an index for generating chaos, and it can also be used to solve the vanishing/exploding gradient problem in gradient-based learning. Furthermore, completely new reinforcement learning named “Dynamic Reinforcement Learning” in which the present output value is not learned directly but dynamics is learned by adjusting the sensitivity according to TD error (the difference of actual state value from its prediction), has come up.

Academic Significance and Societal Importance of the Research Achievements

各ニューロンのローカルな指標「感度」でニューラルネット全体のダイナミクスを制御すること,さらに,従来の静的な発想に基づく「現在の出力値を目的のものに近づける」ための学習という考えから脱し,動的な処理の学習に向けた「評価が良い場合は再現性を上げるためダイナミクスを収束へ,悪い場合は探索を強化するためダイナミクスを発散(カオス)へ」という「ダイナミック強化学習」は,全く新しい学習パラダイムを切り拓くものである。今後,より高次な機能が求められるに従い,求められるものが静的なものからより動的なものへとシフトしていく中で,学習の新たな根本原理としての役割を担うポテンシャルを持っていると期待している。

Report

(6 results)
  • 2019 Annual Research Report   Final Research Report ( PDF )
  • 2018 Research-status Report
  • 2017 Research-status Report
  • 2016 Research-status Report
  • 2015 Research-status Report
  • Research Products

    (49 results)

All 2019 2018 2017 2016 2015

All Journal Article (23 results) (of which Peer Reviewed: 11 results,  Acknowledgement Compliant: 7 results) Presentation (26 results) (of which Int'l Joint Research: 10 results,  Invited: 2 results)

  • [Journal Article] Learning Time Constant of Continuous-Time Neurons with Gradient Descent2019

    • Author(s)
      Matsuki Toshitaka, Shibata Katsunari
    • Journal Title

      In: P. P. Abdul Majeed A., Mat-Jizat J., Hassan M., Taha Z., Choi H., Kim J. (eds) RITA 2018. Lecture Notes in Mechanical Engineering. Springer

      Volume: - Pages: 149-159

    • DOI

      10.1007/978-981-13-8323-6_13

    • ISBN
      9789811383229, 9789811383236
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Chaos-Based Reinforcement Learning When Introducing Refractoriness in Each Neuron2019

    • Author(s)
      Katsuki Sato and Katsunari Shibata
    • Journal Title

      In: Kim JH., Myung H., Lee SM. (eds) Robot Intelligence Technology and Applications. RiTA 2018. Communications in Computer and Information Science

      Volume: 1015 Pages: 76-84

    • DOI

      10.1007/978-981-13-7780-8_7

    • ISBN
      9789811377792, 9789811377808
    • Related Report
      2019 Annual Research Report 2018 Research-status Report
    • Peer Reviewed
  • [Journal Article] リカレントネットにおける感度調整学習でのカオスダイナミクスの生成と維持2019

    • Author(s)
      徳丸侑輝, 柴田克成
    • Journal Title

      第38回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 75-78

    • Related Report
      2019 Annual Research Report
  • [Journal Article] 多層リードアウトを持つリザバを用いた強化学習におけるネットワーク構造の検討2019

    • Author(s)
      吉岡晴海, 松木俊貴, 柴田克成
    • Journal Title

      第38回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 79-82

    • Related Report
      2019 Annual Research Report
  • [Journal Article] カオスニューラルネットを用いた記憶タスクの強化学習におけるカオス性の影響2019

    • Author(s)
      大石将人, 柴田克成
    • Journal Title

      第38回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 83-86

    • Related Report
      2019 Annual Research Report
  • [Journal Article] 多層ニューラルネットにおける勾配消失問題解決法としての感度調整学習2019

    • Author(s)
      長谷部圭亮, 柴田克成
    • Journal Title

      第38回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 87-90

    • Related Report
      2019 Annual Research Report
  • [Journal Article] Reinforcement Learning of a Memory Task Using an Echo State Network with Multi-layer Readout2018

    • Author(s)
      Matsuki Toshitaka、Shibata Katsunari
    • Journal Title

      In: Kim JH. et al. (eds) Robot Intelligence Technology and Applications 5. RiTA 2017. Advances in Intelligent Systems and Computing

      Volume: 751 Pages: 17-26

    • DOI

      10.1007/978-3-319-78452-6_2

    • ISBN
      9783319784519, 9783319784526
    • Related Report
      2018 Research-status Report
    • Peer Reviewed
  • [Journal Article] Functions that Emerge through End-to-end Reinforcement Learning - The Direction for Artificial General Intelligence -2017

    • Author(s)
      Katsunari Shibata
    • Journal Title

      Proc. of The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17

      Volume: -

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] Communications that Emerge through Reinforcement Learning Using a (Recurrent) Neural Network2017

    • Author(s)
      Katsunari Shibata
    • Journal Title

      Proc. of The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17

      Volume: -

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] New Reinforcement Learning Using a Chaotic Neural Network for Emergence of “Thinking”2017

    • Author(s)
      Katsunari Shibata
    • Journal Title

      Proc. of The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17

      Volume: -

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] Influence of the Chaotic Property on Reinforcement Learning Using a Chaotic Neural Network2017

    • Author(s)
      Yuki Goto, Katsunari Shibata
    • Journal Title

      Proc. of Int'l Conf. on Neural Information Processing (ICONIP)2017, LNCS 10634

      Volume: - Pages: 759-767

    • DOI

      10.1007/978-3-319-70087-8_78

    • ISBN
      9783319700861, 9783319700878
    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] カオスニューラルネットを用いた強化学習におけるカオス性の影響2017

    • Author(s)
      後藤祐樹, 柴田克成
    • Journal Title

      第36回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 97-100

    • Related Report
      2017 Research-status Report
  • [Journal Article] カオスニューラルネットの内部ダイナミクスを利用した記憶タスクの報酬に基づく学習2017

    • Author(s)
      松木俊貴, 柴田克成
    • Journal Title

      第36回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 101-104

    • Related Report
      2017 Research-status Report
  • [Journal Article] カオスニューラルネットを用いた強化学習における不応性を有するカオスニューロンの導入2017

    • Author(s)
      佐藤克樹, 後藤祐樹, 柴田克成
    • Journal Title

      第36回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 109-112

    • Related Report
      2017 Research-status Report
  • [Journal Article] ニューラルネットワークを使った強化学習による行動学習を通した音声認識機能の創発2017

    • Author(s)
      江越正大, 柴田克成
    • Journal Title

      第36回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 129-132

    • Related Report
      2017 Research-status Report
  • [Journal Article] 深層学習が示唆するend-to-end強化学習に基づく機能創発アプローチの重要性と 思考の創発に向けたカオスニューラルネットを用いた新しい強化学習2017

    • Author(s)
      柴田克成,後藤祐樹
    • Journal Title

      認知科学

      Volume: 24巻,1号 Pages: 96-117

    • NAID

      130006038535

    • Related Report
      2016 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Emergence of Higher Exploration in Reinforcement Learning Using a Chaotic Neural Network2016

    • Author(s)
      Yuki Goto and Katsunari Shibata
    • Journal Title

      Proc. of Int'l Conf. on Neural Information Processing (ICONIP)2016, LNCS 9947

      Volume: - Pages: 40-48

    • DOI

      10.1007/978-3-319-46687-3_5

    • ISBN
      9783319466866, 9783319466873
    • Related Report
      2016 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] Reward-Based Learning of a Memory-Required Task Based on the Internal Dynamics of a Chaotic Neural Network2016

    • Author(s)
      Toshitaka Matsuki and Katsunari Shibata
    • Journal Title

      Proc. of Int'l Conf. on Neural Information Processing (ICONIP)2016, LNCS 9947

      Volume: - Pages: 376-383

    • DOI

      10.1007/978-3-319-46687-3_42

    • ISBN
      9783319466866, 9783319466873
    • Related Report
      2016 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] カオスニューラルネットを用いた強化学習における高次探索の創発2016

    • Author(s)
      後藤祐樹,柴田克成
    • Journal Title

      第32回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 120-123

    • Related Report
      2016 Research-status Report
    • Acknowledgement Compliant
  • [Journal Article] カオスニューラルネットの内部ダイナミクスを利用した記憶タスクの報酬に基づく学習2016

    • Author(s)
      松木俊貴,柴田克成
    • Journal Title

      第32回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 124-127

    • Related Report
      2016 Research-status Report
    • Acknowledgement Compliant
  • [Journal Article] BPWT を用いたリカレントネットの教師あり学習における相互結合部の重み値の影響2016

    • Author(s)
      山本一真,柴田克成
    • Journal Title

      第32回計測自動制御学会九州支部学術講演会予稿集

      Volume: - Pages: 128-131

    • Related Report
      2016 Research-status Report
    • Acknowledgement Compliant
  • [Journal Article] Reinforcement Learning with Internal-Dynamics-based Exploration Using a Chaotic Neural Network2015

    • Author(s)
      Katsunari Shibata and Yuta Sakashita
    • Journal Title

      Proc. of Int'l Joint Conf. on Neural Networks (IJCNN)2015

      Volume: -

    • Related Report
      2015 Research-status Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] 動的ニューロンモデルを用いたニューラルネットワークへの因果トレースの適用2015

    • Author(s)
      山本一真,柴田克成
    • Journal Title

      第25回インテリジェント・システム・シンポジウム論文集

      Volume: - Pages: 136-141

    • Related Report
      2015 Research-status Report
  • [Presentation] リカレントネットにおける感度調整学習でのカオスダイナミクスの生成と維持2019

    • Author(s)
      徳丸侑輝, 柴田克成
    • Organizer
      第38回計測自動制御学会九州支部学術講演会
    • Related Report
      2019 Annual Research Report
  • [Presentation] 多層リードアウトを持つリザバを用いた強化学習におけるネットワーク構造の検討2019

    • Author(s)
      吉岡晴海, 松木俊貴, 柴田克成
    • Organizer
      第38回計測自動制御学会九州支部学術講演会
    • Related Report
      2019 Annual Research Report
  • [Presentation] カオスニューラルネットを用いた記憶タスクの強化学習におけるカオス性の影響2019

    • Author(s)
      大石将人, 柴田克成
    • Organizer
      第38回計測自動制御学会九州支部学術講演会
    • Related Report
      2019 Annual Research Report
  • [Presentation] 多層ニューラルネットにおける勾配消失問題解決法としての感度調整学習2019

    • Author(s)
      長谷部圭亮, 柴田克成
    • Organizer
      第38回計測自動制御学会九州支部学術講演会
    • Related Report
      2019 Annual Research Report
  • [Presentation] End-to-End強化学習による知能創発と「思考」創発へ向けた新しい強化学習2018

    • Author(s)
      柴田克成
    • Organizer
      第62回システム制御情報学会研究発表講演会
    • Related Report
      2018 Research-status Report
    • Invited
  • [Presentation] Learning Time Constant of Continuous-Time Neurons with Gradient Descent2018

    • Author(s)
      Toshitaka Matsuki & Katsunari Shibata
    • Organizer
      The 6th Int'l Conf. on Robot Intelligent Technology and Applications
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Presentation] Chaos-based Reinforcement Learning when Introducing Refractoriness in Each Neuron2018

    • Author(s)
      Katsuki Sato & Katsunari Shibata
    • Organizer
      The 6th Int'l Conf. on Robot Intelligent Technology and Applications
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Presentation] 勾配法による動的なニューロンモデルの時定数の学習2018

    • Author(s)
      松木俊貴,柴田克成
    • Organizer
      第37回計測自動制御学会九州支部学術講演会
    • Related Report
      2018 Research-status Report
  • [Presentation] カオスベース強化学習への誤差逆伝播法の適用2018

    • Author(s)
      佐藤克樹,柴田克成
    • Organizer
      第37回計測自動制御学会九州支部学術講演会
    • Related Report
      2018 Research-status Report
  • [Presentation] Reinforcement Learning of a Memory Task using an Echo State Network with Multi-Layer Readout2017

    • Author(s)
      Toshitaka Matsuki, Katsunari Shibata
    • Organizer
      The Fifth Int'l Conf. on Robot Intelligence Technology and Applications
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] Influence of the Chaotic Property on Reinforcement Learning Using a Chaotic Neural Network2017

    • Author(s)
      Yuki Goto, Katsunari Shibata
    • Organizer
      Int'l Conf. on Neural Information Processing (ICONIP)2017
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] New Reinforcement Learning Using a Chaotic Neural Network for Emergence of “Thinking”2017

    • Author(s)
      Katsunari Shibata
    • Organizer
      The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] Communications that Emerge through Reinforcement Learning Using a (Recurrent) Neural Network2017

    • Author(s)
      Katsunari Shibata
    • Organizer
      The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] Functions that Emerge through End-to-end Reinforcement Learning - The Direction for Artificial General Intelligence -2017

    • Author(s)
      Katsunari Shibata
    • Organizer
      The 3rd Multidiscipliary Conf. on Reinforcement Learning and Decision Making (RLDM)17
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] カオスニューラルネットを用いた強化学習におけるカオス性の影響2017

    • Author(s)
      後藤祐樹, 柴田克成
    • Organizer
      第36回計測自動制御学会九州支部学術講演会
    • Related Report
      2017 Research-status Report
  • [Presentation] カオスニューラルネットの内部ダイナミクスを利用した記憶タスクの報酬に基づく学習2017

    • Author(s)
      松木俊貴, 柴田克成
    • Organizer
      第36回計測自動制御学会九州支部学術講演会
    • Related Report
      2017 Research-status Report
  • [Presentation] カオスニューラルネットを用いた強化学習における不応性を有するカオスニューロンの導入2017

    • Author(s)
      佐藤克樹, 後藤祐樹, 柴田克成
    • Organizer
      第36回計測自動制御学会九州支部学術講演会
    • Related Report
      2017 Research-status Report
  • [Presentation] ニューラルネットワークを使った強化学習による行動学習を通した音声認識機能の創発2017

    • Author(s)
      江越正大, 柴田克成
    • Organizer
      第36回計測自動制御学会九州支部学術講演会
    • Related Report
      2017 Research-status Report
  • [Presentation] Emergence of Intelligence through End-to-End Reinforcement Learning and Chaos-based Reinforcement Learning towards the Emergence of "Thinking"2017

    • Author(s)
      Katsunari Shibata
    • Organizer
      OIST(Okinawa Institute of Science and Technology ) Seminar
    • Related Report
      2017 Research-status Report
    • Invited
  • [Presentation] カオスニューラルネットを用いた強化学習における高次探索の創発2016

    • Author(s)
      後藤祐樹,柴田克成
    • Organizer
      第32回計測自動制御学会九州支部学術講演会
    • Place of Presentation
      佐賀大学(佐賀県佐賀市)
    • Year and Date
      2016-11-26
    • Related Report
      2016 Research-status Report
  • [Presentation] カオスニューラルネットの内部ダイナミクスを利用した記憶タスクの報酬に基づく学習2016

    • Author(s)
      松木俊貴,柴田克成
    • Organizer
      第32回計測自動制御学会九州支部学術講演会
    • Place of Presentation
      佐賀大学(佐賀県佐賀市)
    • Year and Date
      2016-11-26
    • Related Report
      2016 Research-status Report
  • [Presentation] BPWT を用いたリカレントネットの教師あり学習における相互結合部の重み値の影響2016

    • Author(s)
      山本一真,柴田克成
    • Organizer
      第32回計測自動制御学会九州支部学術講演会
    • Place of Presentation
      佐賀大学(佐賀県佐賀市)
    • Year and Date
      2016-11-26
    • Related Report
      2016 Research-status Report
  • [Presentation] Emergence of Higher Exploration in Reinforcement Learning Using a Chaotic Neural Network2016

    • Author(s)
      Yuki Goto and Katsunari Shibata
    • Organizer
      Int'l Conf. on Neural Information Processing (ICONIP) 2016
    • Place of Presentation
      Kyoto University (Kyoto, Japan)
    • Year and Date
      2016-10-16
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] Reward-Based Learning of a Memory-Required Task Based on the Internal Dynamics of a Chaotic Neural Network2016

    • Author(s)
      Toshitaka Matsuki and Katsunari Shibata
    • Organizer
      Int'l Conf. on Neural Information Processing (ICONIP) 2016
    • Place of Presentation
      Kyoto University (Kyoto, Japan)
    • Year and Date
      2016-10-16
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] 動的ニューロンモデルを用いたニューラルネットへの因果トレースの適用2015

    • Author(s)
      山本一真,柴田克成
    • Organizer
      第25回インテリジェント・システム・シンポジウム (FAN2015)
    • Place of Presentation
      東北大学 (宮城県,仙台市)
    • Year and Date
      2015-09-25
    • Related Report
      2015 Research-status Report
  • [Presentation] Reinforcement Learning with Internal-Dynamics-based Exploration Using a Chaotic Neural Network2015

    • Author(s)
      Katsunari Shibata and Yuta Sakashita
    • Organizer
      Int'l Joint Conf. on Neural Networks (IJCNN) 2015
    • Place of Presentation
      Killarney Convention Center (Ireland, Killarney)
    • Year and Date
      2015-07-13
    • Related Report
      2015 Research-status Report
    • Int'l Joint Research

URL: 

Published: 2015-04-16   Modified: 2021-02-19  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi