• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Dense 3-axis tactile sensing and AI to implement human-like manual skills in robots

Research Project

Project/Area Number 19H02116
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 20020:Robotics and intelligent system-related
Research InstitutionWaseda University

Principal Investigator

Schmitz Alexander  早稲田大学, 理工学術院, Visiting Researcher (30729455)

Co-Investigator(Kenkyū-buntansha) 尾形 哲也  早稲田大学, 理工学術院, 教授 (00318768)
玉城 絵美  琉球大学, 工学部, 教授 (30515086)
Somlor Sophon  早稲田大学, 理工学術院総合研究所(理工学研究所), 次席研究員(研究院講師) (40791231)
Project Period (FY) 2019-04-01 – 2022-03-31
Project Status Completed (Fiscal Year 2022)
Budget Amount *help
¥17,420,000 (Direct Cost: ¥13,400,000、Indirect Cost: ¥4,020,000)
Fiscal Year 2021: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2020: ¥6,240,000 (Direct Cost: ¥4,800,000、Indirect Cost: ¥1,440,000)
Fiscal Year 2019: ¥6,760,000 (Direct Cost: ¥5,200,000、Indirect Cost: ¥1,560,000)
KeywordsRobotics / Tactile Sensing / Machine Learning / Tactile Sensors / Tactile sensors / Manual skills / Robots
Outline of Research at the Start

Even the most advanced robots currently cannot robustly grasp and manipulate objects of unspecified shape, weight and stiffness. This proposal addresses these issues by developing advanced tactile sensors. Machine learning is used to replicate skills that previously could not be realized.

Outline of Final Research Achievements

We have further developed our tactile sensors. 368 3-axis tactile sensors cover the robot hand. Furthermore, we have further developed a robot hand that allows to cover the palmar side of the robot hand nearly completely with thick skin. We have also further developed the tactile sensors for human hands, which now allow measuring the contact position in addition to the contact force on the fingertip without covering the human skin that is in contact with the object. Furthermore, we have developed sensors that are stretchable, so that they can be used for robot joints for example in the future.

Moreover, we have used the tactile sensors embedded in robot hands and grippers for tactile AI. The robot was able to perform various in-hand manipulations, detect object properties, and predict grasp stability as well as detect slip. We use, implement and improve state of the art machine learning methods such as attention mechanisms, ensemble learning and graph neural networks.

Academic Significance and Societal Importance of the Research Achievements

Robots in the future are expected to work for and with humans. Tactile sensors are crucial for robots to perform human-like grasping and manipulation. Example applications would be the automation of warehouses, factories and agricultural activities.

Report

(4 results)
  • 2022 Final Research Report ( PDF )
  • 2021 Annual Research Report
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • Research Products

    (22 results)

All 2022 2021 2019

All Journal Article (3 results) (of which Int'l Joint Research: 3 results,  Peer Reviewed: 3 results) Presentation (19 results) (of which Int'l Joint Research: 5 results)

  • [Journal Article] A Robotic Grasping State Perception Framework with Multi-Phase Tactile Information and Ensemble Learning2022

    • Author(s)
      Gang Yan, Alexander Schmitz, Satoshi Funabashi, Sophon Somlor, Tito Pradhono Tomo, Shigeki Sugano
    • Journal Title

      IEEE Robotics and Automation Letters

      Volume: PP Issue: 3 Pages: 1-1

    • DOI

      10.1109/lra.2022.3151260

    • Related Report
      2021 Annual Research Report 2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors2022

    • Author(s)
      Satoshi Funabashi, Tomoki Isobe, Fei Hongyi, Atsumu Hiramoto, Alexander Schmitz, Shigeki Sugano, Tetsuya Ogata
    • Journal Title

      IEEE Robotics and Automation Letters

      Volume: 7 Issue: 2 Pages: 2102-2109

    • DOI

      10.1109/lra.2022.3142417

    • Related Report
      2021 Annual Research Report 2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks2022

    • Author(s)
      Satoshi Funabashi, Gang Yan, Fei Hongyi, Alexander Schmitz, Lorenzo Jamone, Tetsuya Ogata, and Shigeki Sugano
    • Journal Title

      IEEE Transactions on Neural Networks and Learning Systems

      Volume: PP Issue: 6 Pages: 1-15

    • DOI

      10.1109/tnnls.2022.3215723

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Presentation] Detection of Slip from Vision and Touch2022

    • Author(s)
      Yan, G., Schmitz, A., Tomo, T. P., Somlor, S., Funabashi, S., Sugano, S.
    • Organizer
      IEEE International Conference on Robotics and Automation
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Detection of Slip from Vision and Touch2022

    • Author(s)
      Yan, G., Schmitz, A., Tomo, T. P., Somlor, S., Funabashi, S., Sugano, S.
    • Organizer
      IEEE International Conference on Robotics and Automation (ICRA 2022)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Multi-Fingered Robot Hand with Remote Center of Motion Mechanisms for Covering Joints with Soft Skin2021

    • Author(s)
      Khullar, G., Schmitz, A., Hsu, C., Sathe, P., Funabashi, S., Sugano, S.
    • Organizer
      IEEE International Conference on Robotics and Biomimetics
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] “Safe Skin” - A Low-Cost Capacitive Proximity-Force-Fusion Sensor For Safety in Robots2021

    • Author(s)
      Wang, Z., Gao, H., Schmitz, A., Somlor, S., Tomo, T.P., Sugano, S.
    • Organizer
      IEEE/RSJ International Conference on Intelligent Robots and Systems
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] SCT-CNN: A Spatio-Channel-Temporal Attention CNN for Grasp Stability Prediction2021

    • Author(s)
      Yan, G., Schmitz, A., Funabashi, S., Somlor, S., Tomo, T.P., Sugano, S.
    • Organizer
      IEEE International Conference on Robotics and Automation
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Implementation of a Remote Center of Motion Robot Finger with Tactile Sensors in the Joints2019

    • Author(s)
      Chincheng HSU, Alexander SCHMITZ, Gagan KHULLAR, Harris KRISTANTO, Zhen WANG, Prathamesh SATHE, Shigeki SUGANO
    • Organizer
      IEEE International Conference on Robotics and Biomimetics (ROBIO 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Development of a 3-axis Human Fingertip Tactile Sensor with an Ortho-Planar Spring2019

    • Author(s)
      Harris Kristanto, Prathamesh Sathe, Chincheng Hsu, Alexander Schmitz, Tito Pradhono Tomo, Sophon Somlor, Shigeki Sugano
    • Organizer
      IEEE International Conference on Robotics and Biomimetics (ROBIO 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Magnetic 3-Axis Soft and Sensitive Fingertip Sensors Integration for the iCub Humanoid Robot2019

    • Author(s)
      Alexis Carlos Holgado, Nicola Piga, Tito Pradhono Tomo, Giulia Vezzani, Alexander Schmitz, Lorenzo Natale, Shigeki Sugano
    • Organizer
      IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Sequential clustering for tactile image compression to enable direct adaptive feedback2019

    • Author(s)
      Andreas Geier, Yan Gang, Tito Pradhono Tomo, Shun Ogasa, Sophon Somlor, Alexander Schmitz, and Shigeki Sugano
    • Organizer
      IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Robot Finger with Remote Center of Motion Mechanism for Covering Joints with Thick Skin2019

    • Author(s)
      hincheng HSU, Alexander SCHMITZ, Kosuke KUSAYANAGI and Shigeki SUGANO
    • Organizer
      IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Compliant Joint Mechanism for 3-axis Human Fingertip Tactile Sensor Based on Distributed Hall Effect Sensors2019

    • Author(s)
      Prathamesh Sathe, Harris Kristanto, Alexander Schmitz, Tito Tomo Pradhono, Sophon Somlor, Shigeki Sugano
    • Organizer
      37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 指固有のCNNを介する3軸触覚情報を利用した未学習物体の操り動作生成手法の提案2019

    • Author(s)
      磯部 智喜, 船橋 賢, 小笠 駿, 尾形 哲也, シュミッツ アレクサンダー, トモ ティト, 菅野 重樹
    • Organizer
      37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] タスクと触覚情報を用いた単一学習器による複数操り動作の実現2019

    • Author(s)
      小笠 駿, 船橋 賢, 磯部 智喜, 尾形 哲也, アレクサンダー シュミッツ, ティト トモ, 菅野 重樹
    • Organizer
      37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 指列動作に着目した神経回路モデルの段階的学習と多指ハンドの様々な物体の操り2019

    • Author(s)
      船橋 賢, 小笠 駿, ケン オー, アレクサンダー シュミッツ, 菅野 重樹
    • Organizer
      37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 多指ロボットハンドの触覚センサの位置・サイズを考慮したCNN構造の解析2019

    • Author(s)
      延 剛, 船橋 賢, アレクサンダー シュミッツ, 尾形 哲也, 菅野 重樹
    • Organizer
      37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 3D Tactile-Driven Two-Fingered In-Hand Manipulation with Finger-Specific CNN2019

    • Author(s)
      Tomoki ISOBE, Satoshi FUNABASHI, Shun OGASA, Tetsuya OGATA, Alexander SCHMITZ, Tito Pradhono TOMO, Shigeki SUGANO
    • Organizer
      JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Multi-In-Hand Manipulation for 3D Tactile Based Robot Hand Using CNNLSTM2019

    • Author(s)
      Satoshi FUNABASHI, Shun OGASA, Tomoki ISOBE, Tetsuya OGATA, Alexander SCHMITZ, Tito Pradhono TOMO, Shigeki SUGANO
    • Organizer
      JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Object Recognition with 3D Tactile Based Multi-Fingered Hand Using Morphology-Specific Convolutional Neural Networks2019

    • Author(s)
      Satoshi FUNABASHI, Yang Gan, Andreas Geier, Alexander SCHMITZ, 尾形 哲也, Shigeki SUGANO
    • Organizer
      JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] In-Hand Manipulation with a Four-Fingered Hand via Finger-Specific Stepwise Learning2019

    • Author(s)
      Satoshi FUNABASHI, Alexander SCHMITZ, Shun OGASA, Shigeki SUGANO
    • Organizer
      JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
    • Related Report
      2019 Annual Research Report

URL: 

Published: 2019-04-18   Modified: 2024-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi