Dense 3-axis tactile sensing and AI to implement human-like manual skills in robots

• Project/Area Number: 19H02116
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Basic Section 20020:Robotics and intelligent system-related
• Research Institution: Waseda University
• Principal Investigator: Schmitz Alexander 早稲田大学, 理工学術院, Visiting Researcher (30729455)
• Co-Investigator(Kenkyū-buntansha): 尾形 哲也 早稲田大学, 理工学術院, 教授 (00318768) ; 玉城 絵美 琉球大学, 工学部, 教授 (30515086) ; Somlor Sophon 早稲田大学, 理工学術院総合研究所(理工学研究所), 次席研究員(研究院講師) (40791231)
• Project Period (FY): 2019-04-01 – 2022-03-31
• Project Status: Completed (Fiscal Year 2022)
• Budget Amount: ¥17,420,000 (Direct Cost: ¥13,400,000、Indirect Cost: ¥4,020,000); Fiscal Year 2021: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000); Fiscal Year 2020: ¥6,240,000 (Direct Cost: ¥4,800,000、Indirect Cost: ¥1,440,000); Fiscal Year 2019: ¥6,760,000 (Direct Cost: ¥5,200,000、Indirect Cost: ¥1,560,000)
• Keywords: Robotics / Tactile Sensing / Machine Learning / Tactile Sensors / Tactile sensors / Manual skills / Robots

Outline of Research at the Start

Even the most advanced robots currently cannot robustly grasp and manipulate objects of unspecified shape, weight and stiffness. This proposal addresses these issues by developing advanced tactile sensors. Machine learning is used to replicate skills that previously could not be realized.

Outline of Final Research Achievements

We have further developed our tactile sensors. 368 3-axis tactile sensors cover the robot hand. Furthermore, we have further developed a robot hand that allows to cover the palmar side of the robot hand nearly completely with thick skin. We have also further developed the tactile sensors for human hands, which now allow measuring the contact position in addition to the contact force on the fingertip without covering the human skin that is in contact with the object. Furthermore, we have developed sensors that are stretchable, so that they can be used for robot joints for example in the future.

Moreover, we have used the tactile sensors embedded in robot hands and grippers for tactile AI. The robot was able to perform various in-hand manipulations, detect object properties, and predict grasp stability as well as detect slip. We use, implement and improve state of the art machine learning methods such as attention mechanisms, ensemble learning and graph neural networks.

Academic Significance and Societal Importance of the Research Achievements

Robots in the future are expected to work for and with humans. Tactile sensors are crucial for robots to perform human-like grasping and manipulation. Example applications would be the automation of warehouses, factories and agricultural activities.

Research Products

• [Journal Article] A Robotic Grasping State Perception Framework with Multi-Phase Tactile Information and Ensemble Learning (2022)
  • Author(s): Gang Yan, Alexander Schmitz, Satoshi Funabashi, Sophon Somlor, Tito Pradhono Tomo, Shigeki Sugano
  • Journal Title: IEEE Robotics and Automation Letters Volume: PP Issue: 3 Pages: 1-1
  • DOI: 10.1109/lra.2022.3151260
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors (2022)
  • Author(s): Satoshi Funabashi, Tomoki Isobe, Fei Hongyi, Atsumu Hiramoto, Alexander Schmitz, Shigeki Sugano, Tetsuya Ogata
  • Journal Title: IEEE Robotics and Automation Letters Volume: 7 Issue: 2 Pages: 2102-2109
  • DOI: 10.1109/lra.2022.3142417
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks (2022)
  • Author(s): Satoshi Funabashi, Gang Yan, Fei Hongyi, Alexander Schmitz, Lorenzo Jamone, Tetsuya Ogata, and Shigeki Sugano
  • Journal Title: IEEE Transactions on Neural Networks and Learning Systems Volume: PP Issue: 6 Pages: 1-15
  • DOI: 10.1109/tnnls.2022.3215723
  • Peer Reviewed / Int'l Joint Research
• [Presentation] Detection of Slip from Vision and Touch (2022)
  • Author(s): Yan, G., Schmitz, A., Tomo, T. P., Somlor, S., Funabashi, S., Sugano, S.
  • Organizer: IEEE International Conference on Robotics and Automation
  • Int'l Joint Research
• [Presentation] Detection of Slip from Vision and Touch (2022)
  • Author(s): Yan, G., Schmitz, A., Tomo, T. P., Somlor, S., Funabashi, S., Sugano, S.
  • Organizer: IEEE International Conference on Robotics and Automation (ICRA 2022)
  • Int'l Joint Research
• [Presentation] A Multi-Fingered Robot Hand with Remote Center of Motion Mechanisms for Covering Joints with Soft Skin (2021)
  • Author(s): Khullar, G., Schmitz, A., Hsu, C., Sathe, P., Funabashi, S., Sugano, S.
  • Organizer: IEEE International Conference on Robotics and Biomimetics
  • Int'l Joint Research
• [Presentation] “Safe Skin” - A Low-Cost Capacitive Proximity-Force-Fusion Sensor For Safety in Robots (2021)
  • Author(s): Wang, Z., Gao, H., Schmitz, A., Somlor, S., Tomo, T.P., Sugano, S.
  • Organizer: IEEE/RSJ International Conference on Intelligent Robots and Systems
  • Int'l Joint Research
• [Presentation] SCT-CNN: A Spatio-Channel-Temporal Attention CNN for Grasp Stability Prediction (2021)
  • Author(s): Yan, G., Schmitz, A., Funabashi, S., Somlor, S., Tomo, T.P., Sugano, S.
  • Organizer: IEEE International Conference on Robotics and Automation
  • Int'l Joint Research
• [Presentation] Implementation of a Remote Center of Motion Robot Finger with Tactile Sensors in the Joints (2019)
  • Author(s): Chincheng HSU, Alexander SCHMITZ, Gagan KHULLAR, Harris KRISTANTO, Zhen WANG, Prathamesh SATHE, Shigeki SUGANO
  • Organizer: IEEE International Conference on Robotics and Biomimetics (ROBIO 2019)
• [Presentation] Development of a 3-axis Human Fingertip Tactile Sensor with an Ortho-Planar Spring (2019)
  • Author(s): Harris Kristanto, Prathamesh Sathe, Chincheng Hsu, Alexander Schmitz, Tito Pradhono Tomo, Sophon Somlor, Shigeki Sugano
  • Organizer: IEEE International Conference on Robotics and Biomimetics (ROBIO 2019)
• [Presentation] Magnetic 3-Axis Soft and Sensitive Fingertip Sensors Integration for the iCub Humanoid Robot (2019)
  • Author(s): Alexis Carlos Holgado, Nicola Piga, Tito Pradhono Tomo, Giulia Vezzani, Alexander Schmitz, Lorenzo Natale, Shigeki Sugano
  • Organizer: IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids 2019)
• [Presentation] Sequential clustering for tactile image compression to enable direct adaptive feedback (2019)
  • Author(s): Andreas Geier, Yan Gang, Tito Pradhono Tomo, Shun Ogasa, Sophon Somlor, Alexander Schmitz, and Shigeki Sugano
  • Organizer: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
• [Presentation] Robot Finger with Remote Center of Motion Mechanism for Covering Joints with Thick Skin (2019)
  • Author(s): hincheng HSU, Alexander SCHMITZ, Kosuke KUSAYANAGI and Shigeki SUGANO
  • Organizer: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
• [Presentation] Compliant Joint Mechanism for 3-axis Human Fingertip Tactile Sensor Based on Distributed Hall Effect Sensors (2019)
  • Author(s): Prathamesh Sathe, Harris Kristanto, Alexander Schmitz, Tito Tomo Pradhono, Sophon Somlor, Shigeki Sugano
  • Organizer: 37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
• [Presentation] 指固有のCNNを介する3軸触覚情報を利用した未学習物体の操り動作生成手法の提案 (2019)
  • Author(s): 磯部 智喜, 船橋 賢, 小笠 駿, 尾形 哲也, シュミッツ アレクサンダー, トモ ティト, 菅野 重樹
  • Organizer: 37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
• [Presentation] タスクと触覚情報を用いた単一学習器による複数操り動作の実現 (2019)
  • Author(s): 小笠 駿, 船橋 賢, 磯部 智喜, 尾形 哲也, アレクサンダー シュミッツ, ティト トモ, 菅野 重樹
  • Organizer: 37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
• [Presentation] 指列動作に着目した神経回路モデルの段階的学習と多指ハンドの様々な物体の操り (2019)
  • Author(s): 船橋 賢, 小笠 駿, ケン オー, アレクサンダー シュミッツ, 菅野 重樹
  • Organizer: 37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
• [Presentation] 多指ロボットハンドの触覚センサの位置・サイズを考慮したCNN構造の解析 (2019)
  • Author(s): 延 剛, 船橋 賢, アレクサンダー シュミッツ, 尾形 哲也, 菅野 重樹
  • Organizer: 37th Annual Conference of the Robotics Society of Japan (RSJ 2019)
• [Presentation] 3D Tactile-Driven Two-Fingered In-Hand Manipulation with Finger-Specific CNN (2019)
  • Author(s): Tomoki ISOBE, Satoshi FUNABASHI, Shun OGASA, Tetsuya OGATA, Alexander SCHMITZ, Tito Pradhono TOMO, Shigeki SUGANO
  • Organizer: JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
• [Presentation] Multi-In-Hand Manipulation for 3D Tactile Based Robot Hand Using CNNLSTM (2019)
  • Author(s): Satoshi FUNABASHI, Shun OGASA, Tomoki ISOBE, Tetsuya OGATA, Alexander SCHMITZ, Tito Pradhono TOMO, Shigeki SUGANO
  • Organizer: JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
• [Presentation] Object Recognition with 3D Tactile Based Multi-Fingered Hand Using Morphology-Specific Convolutional Neural Networks (2019)
  • Author(s): Satoshi FUNABASHI, Yang Gan, Andreas Geier, Alexander SCHMITZ, 尾形 哲也, Shigeki SUGANO
  • Organizer: JSME annual Conference on Robotics and Mechatronics (Robomec 2019)
• [Presentation] In-Hand Manipulation with a Four-Fingered Hand via Finger-Specific Stepwise Learning (2019)
  • Author(s): Satoshi FUNABASHI, Alexander SCHMITZ, Shun OGASA, Shigeki SUGANO
  • Organizer: JSME annual Conference on Robotics and Mechatronics (Robomec 2019)