• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Emergent process modeling of fingering and hard-and-soft object handling based on visual-haptic fusion

Research Project

Project/Area Number 18H03313
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 61050:Intelligent robotics-related
Research InstitutionRitsumeikan University

Principal Investigator

Nobutaka Shimada  立命館大学, 情報理工学部, 教授 (10294034)

Co-Investigator(Kenkyū-buntansha) 野間 春生  立命館大学, 情報理工学部, 教授 (00374108)
李 周浩  立命館大学, 情報理工学部, 教授 (80366434)
松尾 直志  立命館大学, 情報理工学部, 助教 (80449545)
Project Period (FY) 2018-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥15,990,000 (Direct Cost: ¥12,300,000、Indirect Cost: ¥3,690,000)
Fiscal Year 2020: ¥3,510,000 (Direct Cost: ¥2,700,000、Indirect Cost: ¥810,000)
Fiscal Year 2019: ¥5,330,000 (Direct Cost: ¥4,100,000、Indirect Cost: ¥1,230,000)
Fiscal Year 2018: ¥7,150,000 (Direct Cost: ¥5,500,000、Indirect Cost: ¥1,650,000)
Keywords深層学習 / プロセスモデル / 把持パターン想起 / 視触覚統合 / 多指ハンド / 動作生成 / 他指ハンド
Outline of Final Research Achievements

We have developed a framework for automatic process modeling of object manipulation by robotic hand that integrates tactile and visual deformation information in a machine learning framework. We proposed (1) a transitional description model of 3-D shape of the manipulated object that sequentially considers uncertain and imcomplete observations, (2) a ROS2-based distributed system for logging human activities in daily life, (3) a method for creating a deep model that recalls the grasping posture based on the object shape and appearance, (4) a method for creating a deep model that recursively recalls the next operation to be taken to achieve a goal based on the current object state, and (5) Development of an ultra-compact tactile sensor and classification of tracing tactile sensations based on dynamical tactile information.

Academic Significance and Societal Importance of the Research Achievements

人が日常普通に行う道具の操り操作は現状のロボットにはまだ模倣ができない。その困難を視触覚の統合によるロボット制御によって克服するべく、触覚側では素材識別のためのなぞり行動を素材適応的に生成することで識別性能を向上できる可能性を示し、視覚側では物体形状の記述子、把持姿勢との関連モデリングによる把持想起、巨視的手順のモデリングと模倣といった要素技術を獲得した。これにより本課題の本丸である視触覚統合モデリング課題への取り組みが端緒についた。

Report

(4 results)
  • 2021 Final Research Report ( PDF )
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • Research Products

    (58 results)

All 2022 2021 2020 2019 2018

All Journal Article (4 results) (of which Peer Reviewed: 4 results,  Open Access: 2 results) Presentation (54 results) (of which Int'l Joint Research: 24 results)

  • [Journal Article] Recalling of multiple grasping methods from an object image with a convolutional neural network2021

    • Author(s)
      Sanada Makoto、Matsuo Tadashi、Shimada Nobutaka、Shirai Yoshiaki
    • Journal Title

      ROBOMECH Journal

      Volume: 8 Issue: 1

    • DOI

      10.1186/s40648-021-00206-4

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society2021

    • Author(s)
      Taniguchi Tadahiro、El Hafi Lotfi、Hagiwara Yoshinobu、Taniguchi Akira、Shimada Nobutaka、Nishiura Takanobu
    • Journal Title

      Advanced Robotics

      Volume: 35 Issue: 11 Pages: 664-674

    • DOI

      10.1080/01691864.2021.1928552

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Rain Streaks and Snowflakes Removal for Video Sequences via Motion Compensation and Matrix Completion2020

    • Author(s)
      Zhou Yutong、Shimada Nobutaka
    • Journal Title

      SN Computer Science

      Volume: 1 Issue: 6

    • DOI

      10.1007/s42979-020-00333-6

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Multi-Scale Affined-HOF and Dimension Selection for View-Unconstrained Action Recognition2020

    • Author(s)
      Dinh Tuan Tran, Hirotake Yamazoe, Joo-Ho Lee
    • Journal Title

      Applied Intelligence

      Volume: Vol.50 Issue: 5 Pages: 1-19

    • DOI

      10.1007/s10489-019-01572-8

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Presentation] MEMS 触覚センサを用いたフォークの把持力に基づく食材判別へのリザバーコンピューティングの応用2022

    • Author(s)
      坪倉 奏太, 安藤 潤人, 北野 勝則, 野間 春生, 寒川 雅之, 秋田 純一
    • Organizer
      ハプティクス研究委員会第28回研究会
    • Related Report
      2020 Annual Research Report
  • [Presentation] Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model2021

    • Author(s)
      Yutong Zhou and Nobutaka Shimada
    • Organizer
      IEEE International Conference on Automatic Face and Gesture Recognition 2021 (FG2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Scene Descriptor Expressing Ambiguity in Information Recovery Based on Incomplete Partial Observation2021

    • Author(s)
      Takaaki FUKUI, Tadashi MATSUO and Nobutaka SHIMADA
    • Organizer
      2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] ROS2-based Distributed System Implementation for Logging Indoor Human Activities2021

    • Author(s)
      Kyohei Yoshida, Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      International Conference on Intelligent Computing (ICIC2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Non-tactile Thumb Tip Measurement System for Encouraging Rehabilitation After Surgery2021

    • Author(s)
      Erika Aoki, Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      International Conference on Intelligent Computing (ICIC2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model2021

    • Author(s)
      Yutong Zhou, Nobutaka Shimada
    • Organizer
      第24回画像の認識・理解シンポジウム(MIRU2021)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] ROS2 を用いた室内シーン変遷ロギングシステムの実装2021

    • Author(s)
      吉田享平、島田伸敬、松尾直志
    • Organizer
      ロボティクス・メカトロニクス 講演会 2021
    • Related Report
      2020 Annual Research Report
  • [Presentation] オペラブルリアリティのためのMRにおける収納家具の対話的取り込み2021

    • Author(s)
      前田 康佑, 野村 朋哉, 島田 伸敬, 松尾 直志
    • Organizer
      2021年電子情報通信学会総合大会
    • Related Report
      2020 Annual Research Report
  • [Presentation] リハビリ支援のための拇指先端可動域計測および可視化アプリ2021

    • Author(s)
      青木 映里香, 島田 伸敬, 松尾 直志
    • Organizer
      2021年電子情報通信学会総合大会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 日常的な物体操作の自動記録と解析に基づく物体把持方法の想起2021

    • Author(s)
      有本 光希, 島田 伸敬, 松尾 直志
    • Organizer
      2021年電子情報通信学会総合大会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 不完全な部分観測情報に基づく情報復元の曖昧さを表現する状況記述子の提案2021

    • Author(s)
      福井 尚卿, 松尾 直志, 島田 伸敬
    • Organizer
      第225回コンピュータビジョンとイメージメディア研究発表会(CVIM225)
    • Related Report
      2020 Annual Research Report
  • [Presentation] Spatial and Channel-wise Attention in Generative Adversarial Network for Text-to-Face Synthesis and Manipulation2021

    • Author(s)
      Yutong Zhou, Nobutaka Shimada
    • Organizer
      第225回コンピュータビジョンとイメージメディア研究発表会(CVIM225)
    • Related Report
      2020 Annual Research Report
  • [Presentation] MEMS 触覚センサ向け機械学習のためのFPGA によるSNN パーセプトロン実装2021

    • Author(s)
      青木雅典, 土橋嬉真花,安藤潤人,寒川雅之,秋田純一,野間春生
    • Organizer
      第26回バーチャルリアリティ学会大会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 物理的・化学的耐久性を兼備するフッ素エラストマ積層コーティングPDMS封止カンチレバー型触覚センサ2021

    • Author(s)
      髙橋佑司,髙橋拓海,安部隆,野間春生,寒川雅之
    • Organizer
      第38回「センサ・マイクロマシンと応用システム」シンポジウム
    • Related Report
      2020 Annual Research Report
  • [Presentation] Spatial and Channel-wise Attention in Generative Adversarial Networks for Text to Face Synthesis2020

    • Author(s)
      Yutong Zhou, Nobutaka Shimada
    • Organizer
      The 1st International Workshop on Intelligent Systems(IWIS 2020)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 不完全な観測情報に基づく 3D再構成の曖昧さを表現する空間形状記述子の提案2020

    • Author(s)
      福井 尚卿, 松尾 直志, 島田 伸敬
    • Organizer
      第21回計測自動制御学会システムインテグレーション部門講演会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 深層学習による失敗自動対応可能な人間の作業プロセスのモデル化2020

    • Author(s)
      チャン・ディン・トゥアン, 李周浩, 島田伸敬
    • Organizer
      ロボティクス・メカトロニクス講演会 2020(ROBOMECH 2020 in KANAZAWA)
    • Related Report
      2020 Annual Research Report
  • [Presentation] リバビリテーションのための親指先端可動域の計測と可視化2020

    • Author(s)
      三宅雄大, 松尾直志, 島田伸敬
    • Organizer
      ロボティクス・メカトロニクス講演会 2020(ROBOMECH 2020 in KANAZAWA)
    • Related Report
      2020 Annual Research Report
  • [Presentation] Pointing Direction Estimation for Attention Target Extraction Using Body-Mounted Camera2020

    • Author(s)
      Yusei Oozono, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      2020 17TH INTERNATIONAL CONFERENCE ON UBIQUITOUS ROBOTS (UR)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Improvement of Passive Joint in Wearable Robot Arm2020

    • Author(s)
      Akimichi Kojima, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      2020 17TH INTERNATIONAL CONFERENCE ON UBIQUITOUS ROBOTS (UR)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Emotion Estimation for Automatic Generation of 3D Avatar Motions Based on Dialog Texts2020

    • Author(s)
      Keita Kishima, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      2020 17TH INTERNATIONAL CONFERENCE ON UBIQUITOUS ROBOTS (UR)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 介護練習用ロボットの為の表情による痛み再現システム2020

    • Author(s)
      梶山 主税、李 美蘭、山添 大丈、李 周浩
    • Organizer
      ロボティクス・メカトロニクス講演会 2020(ROBOMECH 2020 in KANAZAWA)
    • Related Report
      2020 Annual Research Report
  • [Presentation] 知能化空間における人間の位置及び行動に基づく異常の検出2020

    • Author(s)
      小林 叶佳、山添 大丈、李 周浩
    • Organizer
      ロボティクス・メカトロニクス講演会 2020(ROBOMECH 2020 in KANAZAWA)
    • Related Report
      2020 Annual Research Report
  • [Presentation] 介護練習用ロボットの為の表情による痛み再現システム2020

    • Author(s)
      梶山 主税, 山添 大丈, 李 周浩
    • Organizer
      第18回高齢社会デザイン(ASD)研究会
    • Related Report
      2020 Annual Research Report
  • [Presentation] ドローンによる夜間警備のための赤外線カメラを用いた人間の行動認識2020

    • Author(s)
      今在家 拓哉, Tran Dinh, Tuan, 李 周浩
    • Organizer
      第21回計測自動制御学会システムインテグレーション部門講演会(SI2020)
    • Related Report
      2020 Annual Research Report
  • [Presentation] 卓上作業支援ロボットにおけるジェスチャ操作の評価2020

    • Author(s)
      山本 凌司, チャン ディン トゥアン, 李 周浩
    • Organizer
      第21回計測自動制御学会システムインテグレーション部門講演会(SI2020)
    • Related Report
      2020 Annual Research Report
  • [Presentation] オペラブルリアリティ:棚内物体の仮想操作インタフェース2020

    • Author(s)
      野村 朋哉, 有本 光希, 松尾 直志, 島田 伸敬
    • Organizer
      2020年電子情報通信学会総合大会
    • Related Report
      2019 Annual Research Report
  • [Presentation] 包丁手技訓練を目的とした把持力信号の 自動ストローク切り出し手法の検討2020

    • Author(s)
      川田智晴, 岡田一志, 大井翔, 松村耕平, 寒川雅之, 野間春生
    • Organizer
      インタラクション2020
    • Related Report
      2019 Annual Research Report
  • [Presentation] Using Motion Compensation and Matrix Completion Algorithm to Remove Rain Streaks and Snow for Video Sequences2019

    • Author(s)
      Yutong Zhou, Nobutaka Shimada
    • Organizer
      The 5th Asian Conference on Pattern Recognition (ACPR 2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Feature Extraction Approach to Handle Variations in Camera Viewpoint for Computer Vision Tasks2019

    • Author(s)
      Dinh Tuan Tran, Hirotake Yamazoe, Nobutaka Shimada, Joo-Ho Lee
    • Organizer
      The 15th Joint Workshop on Machine Perception and Robotics (MPR2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Discovery of Image Pixels Highly Contributing to CNN regression2019

    • Author(s)
      LI Yitian, SHIMADA Nobutaka
    • Organizer
      The 15th Joint Workshop on Machine Perception and Robotics (MPR2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Detection of Unstable Objects by Using Deep Learning for Domestic Environment2019

    • Author(s)
      FUKUI Takaaki, SHIMADA Nobutaka, MATSUO Tadashi
    • Organizer
      The 15th Joint Workshop on Machine Perception and Robotics (MPR2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Recalling Candidates of Grasping Method from an Object Image using Neural Network2019

    • Author(s)
      Makoto Sanada, Tadashi Matsuo, Nobutaka Shimada, Yoshiaki Shirai
    • Organizer
      The 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Analysis of robot's motion impressions for designing behaviors of home service robots2019

    • Author(s)
      Hirotake Yamazoe, Kenta Miki, Takuya Imazaike, Yume Matsushita, Jaemin Chun, Youngsun Kim, Joo-Ho Lee
    • Organizer
      The 1st IFAC Workshop on Robot Control (WROCO 2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Human motion generation based on GAN toward unsupervised 3D human pose estimation2019

    • Author(s)
      Sasuke Yamane, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      The 5th Asian Conference on Pattern Recognition (ACPR 2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 受信者の状態に応じた部分情報を重畳するARゲームの提案2019

    • Author(s)
      河盛 真波, 島田 伸敬
    • Organizer
      第24回バーチャルリアリティ学会大会(VRSJ2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Inferring Task Plans from Visual Demonstration with Recurrent Neural Networks2019

    • Author(s)
      Li Xi, Nobutaka Shimada
    • Organizer
      第22回画像の認識・理解シンポジウム(MIRU2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] Using Motion Compensation and Matrix Completion Algorithm to Remove Rain Streaks and Snow for Video Sequence2019

    • Author(s)
      Yutong Zhou, Nobutaka Shimada
    • Organizer
      第22回画像の認識・理解シンポジウム(MIRU2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] 力入力に対する剛体群の運動応答予測と静力学的構造安定性の推定2019

    • Author(s)
      福井 尚卿, 松尾 直志, 島田 伸敬
    • Organizer
      ロボティクス・メカトロニクス講演会2019(ROBOMECH2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] GANを用いた姿勢既知の教師データを必要としない3次元人物姿勢推定2019

    • Author(s)
      山根 佐介, 山添 大丈, 李 周浩
    • Organizer
      第22回 画像の認識・理解シンポジウム (MIRU2019)
    • Related Report
      2019 Annual Research Report
  • [Presentation] エラストマ封止カンチレバー型触覚センサの検出エリア評価2019

    • Author(s)
      阿部 由杜, 菅 史賢, 安部 隆, 野間 春生, 寒川 雅之
    • Organizer
      第36回 センサ・マイクロマシンと応用システム
    • Related Report
      2019 Annual Research Report
  • [Presentation] Gripping Force based Signature Verification Measured by Tactile Sensors2019

    • Author(s)
      Kazushi Okada, Sho Ooi,Kohei Matsumura, Haruo Noma
    • Organizer
      IEEE World Haptics 2019
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Auto-encoder factorizing into transform invariants and transform parameters2019

    • Author(s)
      Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      The 25th International Workshop on Frontiers of Computer Vision (IW-FCV2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] ニューラルネットワークを用いた物体画像から把持方法候補の想起2019

    • Author(s)
      眞田慎, 松尾直志, 島田伸敬, 白井良明
    • Organizer
      パターン認識・メディア理解研究会2019年1月研究会(PRMU)
    • Related Report
      2018 Annual Research Report
  • [Presentation] ペングリップ型デバイスを用いた個人認証の提案2019

    • Author(s)
      岡田 一志, 大井 翔, 松村 耕平, 野間 春生
    • Organizer
      インタラクション2019
    • Related Report
      2018 Annual Research Report
  • [Presentation] View-Invariant Human Activity Recognition Using Topic Model on Combined ORB-OF Feature2019

    • Author(s)
      DINH TUAN TRAN, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      2019 IEEE/SICE International Symposium on System Integration
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Development of a hand robot that can imitate the procedure of object manipulation accompanied with object state transition,2018

    • Author(s)
      ZHAO YUJIAN, Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      The 14th Joint Workshop on Machine Perception and Robotics (MPR2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Rain Streaks Detection and Removal Using Temporal Correlation and EM-Based Low-Rank Matrix Completion for Video Sequences2018

    • Author(s)
      ZHOU YUTONG, Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      The 14th Joint Workshop on Machine Perception and Robotics (MPR2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Auto-encoder for generating a transform invariant descriptor and transform parameters2018

    • Author(s)
      Tadashi Matsuo, Nobutaka Shimada
    • Organizer
      第21回画像の認識・理解シンポジウム(MIRU2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] Inference of Grasping Pattern from Object Image Based on Interaction Descriptor,2018

    • Author(s)
      Tadashi Matsuo, Takuya Kawakami, Yoko Ogawa, Nobutaka Shimada,
    • Organizer
      The 27th IEEE International Symposium on Industrial Electronics (IEEE-ISIE2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 胸部装着カメラによる注目対象抽出のための指差し方向推定2018

    • Author(s)
      大園 愉生, 山添 大丈, 李 周浩
    • Organizer
      日本機械学会ロボティクス・メカトロニクス講演会2018(ROBOMECH2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] CNN を用いた眼球中心・虹彩中心推定に基づく視線検出手法の検討2018

    • Author(s)
      施 真琴, 内海 章, 山添 大丈, 萩田 紀博, 李 周浩
    • Organizer
      日本機械学会ロボティクス・メカトロニクス講演会2018(ROBOMECH2018)
    • Related Report
      2018 Annual Research Report
  • [Presentation] Investigation of gaze detcection method based on eyeball center and ireis center esrimation using CNN2018

    • Author(s)
      Makoto Sei, Akira Utsumi, Hirotake Yamazoe, Norihiro Hagita, Joo-Ho Lee
    • Organizer
      The 14th joint Workshop on Machine Perception and Robotics (MPR2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pointing direction estimation for attention target extraction by using a chest mounted camera2018

    • Author(s)
      Yusei Oozono, Hirotake Yamazoe, Joo-Ho Lee
    • Organizer
      The 14th joint Workshop on Machine Perception and Robotics (MPR2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research

URL: 

Published: 2018-04-23   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi