• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

3.5D measurements and point cloud processing for handling and inspection of transparent and shiny objects

Research Project

Project/Area Number 19H01096
Research Category

Grant-in-Aid for Scientific Research (A)

Allocation TypeSingle-year Grants
Section一般
Review Section Medium-sized Section 60:Information science, computer engineering, and related fields
Research InstitutionTohoku University

Principal Investigator

Hashimoto Koichi  東北大学, 情報科学研究科, 教授 (80228410)

Project Period (FY) 2019-04-01 – 2024-03-31
Project Status Completed (Fiscal Year 2023)
Budget Amount *help
¥44,590,000 (Direct Cost: ¥34,300,000、Indirect Cost: ¥10,290,000)
Fiscal Year 2023: ¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000)
Fiscal Year 2022: ¥7,280,000 (Direct Cost: ¥5,600,000、Indirect Cost: ¥1,680,000)
Fiscal Year 2021: ¥11,570,000 (Direct Cost: ¥8,900,000、Indirect Cost: ¥2,670,000)
Fiscal Year 2020: ¥10,400,000 (Direct Cost: ¥8,000,000、Indirect Cost: ¥2,400,000)
Fiscal Year 2019: ¥10,790,000 (Direct Cost: ¥8,300,000、Indirect Cost: ¥2,490,000)
Keywords3次元計測 / マニピュレーション / プロジェクタ / 透明物体 / 光沢物体 / 全品検査 / 半透明/光沢物体 / 点群処理 / LT行列 / 3.5D計測 / ビジュアルサーボ / 計測システム / ロボットシステム / スパースセンシング / 反射特性
Outline of Research at the Start

本研究では、食品、薬品、化粧品などの半透明または金属光沢を有する物体を対象とした、「3D形状プラスアルファ」の計測と点群処理を開発する(3.5D計測点群処理)。技術的課題はLight Transport Matrixのスパースセンシングならびにフィードバックプロジェクションによる表面形状と複雑な反射特性の推定である。研究の評価ポイントは、このような物体のハンドリングと全品検査を同時に実用的な時間で実行するロボットシステムの実現である。

Outline of Final Research Achievements

In this study, we developed a "3D shape plus extra" measurement and point cloud processing method for translucent or metallic shiny objects such as food, medicine, cosmetics, etc. The LT matrix (reflection characteristic matrix from all points of the projector to all pixels of the camera) was efficiently obtained by sparse sensing. We also developed a parallel processing algorithm based on the vertical and horizontal decomposition of irradiation patterns, and confirmed the high-speed computing performance on GPUs. In addition, handling of translucent and metallic luster objects was conducted. Picking of lenses, metal screws, and metal blocks in bulk was conducted to verify sufficient performance, and the detection performance of surface flaws and chips was confirmed for transparent plastic products.

Academic Significance and Societal Importance of the Research Achievements

半透明または金属光沢を有する物体に光をあてると、いくつかの反射や透過を繰り返し、物体表面だけではなく別な場所も輝度が高くなる。このように複雑な反射特性を持つため、表面形状を計測するためには高度な計算処理が必要となる。開発した照射パターン縦横分解は、パターンプロジェクションによる3次元計測に新たな科学的解釈を与えるとともに、並列処理の実装に適した計算アルゴリズムを与えた。さらに、GPU(グラフィクスプロセッシングユニット)実装による非常に高速な性能を達成した。ロボットによるハンドリングを実施することで精度検証を行い、プラスチック製品品質検査を実施して表面キズや欠けが検出可能であることを実証した。

Report

(7 results)
  • 2023 Annual Research Report   Final Research Report ( PDF )
  • 2022 Annual Research Report
  • 2021 Annual Research Report
  • 2020 Annual Research Report
  • 2019 Comments on the Screening Results   Annual Research Report
  • Research Products

    (22 results)

All 2023 2022 2021 2020 2019

All Journal Article (20 results) (of which Peer Reviewed: 20 results,  Open Access: 10 results) Presentation (1 results) Patent(Industrial Property Rights) (1 results)

  • [Journal Article] Recognition of Human Relationships Using Interactions and Gazes through Video Analysis in Surveillance Footage2023

    • Author(s)
      Matus Tanonwong, Naoya Chiba, Koichi Hashimoto
    • Journal Title

      2023 IEEE International Conference on Robotics and Biomimetics (ROBIO)

      Volume: - Pages: 1-7

    • DOI

      10.1109/robio58561.2023.10354720

    • Related Report
      2023 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Multi-Objective Deep Reinforcement Learning for Crowd Route Guidance Optimization2023

    • Author(s)
      Ryo Nishida, Yuki Tanigaki, Koichi Hashimoto
    • Journal Title

      Transportation Research Record

      Volume: - Issue: 5 Pages: 1-17

    • DOI

      10.1177/03611981231190635

    • Related Report
      2023 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Homography Estimation Using Marker Projection Control: A Case of Calibration-Free Projection Mapping2023

    • Author(s)
      Shingo Kagami and Koichi Hashimoto
    • Journal Title

      Proc. 22nd World Congress of the International Federation of Automatic Control (IFAC2023)

      Volume: - Issue: 2 Pages: 2951-2956

    • DOI

      10.1016/j.ifacol.2023.10.1418

    • Related Report
      2023 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Comparative Study of Hybridization and Parameter Tuning Improvement Methods for EAs in WFLOP2023

    • Author(s)
      Ziqing Ye, Haichuan Yang, Naoya Chiba, Koichi Hashimoto
    • Journal Title

      CIIS 2023: 2023 the 6th International Conference on Computational Intelligence and Intelligent Systems

      Volume: - Pages: 59-65

    • DOI

      10.1145/3638209.3638219

    • Related Report
      2023 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Evaluation of Perceptual Difference in Dynamic Projection Mapping with and without Movement of the Target Surface2022

    • Author(s)
      Shunya Fukuda, Shingo Kagami, Koichi Hashimoto
    • Journal Title

      Journal of Robotics and Mechatronics

      Volume: 34 Issue: 5 Pages: 1141-1151

    • DOI

      10.20965/jrm.2022.p1141

    • ISSN
      0915-3942, 1883-8049
    • Year and Date
      2022-10-20
    • Related Report
      2022 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Simulation Analysis on Benefits of Introducing Meeting Points Into On-Demand Shared Mobility Services.2022

    • Author(s)
      Ryo Nishida, Ryo Kanamori, Masaki Onishi, Itsuki Noda, Koichi Hashimoto
    • Journal Title

      IEEE Access

      Volume: 10 Pages: 124114-124129

    • DOI

      10.1109/access.2022.3224728

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Sample-Efficient Multimodal Dynamics Modeling for Risk-Sensitive Reinforcement Learning2022

    • Author(s)
      Yashima Ryota、Yamaguchi Akihiko、Hashimoto Koichi
    • Journal Title

      2022 8th International Conference on Mechatronics and Robotics Engineering (ICMRE)

      Volume: - Pages: 21-27

    • DOI

      10.1109/icmre54455.2022.9734091

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Vision-Based Finger Tapping Detection Without Fingertip Observation2021

    • Author(s)
      Narita Shotaro、Graduate School of Information Sciences, Tohoku University 6-6-01 Aramaki Aza Aoba, Aoba-ku, Sendai 980-8579, Japan、Kagami Shingo、Hashimoto Koichi
    • Journal Title

      Journal of Robotics and Mechatronics

      Volume: 33 Issue: 3 Pages: 484-493

    • DOI

      10.20965/jrm.2021.p0484

    • NAID

      130008054361

    • ISSN
      0915-3942, 1883-8049
    • Year and Date
      2021-06-20
    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] MetamorHockey: A Projection-based Virtual Air Hockey Platform Featuring Transformable Mallet Shapes2021

    • Author(s)
      Ueda Shun、Kagami Shingo、Hashimoto Koichi
    • Journal Title

      Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies, SIGGRAPH 2021

      Volume: - Pages: 1-4

    • DOI

      10.1145/3450550.3465341

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Vibration Control for Pivoting by Robot Hand Equipped with CAVS and FingerVision2021

    • Author(s)
      Suzuki Yoshiyuki、Yamaguchi Akihiko、Nojiri Seita、Watanabe Tetsuyou、Hashimoto Koichi
    • Journal Title

      Proceedings - 2021 5th IEEE International Conference on Robotic Computing, IRC 2021

      Volume: - Pages: 18-26

    • DOI

      10.1109/irc52146.2021.00010

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Sparse Estimation under Saturated Condition for 3D Measurement and Its Application for Bin-Picking Robot2020

    • Author(s)
      千葉直也,橋本浩一
    • Journal Title

      Journal of the Japan Society for Precision Engineering

      Volume: 86 Issue: 1 Pages: 106-112

    • DOI

      10.2493/jjspe.86.106

    • NAID

      130007782510

    • ISSN
      0912-0289, 1882-675X
    • Year and Date
      2020-01-05
    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Fast Prediction of a Worker’s Reaching Motion Without a Skeleton Model (F-PREMO)2020

    • Author(s)
      Arai Shogo、Pettersson Andreas Lennart、Hashimoto Koichi
    • Journal Title

      IEEE Access

      Volume: 8 Pages: 90340-90350

    • DOI

      10.1109/access.2020.2992068

    • Related Report
      2020 Annual Research Report 2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] FAst Detection Algorithm for 3D Keypoints (FADA-3K)2020

    • Author(s)
      Arai Shogo、Fukuchi Nobuaki、Hashimoto Koichi
    • Journal Title

      IEEE Access

      Volume: 8 Pages: 189556-189564

    • DOI

      10.1109/access.2020.3025534

    • Related Report
      2020 Annual Research Report 2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Interactive Stickies: Low-latency projection mapping for dynamic interaction with projected images on a movable surface2020

    • Author(s)
      Kagami Shingo、Hashimoto Koichi
    • Journal Title

      ACM SIGGRAPH 2020 Emerging Technologies, SIGGRAPH 2020

      Volume: - Pages: 1-2

    • DOI

      10.1145/3388534.3407291

    • Related Report
      2020 Annual Research Report 2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Perception of Motion-Adaptive Color Images Displayed by a High-Speed DMD Projector2019

    • Author(s)
      Oshiro Wakana、Kagami Shingo、Hashimoto Koichi
    • Journal Title

      Proc. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2019 Workshop on Perception-driven Graphics and Displays)

      Volume: - Pages: 1790-1793

    • DOI

      10.1109/vr.2019.8797850

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Puppeteered rain2019

    • Author(s)
      Kagami Shingo、Higuchi Kotone、Hashimoto Koichi
    • Journal Title

      Proc. ACM SIGGRAPH Conference on Computer Graphics and Interactive Techniques (SIGGARPH 2019 Posters)

      Volume: - Pages: 48-48

    • DOI

      10.1145/3306214.3338603

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Discriminative Recognition of Point Cloud Gesture Classes through One-Shot Learning2019

    • Author(s)
      Joshua Owoyemi, Naoya Chiba, Koichi Hashimoto
    • Journal Title

      Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics

      Volume: 2019 Pages: 2304-2309

    • DOI

      10.1109/robio49542.2019.8961778

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Bin-picking of Randomly Piled Shiny Industrial Objects Using Light Transport Matrix Estimation*2019

    • Author(s)
      Naoya Chiba, Mingyu Li, Akira Imakura, Koichi Hashimoto
    • Journal Title

      Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics

      Volume: 2019 Pages: 7-13

    • DOI

      10.1109/robio49542.2019.8961622

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Animated Stickies: Fast Video Projection Mapping onto a Markerless Plane through a Direct Closed-Loop Alignment2019

    • Author(s)
      Kagami Shingo、Hashimoto Koichi
    • Journal Title

      IEEE Transactions on Visualization and Computer Graphics

      Volume: 25 Issue: 11 Pages: 3094-3104

    • DOI

      10.1109/tvcg.2019.2932248

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Fast ADMM L1 minimization by applying SMW formula and multi-row simultaneous estimation for Light Transport Matrix acquisition*2019

    • Author(s)
      Naoya Chiba, Akira Imakura, Koichi Hashimoto
    • Journal Title

      Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics

      Volume: 2019 Pages: 14-21

    • DOI

      10.1109/robio49542.2019.8961736

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Presentation] 群集誘導戦略最適化に向けた多目的深層強化学習に関する研究2022

    • Author(s)
      西田遼、谷垣勇輝、大西正輝、橋本浩一
    • Organizer
      2022年度人工知能学会全国大会
    • Related Report
      2022 Annual Research Report
  • [Patent(Industrial Property Rights)] 3次元形状計測方法及び3次元形状計測装置2021

    • Inventor(s)
      伊東秀謹、鏡慎吾、橋本浩一、鈴木大地、十都善行、稲垣義弘
    • Industrial Property Rights Holder
      国立大学法人東北大学、コニカミノルタ株式会社
    • Industrial Property Rights Type
      特許
    • Patent Publication Number
      2023-083632
    • Filing Date
      2021
    • Acquisition Date
      2023
    • Related Report
      2022 Annual Research Report

URL: 

Published: 2019-04-18   Modified: 2025-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi