• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Robust and precise 3D endoscope system based on pattern projection and deep learning

Research Project

Project/Area Number 18H04119
Research Category

Grant-in-Aid for Scientific Research (A)

Allocation TypeSingle-year Grants
Section一般
Review Section Medium-sized Section 61:Human informatics and related fields
Research InstitutionKindai University (2021-2022)
Hiroshima City University (2018-2020)

Principal Investigator

Ryo Furukawa  近畿大学, 工学部, 教授 (50295838)

Co-Investigator(Kenkyū-buntansha) 田中 信治  広島大学, 病院(医), 教授 (00260670)
佐川 立昌  国立研究開発法人産業技術総合研究所, 情報・人間工学領域, 研究チーム長 (30362627)
岡 志郎  広島大学, 病院(医), 講師 (30403538)
川崎 洋  九州大学, システム情報科学研究院, 教授 (80361393)
宮崎 大輔  広島市立大学, 情報科学研究科, 准教授 (30532957)
馬場 雅志  広島市立大学, 情報科学研究科, 講師 (30281281)
日浦 慎作  兵庫県立大学, 工学研究科, 教授 (40314405)
Project Period (FY) 2018-04-01 – 2023-03-31
Project Status Completed (Fiscal Year 2022)
Budget Amount *help
¥43,420,000 (Direct Cost: ¥33,400,000、Indirect Cost: ¥10,020,000)
Fiscal Year 2022: ¥4,680,000 (Direct Cost: ¥3,600,000、Indirect Cost: ¥1,080,000)
Fiscal Year 2021: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000)
Fiscal Year 2020: ¥8,450,000 (Direct Cost: ¥6,500,000、Indirect Cost: ¥1,950,000)
Fiscal Year 2019: ¥11,830,000 (Direct Cost: ¥9,100,000、Indirect Cost: ¥2,730,000)
Fiscal Year 2018: ¥7,410,000 (Direct Cost: ¥5,700,000、Indirect Cost: ¥1,710,000)
Keywords能動ステレオ / 内視鏡 / 構造化光 / パターン光源 / 微分レンダリング / 3次元内視鏡 / パターン投光 / アクティブステレオ / 形状統合 / 形状位置合わせ / 3次元内視鏡 / 能動ステレオ法 / DOE / SLAM / 3次元計測 / アクティブステレオ法 / 3次元計測
Outline of Final Research Achievements

In this study, research and development of a three-dimensional endoscope system based on active stereo methods was carried out. Specifically, we developed an ultra-compact wide-angle pattern light source that can be inserted into the forceps hole of an endoscope and can utilise the endoscope's wide-angle camera. For 3D shape estimation with the developed wide-angle pattern, a single-image 3D measurement technique based on high-density correspondence estimation using a deep learning model was developed. In addition, an algorithm was developed to integrate the results of measurements made with multiple frames in the above system without any inconsistencies. This development showed that it is possible to measure high-density 3D shapes using most of the endoscope's field of view and to integrate them in multiple frames.

Academic Significance and Societal Importance of the Research Achievements

本研究では,能動ステレオ法による3次元内視鏡システムについて,計測範囲,計測密度を大幅に高めることに成功した.また,複数のフレームにわたって計測した計測結果を,矛盾なく統合する機能を実現した.これにより,計測システムとしての実用性を,大幅に高めることができた.開発された3次元形状計測システムは,医学的用途のみならず,工業用途(パイプの内側の計測など)などへの応用も期待できる.

Report

(6 results)
  • 2022 Annual Research Report   Final Research Report ( PDF )
  • 2021 Annual Research Report
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • Research Products

    (37 results)

All 2023 2022 2021 2020 2019 2018 Other

All Int'l Joint Research (1 results) Journal Article (6 results) (of which Int'l Joint Research: 1 results,  Peer Reviewed: 5 results,  Open Access: 5 results) Presentation (26 results) (of which Int'l Joint Research: 16 results) Book (1 results) Remarks (3 results)

  • [Int'l Joint Research] Robarts Institute, Western University(カナダ)

    • Related Report
      2022 Annual Research Report
  • [Journal Article] Multi-frame optimisation for active stereo with inverse renderingto obtain consistent shape and projector-camera posesfor 3D endoscopic system2022

    • Author(s)
      Furukawa Ryo、Mikamo Michihiro、Sagawa Ryusuke、Okamoto Yuki、Oka Shiro、Tanaka Shinji、Kawasaki Hiroshi
    • Journal Title

      Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization

      Volume: 0 Issue: 4 Pages: 1-9

    • DOI

      10.1080/21681163.2022.2155578

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Simultaneous estimation of projector and camera poses for multiple oneshot scan using pixel-wise correspondences estimated by U-Nets and GCN2021

    • Author(s)
      Furukawa Ryo、Mikamo Michihiro、Kawasaki Hiroshi、Sagawa Ryusuke、Oka Shiro、Kotachi Takahiro、Okamoto Yuki、Tanaka Shinji
    • Journal Title

      Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization

      Volume: 10 Issue: 5 Pages: 540-548

    • DOI

      10.1080/21681163.2021.2009376

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Learning-based active 3D measurement technique using light field created by video projectors2019

    • Author(s)
      Yuki Shiba1* , Satoshi Ono1 , Ryo Furukawa2 , Shinsaku Hiura2 and Hiroshi Kawasaki3
    • Journal Title

      IPSJ Transactions on Computer Vision and Applications

      Volume: Vol. 11, numbe 6 Issue: 1 Pages: 1-14

    • DOI

      10.1186/s41074-019-0058-y

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan2019

    • Author(s)
      Ryo Furukawa 1 ; Genki Nagamatsu 2 ; Shiro Oka 3 ; Takahiro Kotachi 4 ; Yuki Okamoto 4 ; Shinji Tanaka 4 ; Hiroshi Kawasaki 2
    • Journal Title

      MICCAI workshops AE-CAI, CARE (MIAR)

      Volume: Vol. 6, Iss. 6 Issue: 6 Pages: 249-254

    • DOI

      10.1049/htl.2019.0070

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] 構造化光投影による3次元内視鏡の開発2019

    • Author(s)
      古川 亮,川崎 洋
    • Journal Title

      光技術コンタクト

      Volume: 57 Pages: 47-52

    • Related Report
      2018 Annual Research Report
  • [Journal Article] A structured light laser probe for gastrointestinal polyp size measurement: a preliminary comparative study2018

    • Author(s)
      Marco Visentini-Scarzanella,Hiroshi Kawasaki,Ryo Furukawa,Marco Augusto Bonino,Simone Arolfo,Giacomo Lo Secco,Alberto Arezzo,Arianna Menciassi,Paolo Dario,Gastone Ciuti
    • Journal Title

      Endoscopy International Open

      Volume: Issue 05 - Volume 06 Issue: 05 Pages: 602-609

    • DOI

      10.1055/a-0577-2798

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Presentation] Single and multi-frame auto-calibration for 3D endoscopy with differential rendering2023

    • Author(s)
      Ryo Furukawa, Ryusuke Sagawa, Shiro Oka, Shinji Tanaka, and Hiroshi Kawasaki
    • Organizer
      Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Single-shot dense active stereo with pixel-wise phase estimation based on grid-structure using CNN and correspondence estimation using GCN2022

    • Author(s)
      Furukawa Ryo、Mikamo Michihiro、Sagawa Ryusuke、Kawasaki Hiroshi
    • Organizer
      IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 3D endoscope system with AR display superimposing dense and wide-angle-of-view 3D points obtained by using micro pattern projector2022

    • Author(s)
      Mikamo Michihiro、Furukawa Ryo、Oka Shiro、Kotachi Takahiro、Okamoto Yuki、Tanaka Shinji、Sagawa Ryusuke、Kawasaki Hiroshi
    • Organizer
      44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] データ拡張による投影パターン検出の安定化と MRF 最適化によ る対応付の改善によるワンショット三次元形状計測の高精度化2022

    • Author(s)
      原田寛大,三鴨道弘,古川亮,佐川立 昌,川崎洋
    • Organizer
      画像の認識理解シンポジウム MIRU2022
    • Related Report
      2022 Annual Research Report
  • [Presentation] アクティブステレオ法における複数フレーム計測結果の全体 最適化法2022

    • Author(s)
      古川亮, 三鴨道弘 , 佐川立昌, 川崎洋
    • Organizer
      画像の認識理解シンポジウム MIRU2022
    • Related Report
      2022 Annual Research Report
  • [Presentation] Single-shot dense active stereo with pixel-wise phase estimation based on grid-structure using CNN and correspondence estimation using GCN2022

    • Author(s)
      Ryo Furukawa, Michihiro Mikamo, Ryusuke Sagawa, Hiroshi Kawasaki
    • Organizer
      IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] GCN-Calculated Graph-Feature Embedding for 3D Endoscopic System Based on Active Stereo2021

    • Author(s)
      Michihiro Mikamo, Hiroshi Kawasaki, Ryusuke Sagawa, Ryo Furukawa
    • Organizer
      IEEE/CVF Winter Conference on ApplicaIW-FCV 2021: Frontiers of Computer Visiontions of Computer Vision (WACV)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Method for Adding Motion-Blur on Arbitrary Objects by Using Auto-Segmentation and Color Compensation Techniques2021

    • Author(s)
      Michihiro Mikamo, Ryo Furukawa, Hiroshi Kawasaki
    • Organizer
      Michihiro Mikamo, Ryo Furukawa, Hiroshi Kawasaki
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Active Stereo Method for 3D Endoscopes using Deep-layer GCN and Graph Representation with Proximity Information2021

    • Author(s)
      Michihiro Mikamo, Ryo Furukawa, Shiro Oka, Takahiro Kotachi, Yuki Okamoto, Shinji Tanaka, Ryusuke Sagawa, Hiroshi Kawasaki
    • Organizer
      43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 公開 Oneshot dense 3D active stereo based on pixel-wise correspondence estimation using U-Nets and GCN2021

    • Author(s)
      Michihiro Mikamo, Ryusuke Sagawa, Hiroshi Kawasaki, Ryo Furukawa
    • Organizer
      画像の認識理解シンポジウム MIRU2021
    • Related Report
      2021 Annual Research Report
  • [Presentation] GCN-Calculated Graph-Feature Embedding for 3D Endoscopic System Based on Active Stereo2021

    • Author(s)
      Michihiro Mikamo, Hiroshi Kawasaki, Ryusuke Sagawa, Ryo Furukawa
    • Organizer
      Frontiers of Computer Vision - 27th International Workshop, IW-FCV 2021,
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Domain invariantな画像特徴によるCGからの能動ステレオシステムの学習2020

    • Author(s)
      三鴨 道弘 川崎 洋 佐川 立昌 古川 亮
    • Organizer
      情報処理学会 CVIM研究会
    • Related Report
      2020 Annual Research Report
  • [Presentation] Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network2020

    • Author(s)
      Ryo Furukawa, Shiro Oka, Takahiro Kotachi, Yuki Okamoto, Shinji Tanaka, Ryusuke Sagawa, Hiroshi Kawasaki
    • Organizer
      42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
    • Related Report
      2020 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Single-wavelength and multi-parallel dotted-and solid-lines for dense and robust active 3D reconstruction2019

    • Author(s)
      Genki Nagamatsu, Ryo Furukawa, Ryusuke Sagawa, Hiroshi Kawasaki
    • Organizer
      2019 16th International Conference on Machine Vision Applications (MVA)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Simultaneous shape registration and active stereo shape reconstruction using modified bundle adjustment2019

    • Author(s)
      Ryo Furukawa, Genki Nagamatsu, Hiroshi Kawasaki
    • Organizer
      2019 International Conference on 3D Vision (3DV)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 3次元内視鏡におけるambient光源にロバストな投影パターン特徴抽出2019

    • Author(s)
      古川 亮, 内藤 雅仁, 宮崎 大輔, 馬場 雅志, 日浦 慎作(広島市大), 岡 志郎, 田中 信治(広島大病院), 川崎 洋(九大)
    • Organizer
      画像の認識理解シンポジウム
    • Related Report
      2019 Annual Research Report
  • [Presentation] 疎な構造化光による疎な3次元復元形状のパターン間の陰影学習による高密度化2019

    • Author(s)
      岩口 尭史, 栗田 拓弥, 時枝 康大(九大), 古川 亮(広島市大), 川崎 洋(九大)
    • Organizer
      画像の認識理解シンポジウム
    • Related Report
      2019 Annual Research Report
  • [Presentation] CNNによる特徴抽出と複数フレームの形状統合による広範囲かつ高精度な3次元内視鏡システム2019

    • Author(s)
      古川 亮, 内藤 雅仁, 溝森 将輝, 馬場 雅志, 宮崎 大輔, 日浦 慎作, 岡 志郎, 田中 信治, 川崎 洋
    • Organizer
      電子情報通信学会,医用画像研究会,MI2018-88
    • Related Report
      2018 Annual Research Report
  • [Presentation] Robust structured light system against subsurface scattering effects achieved by CNN-based pattern detection and decoding algorithm2018

    • Author(s)
      Ryo Furukawa, Daisuke Miyazaki, Masashi Baba, Shinsaku Hiura, Hiroshi Kawasaki
    • Organizer
      Computer Vision, ECCV 2018 Workshops
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Improvement of image quality by using viewpoint following in multi-layer light field display2018

    • Author(s)
      Ryo Furukawa, Tsukasa Tadano, Shinsaku Hiura, Hiroshi Kawasaki
    • Organizer
      3D Image Acquisition and Display: Technology, Perception and Applications: OSA imaging congress
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Grid-based oneshot scan using dot-line pattern2018

    • Author(s)
      Ryo Furukawa, Hiroshi Kawasaki
    • Organizer
      3D Image Acquisition and Display: Technology, Perception and Applications: OSA imaging congress
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Multi-scale cnn stereo and pattern removal technique for underwater active stereo system2018

    • Author(s)
      Kazuto Ichimaru, Ryo Furukawa, Hiroshi Kawasaki
    • Organizer
      International Conference on 3D Vision (3DV)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Wide-Area Shape Reconstruction by 3D Endoscopic System Based on CNN Decoding, Shape Registration and Fusion2018

    • Author(s)
      Ryo Furukawa, Masaki Mizomori, Shinsaku Hiura, Shiro Oka, Shinji Tanaka, Hiroshi Kawasaki
    • Organizer
      Computer Assisted Robotic Endoscopy: MICCAI 2018 workshop
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 能動ステレオ法のためのパターン検出におけるCNNによる複数特徴の同時抽出2018

    • Author(s)
      溝森 将輝, 内藤 雅仁, 古川 亮, 馬場 雅志, 宮崎 大輔, 日浦 慎作, 川崎 洋
    • Organizer
      第 21 回 画像の認識・理解シンポジウム
    • Related Report
      2018 Annual Research Report
  • [Presentation] Grid-pattern feature detection technique using U-Nets for 3D endoscope2018

    • Author(s)
      Masahito Naito, Ryo Furukawa, Masashi Okitomo, Daisuke Miyazaki, Masashi Baba, Shinsaku Hiura, Yoji Sanomura, Shinji Tanaka, and Hiroshi Kawasaki
    • Organizer
      第 21 回 画像の認識・理解シンポジウム
    • Related Report
      2018 Annual Research Report
  • [Presentation] Underwater accurate shape reconstruction using CNNs2018

    • Author(s)
      Kazuto Ichimaru, Ryo Furukawa, Hiroshi Kawasaki
    • Organizer
      第 21 回 画像の認識・理解シンポジウム
    • Related Report
      2018 Annual Research Report
  • [Book] Active Lighting and Its Application for Computer Vision, 40 Years of History of Active Lighting Techniques2020

    • Author(s)
      Katsushi Ikeuchi, Yasuyuki Matsushita, Ryusuke Sagawa, Hiroshi Kawasaki, Yasuhiro Mukaigawa, Ryo Furukawa, Daisuke Miyazaki
    • Total Pages
      536
    • Publisher
      Springer
    • ISBN
      9783030565770
    • Related Report
      2020 Annual Research Report
  • [Remarks] 「人体の内外表面形状すべてをリアルタイム計測するシステム」

    • URL

      https://www.hyper3dsensing.com/

    • Related Report
      2022 Annual Research Report
  • [Remarks] 「人体の内外表面形状すべてをリアルタイム計測するシステム」

    • URL

      https://www.hyper3dsensing.com/index.html

    • Related Report
      2021 Annual Research Report
  • [Remarks] 人体の内外表面形状すべてをリアルタイム計測するシステム

    • URL

      http://www.hyper3dsensing.com/

    • Related Report
      2019 Annual Research Report

URL: 

Published: 2018-04-23   Modified: 2024-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi