Research on Appearance and 3D imaging

• Project/Area Number: 19H01123
• Research Category: Grant-in-Aid for Scientific Research (A)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Medium-sized Section 61:Human informatics and related fields
• Research Institution: Osaka University
• Principal Investigator: 松下 康之 大阪大学, 大学院情報科学研究科, 教授 (30756507)
• Project Period (FY): 2019-04-01 – 2024-03-31
• Project Status: Declined (Fiscal Year 2023)
• Budget Amount: ¥45,630,000 (Direct Cost: ¥35,100,000、Indirect Cost: ¥10,530,000); Fiscal Year 2023: ¥5,850,000 (Direct Cost: ¥4,500,000、Indirect Cost: ¥1,350,000); Fiscal Year 2022: ¥5,850,000 (Direct Cost: ¥4,500,000、Indirect Cost: ¥1,350,000); Fiscal Year 2021: ¥6,890,000 (Direct Cost: ¥5,300,000、Indirect Cost: ¥1,590,000); Fiscal Year 2020: ¥11,050,000 (Direct Cost: ¥8,500,000、Indirect Cost: ¥2,550,000); Fiscal Year 2019: ¥15,990,000 (Direct Cost: ¥12,300,000、Indirect Cost: ¥3,690,000)
• Keywords: コンピュータビジョン / 三次元形状復元 / 質感推定 / 三次元イメージング / 反射率推定 / 質感

Outline of Research at the Start

本研究では，実世界物体を対象にカメラを用いた高精度な質感と三次元形状のディジタル化技術の確立に取り組む．これまでに困難であった画像を用いた質感のディジタル化に取り組み、様々な実環境で有効な「質感＋三次元」イメージング技術を開発する．本研究課題の実現により，実世界物体の高精細な「質感＋三次元形状」のイメージングが手軽に利用できるようになり，撮影結果を利用して対象シーンを任意の光源下でレンダリングすることができる他，物体認識等の基盤技術の精度向上に貢献できる．

Outline of Annual Research Achievements

2022年度には，(1)実世界物体を対象とした「質感＋三次元」イメージングの展開，(2)簡易なイメージングセットアップによる三次元計測技術，及び，(3)獲得した法線から面としての三次元形状（メッシュなど）を復元するための法線積分について検討を進めた．項目(1)では，多波長照度差ステレオ及び未較正照度差ステレオ法に関する研究を進め，より実用的な質感＋三次元推定技術の開発を行った．項目(2)では，スマートフォン上のカメラとLEDを利用した簡易なイメージングセットアップによる三次元復元技術を開発し，屋内・屋外シーンにおける提案手法の有効性を確認した．項目(3)では，新たに双方向(bilateral)的な法線積分法を開発し，既存手法に比して高精度に三次元形状の復元が可能となることを示した．これらの結果として，コンピュータビジョン研究分野のトップ国際論文誌であるIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)へ1本，International Journal of Computer Vision (IJCV)へ2本，トップ国際会議の一つであるEuropean Conference on Computer Vision (ECCV) 2022にて1本の論文を発表した．
本研究課題は今年度でひとまず終了とし，次年度から関連する内容を基盤S研究にて加速させる予定である．

Research Progress Status

翌年度、交付申請を辞退するため、記入しない。

Strategy for Future Research Activity

翌年度、交付申請を辞退するため、記入しない。

Research Products

• [Int'l Joint Research] Peking University(中国)
• [Int'l Joint Research] Simon Fraser University(カナダ)
• [Int'l Joint Research] Meta(米国)
• [Int'l Joint Research] Peking University(中国)
• [Int'l Joint Research] Australian National University(オーストラリア)
• [Journal Article] Shape and Albedo Recovery by Your Phone using Stereoscopic Flash and No-Flash Photography (2022)
  • Author(s): Xu Cao, Michael Waechter, Boxin Shi, Ye Gao, Bo Zheng, Fumio Okura, Yasuyuki Matsushita
  • Journal Title: International Journal of Computer Vision (IJCV) Volume: 130 Issue: 6 Pages: 1403-1415
  • DOI: 10.1007/s11263-022-01597-6
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Multispectral Photometric Stereo for Spatially-Varying Spectral Reflectances (2022)
  • Author(s): Heng Guo, Fumio Okura, Boxin Shi, Takuya Funatomi, Yasuhiro Mukaigawa, and Yasuyuki Matsushita
  • Journal Title: International Journal of Computer Vision (IJCV) Volume: 130 Issue: 9 Pages: 2166-2183
  • DOI: 10.1007/s11263-022-01634-4
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Patch-Based Uncalibrated Photometric Stereo Under Natural Illumination (2022)
  • Author(s): Heng Guo, Zhipeng Mo, Boxin Shi, Feng Lu, Sai-Kit Yeung, Ping Tan, and Yasuyuki Matsushita
  • Journal Title: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) Volume: 44 Issue: 11 Pages: 7809-7823
  • DOI: 10.1109/tpami.2021.3115229
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Bilateral Normal Integration (2022)
  • Author(s): Cao Xu、Santo Hiroaki、Shi Boxin、Okura Fumio、Matsushita Yasuyuki
  • Journal Title: In Proceedings of European Conference on Computer Vision (ECCV) Volume: 1 Pages: 552-567
  • DOI: 10.1007/978-3-031-19769-7_32
  • ISBN: 9783031197680, 9783031197697
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Deep Photometric Stereo for Non-Lambertian Surfaces (2022)
  • Author(s): Chen Guanying、Han Kai、Shi Boxin、Matsushita Yasuyuki、Wong Kwan-Yee K.
  • Journal Title: IEEE Transactions on Pattern Analysis and Machine Intelligence Volume: 44 Issue: 1 Pages: 129-142
  • DOI: 10.1109/tpami.2020.3005397
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Deep Photometric Stereo Networks for Determining Surface Normal and Reflectances (2020)
  • Author(s): Santo Hiroaki、Samejima Masaki、Sugano Yusuke、Shi Boxin、Matsushita Yasuyuki
  • Journal Title: IEEE Transactions on Pattern Analysis and Machine Intelligence Volume: 1 Issue: 1 Pages: 1-1
  • DOI: 10.1109/tpami.2020.3005219
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Light structure from pin motion: Geometric point light source calibration (2020)
  • Author(s): Hiroaki Santo, Michael Waechter, Wen-Yan Lin, Yusuke Sugano, Yasuyuki Matsushita
  • Journal Title: International Journal of Computer Vision (IJCV) Volume: - Issue: 7 Pages: 1889-1912
  • DOI: 10.1007/s11263-020-01312-3
  • Peer Reviewed / Open Access
• [Presentation] Normal Integration via Inverse Plane Fitting With Minimum Point-to-Plane Distance (2021)
  • Author(s): Xu Cao, Boxin Shi, Fumio Okura, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • Int'l Joint Research
• [Presentation] Lighting, Reflectance and Geometry Estimation From 360deg Panoramic Stereo (2021)
  • Author(s): Junxuan Li, Hongdong Li, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • Int'l Joint Research
• [Presentation] Symmetric-light Photometric Stereo (2021)
  • Author(s): Kazuma Minami, Hiroaki Santo, Fumio Okura, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
  • Int'l Joint Research
• [Presentation] Shape-coded ArUco: Fiducial Marker for Bridging 2D and 3D Modalities (2021)
  • Author(s): Lilika Makabe, Hiroaki Santo, Fumio Okura, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
  • Int'l Joint Research
• [Presentation] What is Learned in Deep Uncalibrated Photometric Stereo? (2020)
  • Author(s): Guanying Chen, Michael Waechter, Boxin Shi, Kwan-Yee Kenneth Wong, Yasuyuki Matsushita
  • Organizer: European Conference on Computer Vision (ECCV)
  • Int'l Joint Research
• [Presentation] Deep near-light photometric stereo for spatially varying reflectances (2020)
  • Author(s): Hiroaki Santo, Michael Waechter, Yasuyuki Matsushita
  • Organizer: European Conference on Computer Vision (ECCV)
  • Int'l Joint Research
• [Presentation] Stereoscopic Flash and No-Flash Photography for Shape and Albedo Recovery (2020)
  • Author(s): Xu Cao, Michael Waechter, Boxin Shi, Ye Gao, Bo Zheng, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • Int'l Joint Research
• [Presentation] Photometric Stereo via Discrete Hypothesis-and-Test Search (2020)
  • Author(s): Kenji Enomoto, Michael Waechter, Kiriakos N. Kutulakos, Yasuyuki Matsushita
  • Organizer: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • Int'l Joint Research