Estimation of density distribution of inhomogeneous scattering media with deep learning and physics-based model

2022 Fiscal Year Final Research Report

• Project/Area Number: 21K21317
• Research Category: Grant-in-Aid for Research Activity Start-up
• Allocation Type: Multi-year Fund
• Review Section: 1002:Human informatics, applied informatics and related fields
• Research Institution: Nara Institute of Science and Technology
• Principal Investigator: Fujimura Yuki 奈良先端科学技術大学院大学, 先端科学技術研究科, 助教 (40908729)
• Project Period (FY): 2021-08-30 – 2023-03-31
• Keywords: コンピュータビジョン / 散乱媒体 / 深層学習 / フォトンマッピング

Outline of Final Research Achievements

This research aims to estimate the density distribution of inhomogeneous participating media with deep learning and a physics-based model. In participating media such as fog or smoke, incident light is scattered by suspended particles, which causes scattered light. In this research, we developed the method to estimate the density distribution of participating media only with commonly-used RGB cameras by exploiting deep-learning technology. The input of the developed method is multiple images captured from different views. A density volume is modeled as the output of a convolutional neural network, and the volume is optimized by reconstructing observed images with photon mapping, which is one of physics-based renderers.

Free Research Field

コンピュータビジョン

Academic Significance and Societal Importance of the Research Achievements

本研究では深層学習技術とフォトンマッピングを用いて散乱媒体濃度分布の推定を行った．CGでの画像生成に用いられるフォトンマッピングを最適化に用いるためには，画像を生成するすべての過程を微分可能な形で実装する必要がある．本研究は世界で初めてフォトンマッピングを微分可能な形で実装した．実世界の散乱媒体の濃度分布を推定することは，物理モデルの解析や写実的なCGの生成など社会的にも重要である．