Photo-realistic Image-based Modeling and Rendering in the Wild

2017 Fiscal Year Annual Research Report

• Project/Area Number: 17F17350
• Research Institution: Osaka University
• Principal Investigator: 松下 康之 大阪大学, 情報科学研究科, 教授 (30756507)
• Co-Investigator(Kenkyū-buntansha): WAECHTER MICHAEL 大阪大学, 情報科学研究科, 外国人特別研究員
• Project Period (FY): 2017-11-10 – 2020-03-31
• Keywords: コンピュータビジョン / 三次元復元

Outline of Annual Research Achievements

a)For removing photometric outliers in the texturing of multi-view stereo reconstructions, we looked into mathematical techniques that perform rank reduction on matrices of observations with high data redundancy while also detecting outliers. We have generated some initial datasets, both synthetically rendered and real-world multi-view stereo datasets that suffer from the outlier problems we try to tackle. Currently we are looking into methods that jointly perform rank reduction and image alignment since misalignment influences outlier removal and vice versa.

b)We developed a method for calibrating point light source positions, which is required for image-based modeling techniques such as photometric stereo. Previous works on this are expensive to set up and tedious to use. Our method works with needle pins that cast shadows on a flat board. Given shadow observations under enough board poses, we can jointly estimate light and needle pin positions. The work is scientifically interesting because it shows a mathematical connection between point lights, pinhole cameras and structure from motion.

c)WeI started collaborating in a project where we analyze the auto-recalibration of stereo fisheye cameras. This is highly relevant, e.g., in autonomous driving or consumer products, where fisheye lenses provide better overview over the scene. It seems that, in contrast to the well-studied pinhole and radial distortion camera models, fisheye models do not exhibit a mathematical ambiguity between camera intrinsics and extrinsics during the calibration.

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

11月からの5ヶ月の間に目的としていた(A) 「見え」のモデリングと(B) 局所的な画像のアライメントの定式化を達成できた。3次元復元のための画像の収集も始めており、概ね順調に進展している。

Strategy for Future Research Activity

We continue preparing datasets for the 3D reconstruction texturing outlier removal, in particular real-world datasets of various sizes where our investigated effects (outliers + misalignment) occur to a strong degree.
Then I want to implement two solution methods, RASL (a method based on matrix rank reduction and simultaneous image alignment) and an own solution based on optical flow and robust homography fitting, to compare them in terms of 1) their solution quality and 2) their sensitivity to parameter choice. The problem I experienced so far with low-rank approximation methods is that parameter configurations working in one scene part often do perform poorly in other scene parts and it is not obvious how to choose the parameters appropriately.
The goal is to submit this work to the Conference on Computer Vision and Pattern Recognition in November.