ライトフィールドカメラを用いた複合現実感及び隠消現実感に関する研究

• Project/Area Number: 16J05114
• Research Category: Grant-in-Aid for JSPS Fellows
• Allocation Type: Single-year Grants
• Section: 国内
• Research Field: Perceptual information processing
• Research Institution: Keio University
• Principal Investigator: 森 尚平 慶應義塾大学, 理工学部, 特別研究員(PD)
• Project Period (FY): 2016-04-22 – 2019-03-31
• Project Status: Completed (Fiscal Year 2018)
• Budget Amount: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000); Fiscal Year 2018: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000); Fiscal Year 2017: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000); Fiscal Year 2016: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
• Keywords: 複合現実感 / 隠消現実感 / 多視点カメラ / Light Field Rendering / Image-Based Rendering / Multi-View Cameras

Outline of Annual Research Achievements

本研究の目的は，実空間に仮想物体を実時間合成する複合現実感 (Mixed Reality; MR) と実空間から実物体を実時間除去する隠消現実感 (Diminished Reality; DR) における，ライトフィールドデータの利用方法の模索，手法の提案，実動作するシステム開発を通した実証である．本年度は，就職先の決定に伴い研究終了時期が早まったため，早期にこれまでの成果を学術論文としてまとめることに注力した．
具体的には，実物体を視覚的に隠蔽・消去・透過する技術であるDRに関する「ライトフィールドの概念に基づく多視点カメラを用いた手法の開発と評価」を行い，国内外の論文誌や会議へと投稿，採択された．主に，ライトフィールドデータに含まれる視覚的に不要なフェンスを除去する技術や医療用教示映像を作成する際に写る医師の頭部や手を透かして見せる応用技術を報告した．そして，MRに関連して，スポーツ分野における競技者の重心位置推定のために多視点カメラを用いた手法を応用した結果を報告した．
上記に加え，留学先にて着手した類似研究，多視点RGB-D画像（カラー画像とカラー画像に位置合わせされた奥行画像の組）シーケンスを用いた画像修復技術（不可視光線情報を周辺領域及び過去の光線情報から推定する手法）の研究も進め，国際会議に採択された．更に，DR/MRにおける研究活動を周辺分野に周知させる目的で昨年度に行った国外での招待講演が評価され，内容を再編集して投稿したOverview論文が国外論文誌へと採録された．また，拡張現実感，コンピュータビジョンと機械学に関する英書を習周辺分野の研究者たちと協力して翻訳した訳本を2冊出版した．

Research Progress Status

平成30年度が最終年度であるため、記入しない。

Strategy for Future Research Activity

平成30年度が最終年度であるため、記入しない。

Research Products

• [Int'l Joint Research] ICG, Graz University of Technology(オーストリア)
• [Journal Article] Superimposing Thermal-Infrared Data on 3D Structure Reconstructed by RGB Visual Odometry (2018)
  • Author(s): Masahiro Yamaguchi, Trong-Phuc Truong, Shohei Mori, Vincent Nozick, Hideo Saito, Shoji Yachida, and Hideaki Sato
  • Journal Title: IEICE Transactions on Information and Systems Volume: E101.D Issue: 5 Pages: 1296-1307
  • DOI: 10.1587/transinf.2017MVP0023
  • NAID: 130006729794
  • ISSN: 0916-8532, 1745-1361
  • Year and Date: 2018-05-01
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Removing fences from sweep motion videos using global 3D reconstruction and fence-aware light field rendering (2018)
  • Author(s): Lueangwattana Chanya, Mori Shohei, and Saito Hideo
  • Journal Title: Computational Visual Media Volume: 5 Issue: 1 Pages: 1-12
  • DOI: 10.1007/s41095-018-0126-8
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] An overview of augmented visualization: observing the real world as desired (2018)
  • Author(s): Mori Shohei and Saito Hideo
  • Journal Title: APSIPA Transactions on Signal and Information Processing Volume: 7 Issue: 1 Pages: 1-9
  • DOI: 10.1017/atsip.2018.13
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] A Work Area Visualization by Multi-View Camera-Based Diminished Reality (2017)
  • Author(s): Shohei Mori, Momoko Maezawa, and Hideo Saito
  • Journal Title: Trans. on MDPI Multimodal Technologies and Interaction Volume: 1 Issue: 3 Pages: 1-12
  • DOI: 10.3390/mti1030018
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] A Survey of Diminished Reality: Techniques for Visually Concealing, Eliminating, and Seeing Through Real Objects (2017)
  • Author(s): Shohei Mori, Sei Ikeda, and Hideo Saito
  • Journal Title: IPSJ Trans. on Computer Vision and Applications Volume: 9 Issue: 1 Pages: 1-14
  • DOI: 10.1186/s41074-017-0028-1
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Visual Inspection by Capturing a Rotating Industrial Part (2017)
  • Author(s): 鶏内朋也，森尚平，斎藤英雄，菅野純一，足立秀之
  • Journal Title: Journal of the Japan Society for Precision Engineering Volume: 83 Issue: 12 Pages: 1184-1191
  • DOI: 10.2493/jjspe.83.1184
  • NAID: 130006243020
  • ISSN: 0912-0289, 1882-675X
  • Peer Reviewed / Open Access
• [Journal Article] See-Through Vision Using a Fish-Eye Camera and a 3D-LIDAR (2017)
  • Author(s): 大石圭，森尚平，斎藤英雄
  • Journal Title: The Journal of The Institute of Image Information and Television Engineers Volume: 71 Issue: 11 Pages: J276-J279
  • DOI: 10.3169/itej.71.J276
  • NAID: 130006179637
  • ISSN: 1342-6907, 1881-6908
  • Peer Reviewed / Open Access
• [Journal Article] Considerations on Binocular Mismatching in Observation-Based Diminished Reality (2017)
  • Author(s): 森 尚平、池田 聖、松木 ひとみ、松永 知典、柴田 史久、木村 朝子、田村 秀行
  • Journal Title: Transactions of the Virtual Reality Society of Japan Volume: 22 Issue: 2 Pages: 251-258
  • DOI: 10.18974/tvrsj.22.2_251
  • NAID: 130005701331
  • ISSN: 1344-011X, 2423-9593
  • Peer Reviewed / Open Access
• [Journal Article] Removing Redundant Line Segments of 3D Model for Camera Pose Estimation (2017)
  • Author(s): 家永直人，Siim Meerits，森尚平，斎藤英雄
  • Journal Title: The Journal of The Institute of Image Information and Television Engineers Volume: 71 Issue: 6 Pages: J189-J191
  • DOI: 10.3169/itej.71.J189
  • NAID: 130005678894
  • ISSN: 1342-6907, 1881-6908
  • Peer Reviewed / Open Access
• [Journal Article] Re-Design and Implementation of MR-based Filmmaking System by Adding Diminished Reality Functions (2016)
  • Author(s): 李金霞，斉藤純哉，森尚平，池田聖，柴田史久，木村朝子，田村秀行
  • Journal Title: Transactions of the Virtual Reality Society of Japan Volume: 21 Issue: 3 Pages: 451-462
  • DOI: 10.18974/tvrsj.21.3_451
  • NAID: 130005268391
  • ISSN: 1344-011X, 2423-9593
  • Peer Reviewed / Open Access
• [Journal Article] DOMINO Toppling: An MR Attraction Focusing on R-V Continuum (2016)
  • Author(s): 平田遼太郎，石橋朋果，チェカネイ，森尚平，池田聖，柴田史久，木村朝子，田村秀行
  • Journal Title: Transactions of the Virtual Reality Society of Japan Volume: 21 Issue: 3 Pages: 463-472
  • DOI: 10.18974/tvrsj.21.3_463
  • NAID: 130005268393
  • ISSN: 1344-011X, 2423-9593
  • Peer Reviewed / Open Access
• [Journal Article] [Paper] Design and Construction of Data Acquisition Facilities for Diminished Reality Research (2016)
  • Author(s): Shohei Mori, Yuya Eguchi, Sei Ikeda, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Journal Title: ITE Transactions on Media Technology and Applications Volume: 4 Issue: 3 Pages: 259-268
  • DOI: 10.3169/mta.4.259
  • NAID: 130005161891
  • ISSN: 2186-7364
  • Peer Reviewed / Open Access / Int'l Joint Research / Acknowledgement Compliant
• [Presentation] マルチカメラ搭載型無影灯を使用した手術動画を利用した作業領域のDR表示 (2019)
  • Author(s): 前澤桃子，大石圭，森尚平，斎藤英雄，梶田大樹，高詰佳史
  • Organizer: 研究報告コンピュータビジョンとイメージメディア
• [Presentation] Perceived Weight of a Rod under Augmented and Diminished Reality Visual Effects (2018)
  • Author(s): Satoshi Hashiguchi, Shohei Mori, Miho Tanaka, Fumihisa Shibata, and Asako Kimura
  • Organizer: The ACM Symp. on Virtual Reality Software and Technology
  • Int'l Joint Research
• [Presentation] Diminishing Fence from Sweep Image Sequences Using Structure from Motion and Light Field Rendering (2018)
  • Author(s): Chanya Lueangwattana, Shohei Mori, and Hideo Saito
  • Organizer: Asia Pacific Workshop on Mixed and Augmented Reality
  • Int'l Joint Research
• [Presentation] 3D PixMix: Image Inpainting in 3D Environment (2018)
  • Author(s): Shohei Mori, Jan Herling, Wolfgang Broll, Norihiko Kawai, Hideo Saito, Dieter Schmalstieg, and Denis Kalkofen
  • Organizer: IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR)-Adjunct
  • Int'l Joint Research
• [Presentation] Estimation of Center of Mass for Sports Scene Using Weighted Visual Hull (2018)
  • Author(s): Tomoya Kaichi, Shohei Mori, Hideo Saito, Kosuke Takahashi, Dan Mikami, Mariko Isogawa, and Hideaki Kimata
  • Organizer: IEEE Conf. on Computer Vision and Pattern Recognition Workshops
  • Int'l Joint Research
• [Presentation] BrightView: Increasing Perceived Brightness of Optical See-Through Head-Mounted Displays Through Unnoticeable Incident Light Reduction (2018)
  • Author(s): Shohei Mori, Sei Ikeda, Alexander Plopski, and Christian Sandor
  • Organizer: IEEE Virtual Reality
  • Int'l Joint Research
• [Presentation] A Refocus-Interface for Diminished Reality Work Area Visualization (2018)
  • Author(s): Momoko Maezawa, Shohei Mori, and Hideo Saito
  • Organizer: Stereoscopic Displays and Applications at IS&T Electronic Imaging Symposium
  • Int'l Joint Research
• [Presentation] Multi-view Surface Inspection Using a Rotating Table (2018)
  • Author(s): Tomoya Kaichi, Shohei Mori, Hideo Saito, Junichi Sugano, and Hideyuki Adachi
  • Organizer: Intelligent Robotics and Industrial Applications Using Computer Vision at IS&T Electronic Imaging
  • Int'l Joint Research
• [Presentation] Augmented Visualization: Observing as Desired (2017)
  • Author(s): Shohei Mori and Hideo Saito
  • Organizer: APSIPA ASC
  • Int'l Joint Research / Invited
• [Presentation] Augmented and Diminished Reality: Computational Imaging of Existence and Non-Existence (2017)
  • Author(s): Shohei Mori
  • Organizer: Int. Display Workshop
  • Int'l Joint Research / Invited
• [Presentation] Reference Framework on vSRT-method Benchmarking for MAR (2017)
  • Author(s): Ryosuke Ichikari, Takeshi Kurata, Koji Makita, Takefumi Taketomi, Hideaki Uchiyama, Tomotsugu Kondo, Shohei Mori, and Fumihisa Shibata
  • Organizer: ICAT-EGVE
  • Int'l Joint Research
• [Presentation] Registration of RGB and Thermal Point Clouds Generated by Structure From Motion (2017)
  • Author(s): Trong Phuc Truong, Masahiro Yamaguchi, Shohei Mori, Vincent Nozick, and Hideo Saito
  • Organizer: Multi-Sensor Fusion for Dynamic Scene Understanding in IEEE Int. Conf. on Computer Vision
  • Int'l Joint Research
• [Presentation] Semantic Object Selection and Detection for Diminished Reality based on SLAM with Viewpoint Class (2017)
  • Author(s): Yoshikatsu Nakajima, Shohei Mori, and Hideo Saito
  • Organizer: ISMAR Adjunct
  • Int'l Joint Research
• [Presentation] An Instant See-Through Vision System Using a Wide Field-of-View Camera and a 3D-Lidar (2017)
  • Author(s): Kei Oishi, Shohei Mori, and Hideo Saito
  • Organizer: ISMAR Adjunct
  • Int'l Joint Research
• [Presentation] A Multi-view Camera-based Diminished Reality for Work Area Visualization (2017)
  • Author(s): Momoko Maezawa, Shohei Mori, and Hideo Saito
  • Organizer: Asia Pacific Workshop on Mixed and Augmented Reality
  • Int'l Joint Research
• [Presentation] Influence on Weight Sensation Caused by Visual Diminishing of Real Objects (2017)
  • Author(s): Miho Tanaka, Ayushi Misra, Kana Oshima, Satoshi Hashiguchi, Shohei Mori, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Organizer: Asia Pacific Workshop on Mixed and Augmented Reality
  • Int'l Joint Research
• [Presentation] Empowering a POB-Diminished Reality Method to Handle Rigid Moving Objects with Real-time Observation (2017)
  • Author(s): Masaru Horita, Daiki Sakauchi, Shohei Mori, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Organizer: Asia Pacific Workshop on Mixed and Augmented Reality
  • Int'l Joint Research
• [Presentation] BrightView: Increasing Perceived Brightness in Optical See-through Head-mounted Displays (2017)
  • Author(s): Shohei Mori, Sei Ikeda, Christian Sandor, and Alexander Plopski
  • Organizer: ISMAR-Adjunct
  • Int'l Joint Research
• [Presentation] Further Experiments and Considerations on Weight Perception Caused by Visual Diminishing of Real Objects (2017)
  • Author(s): Miho Tanaka, Ayushi Misra, Kana Oshima, Satoshi Hashiguchi, Shohei Mori, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Organizer: ISMAR-Adjunct
  • Int'l Joint Research
• [Presentation] Background image registration as a post-processing technique in diminished reality rendering procedures (2017)
  • Author(s): Shohei Mori, Jianing Qie, Sei Ikeda, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Organizer: ISMAR-Adjunct
  • Int'l Joint Research
• [Presentation] Design and Implementation of a Common Dataset for Comparison and Evaluation of Diminished Reality Methods (2017)
  • Author(s): Taiki Morozumi, Shohei Mori, Sei Ikeda, Fumihisa Shibata, Asako Kimura, and Hideyuki Tamura
  • Organizer: ISMAR-Adjunct
  • Int'l Joint Research
• [Presentation] 部品回転時の画素値の推移を利用した外観検査手法 (2017)
  • Author(s): 鶏内朋也，森尚平，斎藤英雄，菅野純一，足立秀之
  • Organizer: ViEW
• [Presentation] 複数視点画像を用いた人間の重心位置推定手法の検討 (2017)
  • Author(s): 鶏内朋也，森尚平，斎藤英雄，高橋康輔，三上弾，五十川麻理子，木全英明
  • Organizer: 信学技報
• [Presentation] 一様に分散させた複数視点からの物体認識による隠消現実感 (2017)
  • Author(s): 中島由勝，森尚平，斎藤英雄
  • Organizer: 日本バーチャルリアリティ学会大会論文集
• [Presentation] 的隠背景可視化のためのライトフィールドレンダリングに基づくDR表示 (2017)
  • Author(s): 前澤桃子，家永直人，森尚平，斎藤英雄
  • Organizer: 研究報告コンピュータビジョンとイメージメディア
• [Presentation] 全方位カメラと3D-LIDARを用いた自由視点画像による隠消現実感 (2017)
  • Author(s): 大石圭，森尚平，斎藤英雄
  • Organizer: 研究報告コンピュータビジョンとイメージメディア
• [Presentation] 回転台を用いた多視点撮影による工業製品の外観検査 (2017)
  • Author(s): 鶏内朋也，森尚平，斎藤英雄，菅野純一
  • Organizer: Meeting on Image Recognino and Understanding
• [Presentation] 遠赤外線カメラとRGBカメラの多視点画像群を用いたRGB画像への温度マップ重畳表示 (2017)
  • Author(s): 山口真弘，Trong Phuc Truong，森尚平，Vincent Nozick，斎藤英雄，谷内田尚司，佐藤秀昭
  • Organizer: Meeting on Image Recognino and Understanding
• [Presentation] Augmented Visualization：望みどおりに見るための映像生成に向けて (2017)
  • Author(s): 斎藤英雄，森尚平
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 名城大学（愛知県・名古屋市）
  • Invited
• [Presentation] 非整列多視点カメラを用いた画像ベース隠背景可視化 (2017)
  • Author(s): 森尚平，前澤桃子, 家永直人，斎藤英雄
  • Organizer: 第2回PoTS映像学シンポジウム
  • Place of Presentation: 慶應義塾大学（神奈川県・横浜市）
• [Presentation] 多視点カメラを用いたライトフィールドレンダリングによる非平面隠背景可視化 (2017)
  • Author(s): 前澤桃子，家永直人，森尚平，斎藤英雄
  • Organizer: 日本バーチャルリアリティ学会複合現実感研究会
  • Place of Presentation: 京都大学（京都府・京都市）
• [Presentation] 隠消現実感技術を用いた実物体の視覚的消去による触力覚への影響 (2017)
  • Author(s): 田中美帆，大嶋佳奈，橋口哲志，森尚平，木村朝子，柴田史久，田村秀行
  • Organizer: 第171回ヒューマンコンピュータインタラクション研究発表会
  • Place of Presentation: 大濱信泉記念館（沖縄県・石垣市）
• [Presentation] テレセントリックレンズを用いた多視点画像による工業部品の傷検出 (2017)
  • Author(s): 鶏内朋也，森尚平，斎藤英雄，菅野純一，足立秀之
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 3D-LIDARと魚眼カメラを用いた隠消現実感 (2017)
  • Author(s): 大石圭，森尚平，斎藤英雄
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 実時間観測の援用により移動物体への拡張を可能にした隠消現実感の実現 (2017)
  • Author(s): 堀田勝，坂内大樹，森尚平，池田聖，柴田史久，木村朝子，田村秀行
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 動的非平面隠背景可視化のためのライトフィールドレンダリング (2017)
  • Author(s): 前澤桃子, 家永直人, 森尚平, 斎藤英雄
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 隠消現実感における両眼視野不整合についての分析と対策(3) ～Light Field Renderingを用いた隠背景再構成に生じるジッタの分析～ (2017)
  • Author(s): 松永知典，森尚平，池田聖，柴田史久，木村朝子，田村秀行
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 隠消現実感手法の客観評価のための共通データセットの開発 (2017)
  • Author(s): 両角泰希，森尚平，池田聖，柴田史久，木村朝子，田村秀行
  • Organizer: 情報処理学会第79回全国大会
  • Place of Presentation: 名古屋大学（愛知県・名古屋市）
• [Presentation] 隠消現実感とPoTS映像学 (2016)
  • Author(s): 森尚平
  • Organizer: 情報科学技術フォーラム
  • Place of Presentation: 富山大学（富山県・富山市）
  • Invited
• [Presentation] Mixed and Diminished Reality for Filmmaking (2016)
  • Author(s): Shohei Mori
  • Organizer: Collective Visual Sensing Workshop
  • Place of Presentation: Keio University (Yokohama, Kanagawa)
  • Int'l Joint Research / Invited
• [Presentation] Detour light field rendering for diminished reality using unstructured multiple views (2016)
  • Author(s): Shohei Mori, Momoko Maezawa, Naoto Ienaga, and Hideo Saito
  • Organizer: Int. Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR)
  • Place of Presentation: Merida (Mexico)
  • Int'l Joint Research
• [Presentation] First deployment of diminished reality for anatomy education (2016)
  • Author(s): Naoto Ienaga, Felix Bork, Siim Meerits, Shohei Mori, Pascal Fallavolita, Nassir Navab, and Hideo Saito
  • Organizer: Int. Workshop on Diminished Reality as Challenging Issue in Mixed and Augmented Reality (IWDR)
  • Place of Presentation: Merida (Mexico)
  • Int'l Joint Research
• [Presentation] 多視点カメラを用いたライトフィールドレンダリングによる隠消現実感表示 (2016)
  • Author(s): 森尚平，前澤桃子，家永直人，斎藤英雄
  • Organizer: 日本バーチャルリアリティ学会複合現実感研究会
  • Place of Presentation: 支笏湖丸駒温泉（北海道・千歳市）
• [Presentation] 動的隠背景の可視化のためのライトフィールドレンダリング (2016)
  • Author(s): 前澤桃子，家永直人，森尚平，斎藤英雄
  • Organizer: 日本バーチャルリアリティ学会大会
  • Place of Presentation: つくば国際会議場（茨城県・つくば市）
• [Book] ARの教科書 (2018)
  • Author(s): （著）Dieter Schmalstieg，Tobias Hollerer，（編）池田聖，酒田信親，山本豪志朗，（訳）池田聖，一刈良介，大槻麻衣，河合紀彦，酒田信親，武富貴史，藤本雄一郎，森尚平，山本豪志朗
  • Total Pages: 528
  • Publisher: マイナビ出版
  • ISBN: 9784839965365
• [Book] OpenCVとPythonによる機械学習プログラミング (2018)
  • Author(s): （著）Michael Beyeler，（訳）池田聖，浦西友樹，中島悠太，森尚平，山添大丈，山本豪志朗
  • Total Pages: 352
  • Publisher: マイナビ出版
  • ISBN: 9784839965303
• [Remarks] 自身の研究成果をまとめたWebサイト
  • URL: https://mugichoko445.bitbucket.io/
• [Remarks] BrightView Project
  • URL: https://github.com/Mugichoko445/BrightView
• [Remarks] Shohei Mori's Website
  • URL: http://hvrl.ics.keio.ac.jp/mori/