Research on removal of real objects from images in video see-through augmented reality

• Project/Area Number: 24700118
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Media informatics/Database
• Research Institution: Nara Institute of Science and Technology
• Principal Investigator: KAWAI Norihiko 奈良先端科学技術大学院大学, 情報科学研究科, 助教 (30610670)
• Project Period (FY): 2012-04-01 – 2015-03-31
• Project Status: Completed (Fiscal Year 2014)
• Budget Amount: ¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000); Fiscal Year 2014: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000); Fiscal Year 2013: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000); Fiscal Year 2012: ¥1,820,000 (Direct Cost: ¥1,400,000、Indirect Cost: ¥420,000)
• Keywords: 隠消現実感 / 画像修復 / 拡張現実感 / 物体除去 / 背景推定 / 形状修復

Outline of Final Research Achievements

This study developed diminished reality methods for visually removing real objects from video images in real time in video see-through augmented reality. Specifically, we developed two methods for the cases when target objects are augmented reality markers and general objects. For augmented reality markers, we achieved real-time marker removal by automatically detecting AR markers and generating background images behind markers using image inpainting, and overlaying the generated images on the marker regions with geometric and photometric corrections. For general objects, we first estimated camera poses and geometry behind a target object on the basis of a SLAM (Simultaneous Localization and Mapping) technique. We then achieved real-time removal of general objects for a 3D scene by generating background images using image inpainting and overlaying the generated images with geometric and photometric corrections using the estimated camera poses and background geometry.

Research Products

• [Journal Article] インペインティングとその隠消現実感への応用 (2014)
  • Author(s): 河合紀彦
  • Journal Title: システム制御情報学会誌 Volume: 58 Pages: 426-431
• [Journal Article] From image inpainting to diminished reality (2014)
  • Author(s): Norihiko Kawai, Tomokazu Sato, and Naokazu Yokoya
  • Journal Title: Proc. Human-Computer Interaction International Volume: 16 Pages: 363-374
  • DOI: 10.1007/978-3-319-07458-0_34
  • NAID: 120006659635
  • ISBN: 9783319074573, 9783319074580
  • Peer Reviewed / Acknowledgement Compliant
• [Journal Article] [Paper] Diminished Reality for AR Marker Hiding Based on Image Inpainting with Reflection of Luminance Changes (2013)
  • Author(s): Norihiko Kawai, Masayoshi Yamasaki, Tomokazu Sato, and Naokazu Yokoya
  • Journal Title: ITE Transactions on Media Technology and Applications Volume: 1 Issue: 4 Pages: 343-353
  • DOI: 10.3169/mta.1.343
  • NAID: 130004648180
  • ISSN: 2186-7364
  • Peer Reviewed
• [Presentation] ARマーカ除去のための実時間背景画像変形 (2015)
  • Author(s): 河合紀彦
  • Organizer: 複合現実感研究会
  • Place of Presentation: 奈良先端科学技術大学院大学 (奈良県生駒市)
  • Year and Date: 2015-01-22 – 2015-01-23
• [Presentation] Diminished reality based on image inpainting for visually removing real objects in real time (2014)
  • Author(s): Norihiko Kawai
  • Organizer: Int. Display Workshops (IDW)
  • Place of Presentation: 朱鷺メッセ (新潟県新潟市)
  • Year and Date: 2014-12-03 – 2014-12-05
  • Invited
• [Presentation] 点群へのプリミティブ当てはめを用いた三次元形状とテクスチャの欠損修復 (2014)
  • Author(s): 河合紀彦
  • Organizer: 電気関係学会関西連合大会
  • Place of Presentation: 奈良先端科学技術大学院大学 (奈良県生駒市)
  • Year and Date: 2014-11-23 – 2014-11-24
• [Presentation] 透視投影歪みを自動補正した画像修復 (2014)
  • Author(s): 笹尾海斗
  • Organizer: 電気関係学会関西連合大会
  • Place of Presentation: 奈良先端科学技術大学院大学 (奈良県生駒市)
  • Year and Date: 2014-11-23 – 2014-11-24
• [Presentation] 画像修復を用いた隠消現実感 (2014)
  • Author(s): 河合紀彦
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
  • Place of Presentation: 岡山コンベンションセンター (岡山県岡山市)
  • Year and Date: 2014-07-28 – 2014-07-31
• [Presentation] Diminished reality using image inpainting (2014)
  • Author(s): Norihiko Kawai
  • Organizer: Int. Conf. on Informatics, Electronics & Vision (ICIEV)
  • Place of Presentation: Dhaka (Bangladesh)
  • Year and Date: 2014-06-15 – 2014-06-17
  • Invited
• [Presentation] Diminished reality by correction of perspective and color with image inpainting (2014)
  • Author(s): Norihiko Kawai
  • Organizer: SIAM Conf. on Imaging Science
  • Place of Presentation: Hong Kong (China)
  • Year and Date: 2014-05-12 – 2014-05-14
  • Invited
• [Presentation] 点群へのプリミティブ当てはめに基づく形状とテクスチャの欠損修復 (2014)
  • Author(s): 中尾 聡志
  • Organizer: 電子情報通信学会 パターン認識・メディア理解研究会
  • Place of Presentation: 福岡大学(福岡市)
• [Presentation] 透視投影歪みの自動補正に基づく画像修復 (2014)
  • Author(s): 笹尾 海斗
  • Organizer: 2014年電子情報通信学会総合大会
  • Place of Presentation: 新潟大学(新潟市)
• [Presentation] Diminished reality considering background structures (2013)
  • Author(s): Norihiko Kawai
  • Organizer: IEEE International Symposium on Mixed and Augmented Reality
  • Place of Presentation: Adelaide, Australia
• [Presentation] 点群への平面当てはめに基づく形状とテクスチャの欠損修復 (2013)
  • Author(s): 中尾 聡志
  • Organizer: 電気関係学会関西支部連合大会
  • Place of Presentation: 大阪電気通信大学(寝屋川市)
• [Presentation] AR marker hiding based on image inpainting and reflection of illumination changes (2012)
  • Author(s): Norihiko Kawai
  • Organizer: 11th IEEE and ACM Int. Sympo. on Mixed and Augmented Reality
  • Place of Presentation: Atlanta, Georgia, USA
• [Presentation] ARマーカの除去のための画像修復に基づく陰影を考慮した隠消現実感
  • Author(s): 河合紀彦
  • Organizer: 画像の認識・理解シンポジウム
  • Place of Presentation: 福岡県福岡市
• [Remarks] 研究成果
  • URL: http://yokoya.naist.jp/~norihi-k/research.html
• [Remarks] Research
  • URL: http://yokoya.naist.jp/~norihi-k/research.html