• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Development of an environment design support mixed reality system capable of estimating the environment using deep learning

Research Project

Project/Area Number 19K12681
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Review Section Basic Section 90010:Design-related
Research InstitutionOsaka University

Principal Investigator

Fukuda Tomohiro  大阪大学, 工学研究科, 准教授 (80379114)

Project Period (FY) 2019-04-01 – 2022-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2021: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000)
Fiscal Year 2020: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Fiscal Year 2019: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Keywords設計支援 / 複合現実 / 深層学習 / 動的オクルージョン処理 / ドローン / 隠消現実 / 景観シミュレーション / インスタンスセグメンテーション / セマンティックセグメンテーション / リアルタイム物体検出 / 環境設計支援 / 景観 / ディープラーニング / 環境推定
Outline of Research at the Start

本研究は,「景観を創造する環境設計の3次元検討用モデルを現実空間の中で融合させて可視化を行うと共に,環境を自動的に推定することにより,関係者が環境の変化を客観的に検討可能なツールの実現」を核心として,都市・建築分野を主な対象として,現状や将来の環境変化を自動的に推定しながら直感的,科学的に計画設計評価が可能なMR(複合現実)システムの開発を行うものである.

Outline of Final Research Achievements

This research aimed to realize mixed reality (MR) for the urban and architectural fields, in which 3D virtual models of environmental design are visualized by merging them with the real world, and the environment can be automatically estimated and examined objectively by the stakeholders.
To this end, this research achieves that the methods of MR occlusion of real space for each object (accurate representation of the back and forth relationship between the virtual model and the real world), an integrated drone and MR that enables MR from the sky, advanced DR (diminished reality) that virtually eliminates the real world, a reflection representation for MR to design near water, and an automatic sample generation for deep learning of building planes and facades were developed, implemented, and validated.

Academic Significance and Societal Importance of the Research Achievements

本研究は,Society5.0すなわち「超スマート社会」の構築に向けて,深層学習とMRの統合を中心として現況と将来の環境認識と設計検討を高度に可能とするシステムを新たに開発した.研究を進める中で,ドローンとMRの統合や深層学習のデータセット生成法の開発へと発展させた.
建築・都市設計分野では,正確な情報公開や説明責任,参加型まちづくりへのニーズは益々高まっており,建設DX(デジタルトランスフォーメーション)の中で,計画・設計内容を直感的,客観的に評価でき合意形成のコストを低減するシステムは必要とされており,本研究の意義は高い.

Report

(4 results)
  • 2021 Annual Research Report   Final Research Report ( PDF )
  • 2020 Research-status Report
  • 2019 Research-status Report
  • Research Products

    (32 results)

All 2021 2020 2019 Other

All Journal Article (4 results) (of which Peer Reviewed: 4 results,  Open Access: 3 results) Presentation (16 results) (of which Int'l Joint Research: 11 results,  Invited: 1 results) Book (1 results) Remarks (11 results)

  • [Journal Article] An enhanced 3D model and generative adversarial network for automated generation of horizontal building mask images and cloudless aerial photographs2021

    • Author(s)
      Ikeno Kazunosuke、Fukuda Tomohiro、Yabuki Nobuyoshi
    • Journal Title

      Advanced Engineering Informatics

      Volume: 50 Pages: 101380-101380

    • DOI

      10.1016/j.aei.2021.101380

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Assessing future landscapes using enhanced mixed reality with semantic segmentation by deep learning2021

    • Author(s)
      Kido Daiki、Fukuda Tomohiro、Yabuki Nobuyoshi
    • Journal Title

      Advanced Engineering Informatics

      Volume: 48 Pages: 101281-101281

    • DOI

      10.1016/j.aei.2021.101281

    • Related Report
      2021 Annual Research Report 2020 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Automatic Object Removal With Obstructed Fa?ades Completion Using Semantic Segmentation and Generative Adversarial Inpainting2021

    • Author(s)
      Zhang Jiaxin、Fukuda Tomohiro、Yabuki Nobuyoshi
    • Journal Title

      IEEE Access

      Volume: 9 Pages: 117486-117495

    • DOI

      10.1109/access.2021.3106124

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Diminished reality system with real-time object detection using deep learning for onsite landscape simulation during redevelopment2020

    • Author(s)
      Kido Daiki、Fukuda Tomohiro、Yabuki Nobuyoshi
    • Journal Title

      Environmental Modelling & Software

      Volume: 131 Pages: 104759-104759

    • DOI

      10.1016/j.envsoft.2020.104759

    • Related Report
      2020 Research-status Report
    • Peer Reviewed
  • [Presentation] Image-based Cityscape Visualization with Automatic Object Removal and Facade Inpainting Using Semantic Segmentation and Generative Adversarial Networks2021

    • Author(s)
      Jiaxin Zhang, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      SimAUD (Symposium on Simulation for Architecture + Urban Design) 2021
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Development of an Augmented Reality System with Reflection Implementation for Landscape Design Visualization using a Planar Reflection Method in Real-Time Rendering2021

    • Author(s)
      Hao Chen, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      Proceedings of the International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Automatic Diminished Reality-Based Virtual Demolition Method using Semantic Segmentation and Generative Adversarial Network for Landscape Assessment2021

    • Author(s)
      Takuya Kikuchi, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      Proceedings of the International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Landscape Visualization by Integrating Augmented Reality and Drones with Occlusion Handling to Link Real and Virtual Worlds: Towards city digital twin realization2021

    • Author(s)
      Naoki Kikuchi, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      Proceedings of the International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Mixed Reality Landscape Visualization Method with Automatic Discrimination Process for Dynamic Occlusion Handling Using Instance Segmentation2021

    • Author(s)
      Mizuki Nakabayashi, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      Proceedings of the International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Automatic Generation of Horizontal Building Mask Images by Using a 3D Model with Aerial Photographs Removed Thin Clouds by Generative Adversarial Network for Deep Learning2021

    • Author(s)
      Kazunosuke Ikeno, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      CAADRIA 2021
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] Automatic Generation of Horizontal Building Mask Images by Using a 3DModel with Aerial Photographs for Deep Learning2020

    • Author(s)
      Kazunosuke Ikeno, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      eCAADe 2020
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] Mobile mixed reality for environmental design using real-time semantic segmentation and video communication: Dynamic occlusion handling and green view index estimation2020

    • Author(s)
      Daiki Kido, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      CAADRIA 2020
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] x-Reality for Architectural & Urban Design: Towards super-smart society2020

    • Author(s)
      Tomohiro Fukuda
    • Organizer
      SIGraDi 2020
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research / Invited
  • [Presentation] A Landscape Simulation Method with One-by-one Dynamic Occlusion Using Instance Segmentation in Mixed Reality: Image generation method focusing on the grounding for occlusion2020

    • Author(s)
      Mizuki Nakabayashi, Tomohiro Fukuda and Nobuyoshi Yabuki
    • Organizer
      日本建築学会 第43回情報・システム・利用・技術シンポジウム
    • Related Report
      2020 Research-status Report
  • [Presentation] Augmented Reality landscape Simulation System with Occlusion Handling that Allows free Movement by a Small Drone: Improved hardware dependency2020

    • Author(s)
      Naoki Kikuchi, Tomohiro Fukuda and Nobuyoshi Yabuki
    • Organizer
      日本建築学会 第43回情報・システム・利用・技術シンポジウム
    • Related Report
      2020 Research-status Report
  • [Presentation] An On-site Landscape Simulation Method that Enables One-by-one Dynamic Occlusion in Mixed Reality with Instance Segmentation2020

    • Author(s)
      Mizuki Nakabayashi, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      2020年度日本建築学会大会(関東)
    • Related Report
      2020 Research-status Report
  • [Presentation] A Landscape Simulation Method with Free-Viewpoints Using a Small Drone and Web-based Augmented Reality2020

    • Author(s)
      Naoki Kikuchi, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      2020年度日本建築学会大会(関東)
    • Related Report
      2020 Research-status Report
  • [Presentation] Mobile Mixed Reality for Environmental Design Using Real-Time Semantic Segmentation and Video Communication: Dynamic Occlusion Handling and Green View Index Estimation2020

    • Author(s)
      Daiki Kido, Tomohiro Fukuda and Nobuyoshi Yabuki
    • Organizer
      The 25th International Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA 2020)
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] Development of a Semantic Segmentation System for Dynamic Occlusion Handling in Mixed Reality for Landscape Simulation2019

    • Author(s)
      Daiki Kido, Tomohiro Fukuda, and Nobuyoshi Yabuki
    • Organizer
      The eCAADe + SIGraDi 2019 Conference
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] Integration of Real-Time Video Communication and Semantic Segmentation over the Internet Toward Dynamic Occlusion Handling in Mobile Mixed Reality for Landscape Simulation2019

    • Author(s)
      Daiki Kido, Tomohiro Fukuda, Nobuyoshi Yabuki
    • Organizer
      日本建築学会 第42回情報・システム・利用・技術シンポジウム
    • Related Report
      2019 Research-status Report
  • [Book] Impact, Volume 2020, Number 22020

    • Author(s)
      Tomohiro Fukuda
    • Total Pages
      3
    • Publisher
      Science Impact Ltd
    • Related Report
      2019 Research-status Report
  • [Remarks] 障害物を除去し建物ファサードの全景を効率的に復元する方法を開発

    • URL

      https://resou.osaka-u.ac.jp/ja/research/2021/20210907_1

    • Related Report
      2021 Annual Research Report
  • [Remarks] X-ray street vision

    • URL

      https://resou.osaka-u.ac.jp/en/research/2021/20210907_1

    • Related Report
      2021 Annual Research Report
  • [Remarks] 航空写真の薄雲を自動で除去 深層学習用建物マスク画像の自動生成

    • URL

      https://resou.osaka-u.ac.jp/ja/research/2021/20210820_1

    • Related Report
      2021 Annual Research Report
  • [Remarks] Eye in the sky

    • URL

      https://resou.osaka-u.ac.jp/en/research/2021/20210820_1

    • Related Report
      2021 Annual Research Report
  • [Remarks] 景観シミュレーション用の DR に新技術:AI を統合して移動物体をリアルタイムに除去

    • URL

      https://resou.osaka-u.ac.jp/ja/research/2020/20200617_1

    • Related Report
      2020 Research-status Report
  • [Remarks] 景観シミュレーション用のMRに新技術:AIを統合して動的なオクルージョン処理と景観指標の推定を実現

    • URL

      https://resou.osaka-u.ac.jp/ja/research/2021/20210324_1

    • Related Report
      2020 Research-status Report
  • [Remarks] 【まちづくりに応用可】大阪大学大学院 仮想空間から移動物体を除去する「DRシステム」開発

    • URL

      https://www.kensetsunews.com/web-kan/467230

    • Related Report
      2020 Research-status Report
  • [Remarks] CAADRIA 2021でBest Paper Award Runner-Upを受賞

    • URL

      https://www.eng.osaka-u.ac.jp/ja/topics/researchresult/6282/

    • Related Report
      2020 Research-status Report
  • [Remarks] 「AIで現実のものを消し去る「DR」とは!?」子供の科学 2020年9月号

    • URL

      https://www.seibundo-shinkosha.net/magazine/kids/47319/

    • Related Report
      2020 Research-status Report
  • [Remarks] 英国の雑誌「Impact」に紹介されました。

    • URL

      http://www.see.eng.osaka-u.ac.jp/topics/media/4242.html

    • Related Report
      2019 Research-status Report
  • [Remarks] 英国の科学雑誌「Impact」に紹介されました。

    • URL

      http://fukudablog.hatenablog.com/entry/2020/04/08/095044

    • Related Report
      2019 Research-status Report

URL: 

Published: 2019-04-18   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi