• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Extended MapReduce framework with distributed masters for high-reliablity and high-performance

Research Project

Project/Area Number 23700036
Research Category

Grant-in-Aid for Young Scientists (B)

Allocation TypeMulti-year Fund
Research Field Software
Research InstitutionOsaka University

Principal Investigator

OKITA Masao  大阪大学, 情報科学研究科, 助教 (50563988)

Project Period (FY) 2011 – 2013
Project Status Completed (Fiscal Year 2013)
Budget Amount *help
¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2013: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Fiscal Year 2012: ¥650,000 (Direct Cost: ¥500,000、Indirect Cost: ¥150,000)
Fiscal Year 2011: ¥2,990,000 (Direct Cost: ¥2,300,000、Indirect Cost: ¥690,000)
Keywords並列処理・分散処理 / MapReduce / 並列分散処理 / 高性能計算 / PageRank / 耐故障性
Research Abstract

To improve MapReduce, a popular large-scale distributed programming framework, we developed a high-reliable framework and techniques for accelerating a MapReduce application.
Firstly, we extend an existing MapReduce framework to improving fault-tolerance. In the existing framework, the master computer is the single point of failure. Our extended framework provides multiple master computers and switches them to avoid a system failure.
Secondly, we developed optimizing techniques for intermediate data processing to improve application performance. One of our technique reduces the amount of intermediate data by compressing data from an alternative viewpoint. Another technique reduces intermediate data transfer by improving data locality in task assignment.

Report

(4 results)
  • 2013 Annual Research Report   Final Research Report ( PDF )
  • 2012 Research-status Report
  • 2011 Research-status Report
  • Research Products

    (27 results)

All 2014 2013 2012 2011 Other

All Journal Article (3 results) (of which Peer Reviewed: 3 results,  Open Access: 1 results,  Acknowledgement Compliant: 1 results) Presentation (24 results)

  • [Journal Article] Accelerating ODE-based Simulation of General and Heterogeneous Biophysical Models using a GPU2014

    • Author(s)
      Tomohiro Okuyama, Masao Okita, Takeshi Abe, Yoshiyuki Asai, Hiroaki Kitano, Taishin Nomura, and Kenichi Hagihara
    • Journal Title

      IEEE Transactions on Parallel and Distributed Systems

      Volume: 201(印刷中) Issue: 8 Pages: 1966-1975

    • DOI

      10.1109/tpds.2013.198

    • Related Report
      2013 Annual Research Report 2013 Final Research Report
    • Peer Reviewed
  • [Journal Article] A Versatile Platform for Multilevel Modeling of Physiological Systems: SBML-PHML Hybrid Modeling and Simulation2014

    • Author(s)
      Asai Y, Abe T, Oka H, Okita M, Hagihara K, Ghosh S, Matsuoka Y, Kurachi Y, Nomura T, Kitano H
    • Journal Title

      Advanced Biomedical Engineering

      Volume: 3 Issue: 0 Pages: 50-58

    • DOI

      10.14326/abe.3.50

    • NAID

      130005001705

    • ISSN
      2187-5219
    • Related Report
      2013 Final Research Report
    • Peer Reviewed / Open Access / Acknowledgement Compliant
  • [Journal Article] A Versatile Platform for Multilevel Modeling of Physiological Systems: SBML-PHML Hybrid Modeling and Simulation2014

    • Author(s)
      Yoshiyuki Asai, Takeshi Abe, Hideki Oka, Masao Okita, Ken ichi Hagihara, Samik Ghosh, Yukiko Matsuoka, Yoshihisa Kurachi, Taishin Nomura, Hiroaki Kitano
    • Journal Title

      Advanced Biomedical Engineering

      Volume: 99 Pages: 1-1

    • NAID

      130005001705

    • Related Report
      2013 Annual Research Report
    • Peer Reviewed
  • [Presentation] A versatile platform for multilevel modeling of physiological systems : Template/instance framework for large-scale modeling and simulation2013

    • Author(s)
      Yoshiyuki Asai, Takeshi Abe, Hideki Oka, Masao Okita, Tomohiro Okuyama, Kenichi Hagihara, Samik Ghosh, Yukiko Matsuoka, and Hiroaki Kitano
    • Organizer
      35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'13)
    • Place of Presentation
      Osaka, Japan
    • Year and Date
      2013-07-06
    • Related Report
      2013 Final Research Report
  • [Presentation] MapReduce を用いたグラフアプリケーションにおける重複メッセージの排除による高速化2013

    • Author(s)
      黒松信行, 置田真生, 萩原兼一
    • Organizer
      第11回先進的計算基盤システムシンポジウム論文集(SACSIS 2013)
    • Place of Presentation
      仙台, 日本
    • Year and Date
      2013-05-22
    • Related Report
      2013 Final Research Report
  • [Presentation] Towards massively parallel general biophysical simulation on the K computer'2013

    • Author(s)
      Masao Okita, Tomohiro Okuyama, Takeshi Asai, Yoshiyuki Asai, Hiroaki Kitano, and Hagihara Kenichi
    • Organizer
      the 2nd HD Physiology International Symposium 2013,
    • Place of Presentation
      Tokyo, Japan
    • Related Report
      2013 Annual Research Report
  • [Presentation] MapReduce を用いたグラフアプリケーションにおける重複メッセージの排除による高速化2013

    • Author(s)
      黒松信行, 置田真生, 萩原兼一
    • Organizer
      第11回先進的計算基盤システムシンポジウム(SACSIS 2013)
    • Place of Presentation
      仙台,日本
    • Related Report
      2012 Research-status Report
  • [Presentation] Evolving fault-tolerance in Hadoop with robust auto-recovering JobTracker2012

    • Author(s)
      Nobuyuki Kuromatsu, Masao Okita, and Kenichi Hagihara
    • Organizer
      Proceedings of the 3rd International Conference on Networking and Computing (ICNC 2012)
    • Place of Presentation
      Okinawa, Japan
    • Year and Date
      2012-12-07
    • Related Report
      2013 Final Research Report
  • [Presentation] Hadoop の性能ボトルネックを特定するための実行トレース可視化ツール2012

    • Author(s)
      古谷達朗, 置田真生, 萩原兼一
    • Organizer
      平成24年度情報処理学会関西支部支部大会講演論文集
    • Place of Presentation
      大阪, 日本
    • Year and Date
      2012-09-21
    • Related Report
      2013 Final Research Report
  • [Presentation] Multilevel modeling of Physiological Systems and Simulation Platform : PhysioDesigner, Flint and Flint K3 service2012

    • Author(s)
      Yoshiyuki Asai, Takeshi Abe, Masao Okita, Tomohiro Okuyama, Nobukazu Yoshioka, Shigetoshi Yokoyama, Masaru Nagaku, Kenichi Hagihara, and Hiroaki Kitano
    • Organizer
      Proceedings of the 12th IEEE/IPSJ International Symposium on Applications and the Internet (SAINT 2012)
    • Place of Presentation
      Izmir, Turkey
    • Year and Date
      2012-07-17
    • Related Report
      2013 Final Research Report
  • [Presentation] 異種混合生体シミュレータinsilicoSimの自動並列化2012

    • Author(s)
      置田真生
    • Organizer
      日本機械学会第24回バイオエンジニアリング講演会
    • Place of Presentation
      大阪大学豊中キャンパス(大阪府)
    • Related Report
      2011 Research-status Report
  • [Presentation] 並列シミュレータ insilicoSim における モテル規模に応じたエージェント割り当て手法2012

    • Author(s)
      福井伸弥
    • Organizer
      2012年電子情報通信学会総合大会
    • Place of Presentation
      岡山大学津島キャンパス(岡山県)
    • Related Report
      2011 Research-status Report
  • [Presentation] 系列パターン抽出のMapReduce 実装におけるタスク分割方式の検討2011

    • Author(s)
      井上佑希, 置田真生, 萩原兼一
    • Organizer
      情報処理学会研究報告, 2010-HPC-130
    • Place of Presentation
      鹿児島,日本
    • Year and Date
      2011-07-29
    • Related Report
      2013 Final Research Report
  • [Presentation] Hadoop におけるジョブトラッカの耐故障機能の検討2011

    • Author(s)
      黒松信行, 置田真生, 萩原兼一
    • Organizer
      第9回先進的計算基盤システムシンポジウム論文集(SACSIS 2011)
    • Place of Presentation
      東京, 日本
    • Year and Date
      2011-05-26
    • Related Report
      2013 Final Research Report
  • [Presentation] Hadoopにおけるジョブトラッカの耐故障機能の検討2011

    • Author(s)
      黒松信行
    • Organizer
      第9回先進的計算基盤システムシンポジウム
    • Place of Presentation
      秋葉原コンベンションホール(東京都)
    • Related Report
      2011 Research-status Report
  • [Presentation] 系列パターン抽出のMapReduce実装におけるタスク分割方式の検討2011

    • Author(s)
      井上佑希
    • Organizer
      第130回情報処理学会ハイパフォーマンスコンピューティング研究会
    • Place of Presentation
      かごしま県民交流センター(鹿児島県)
    • Related Report
      2011 Research-status Report
  • [Presentation] インタプリタ型汎用生体シミュレータinsilicoSimのGPUによる高速化2011

    • Author(s)
      奥山倫弘
    • Organizer
      第130回情報処理学会ハイパフォーマンスコンピューティング研究会
    • Place of Presentation
      かごしま県民交流センター(鹿児島県)
    • Related Report
      2011 Research-status Report
  • [Presentation] 汎用生体シミュレータinsilicoSimにおける生体モデル内の依存関係に着目したスケジューリング2011

    • Author(s)
      松井龍
    • Organizer
      平成23年度情報処理学会関西支部支部大会
    • Place of Presentation
      大阪大学中之島センター(大阪府)
    • Related Report
      2011 Research-status Report
  • [Presentation] Accelerating General and Heterogeneous Biophysical Simulation Using the GPU Interpreter for Solving ODEs2011

    • Author(s)
      Tomohiro Okuyama
    • Organizer
      the 4th Global COE International Symposium on Physiome and Systems Biology for Integrated Life Sciences and Predictive Medicine
    • Place of Presentation
      千里ライフサイエンスセンター(大阪府)
    • Related Report
      2011 Research-status Report
  • [Presentation] PhysioDesigner and Flint: a platform for multilevel modeling of physiological systems and simulation

    • Author(s)
      Yoshiyuki Asai, Takeshi Abe, Hideki Oka, Masao Okita, Tomohiro Okuyama, Kenichi Hagihara, Yoshihisa Kurachi, Hiroaki Kitano
    • Organizer
      5th International Conference on Computational Bioengineering (ICCB 2013)
    • Place of Presentation
      Leuven, Belgium
    • Related Report
      2013 Annual Research Report
  • [Presentation] A versatile platform for multilevel modeling of physiological systems: Template/instance framework for large-scale modeling and simulation

    • Author(s)
      Yoshiyuki Asai, Takeshi Abe, Hideki Oka, Masao Okita, Tomohiro Okuyama, Kenichi Hagihara, Samik Ghosh, Yukiko Matsuoka, Hiroaki Kitano
    • Organizer
      35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'13)
    • Place of Presentation
      Osaka, Japan
    • Related Report
      2013 Annual Research Report
  • [Presentation] 汎用生体モデルの高速な並列シミュレーションのための階層的依存グラフの自動分割

    • Author(s)
      吉川禎, 中村亜紀, 置田真生, 安部武志, 浅井義之, 北野宏明, 野村泰伸, 萩原兼一
    • Organizer
      情報処理学会研究報告, 2014-HPC-143
    • Place of Presentation
      石川、日本
    • Related Report
      2013 Annual Research Report
  • [Presentation] GPU クラスタを用いた大規模生体シミュレーションにおける CPU・GPU 間データ転送の効率化

    • Author(s)
      大越雄一, 置田真生, 安部武志, 浅井義之, 北野宏明, 萩原兼一
    • Organizer
      情報処理学会研究報告, 2013-HPC-140
    • Place of Presentation
      福岡、日本
    • Related Report
      2013 Annual Research Report
  • [Presentation] Accelerating General and Heterogeneous Biophysical Simulations Using the GPU

    • Author(s)
      Tomohiro Okuyama, Masao Okita, Takeshi Abe, Yoshiyuki Asai, Taishin Nomura, Hiroaki Kitano, Kenichi Hagihara
    • Organizer
      The 4th GPU Technology Conference (GTC 2013)
    • Place of Presentation
      San Jose, CA, USA
    • Related Report
      2012 Research-status Report
  • [Presentation] Evolving fault-tolerance in Hadoop with robust auto-recovering JobTracker

    • Author(s)
      Nobuyuki Kuromatsu, Masao Okita, Kenichi Hagihara
    • Organizer
      The International Workshop on Networking, Computing, Systems, and Software (NCSS 2012)
    • Place of Presentation
      Okinawa, Japan
    • Related Report
      2012 Research-status Report
  • [Presentation] Hadoopの性能ボトルネックを特定するための実行トレース可視化ツール

    • Author(s)
      古谷達朗, 置田真生, 萩原兼一
    • Organizer
      平成24年度情報処理学会関西支部支部大会
    • Place of Presentation
      大阪, 日本
    • Related Report
      2012 Research-status Report
  • [Presentation] 生体モデルにおける数式の類似性に着目したOpenMPによる並列シミュレーションの検討

    • Author(s)
      吉川禎, 奥山倫弘, 置田真生, 浅井義之, 安部武志, 野村泰伸, 八木哲也, 萩原兼一
    • Organizer
      第10回先進的計算基盤システムシンポジウム(SACSIS 2012)
    • Place of Presentation
      神戸, 日本
    • Related Report
      2012 Research-status Report

URL: 

Published: 2011-08-05   Modified: 2019-07-29  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi