Peta-scale Data Processing for Large-scale Heterogeneous Clusters

• Project/Area Number: 23700031
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Software
• Research Institution: Tokyo Institute of Technology
• Principal Investigator: SATO Hitoshi 東京工業大学, 学術国際情報センター, 特任助教 (00550633)
• Project Period (FY): 2011 – 2012
• Project Status: Completed (Fiscal Year 2012)
• Budget Amount: ¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000); Fiscal Year 2012: ¥1,950,000 (Direct Cost: ¥1,500,000、Indirect Cost: ¥450,000); Fiscal Year 2011: ¥2,210,000 (Direct Cost: ¥1,700,000、Indirect Cost: ¥510,000)
• Keywords: 並列処理 / 分散処理 / 大規模データ処理 / ハイパフォーマンスコンピューティング / GPGPU / MapReduce / アクセラレータ

Research Abstract

We developed a MapReduce framework for GPU-based heterogeneous clusters as an instance of a future large-scale new platform with heterogeneous many core processors for big data applications. We confirmed performance scalability and overhead of our MapReduce framework by running MapReduce-based graph processing applications using 256 nodes 768GPUs of the TSUBAME2.0 supercomputer. Our results lead the foundation of softwaretechnology for next-generation extreme big data processing.

Research Products

• [Presentation] A Scalable Implementation of a MapReduce-based Graph Processing Algorithm for Large-scale Heterogeneous Supercomputers (2013)
  • Author(s): Koichi Shirahata, Hitoshi Sato, Toyotaro Suzumura, and Satoshi Matsuoka
  • Organizer: In Proceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid (CCGrid2013)
  • Place of Presentation: Delft, The Netherlands
  • Year and Date: 2013-05-15
• [Presentation] A Scalable Implementation of a MapReduce-based Graph Algorithm for Large-scale Heterogeneous Supercomputers (2013)
  • Author(s): Koichi Shirahata, Hitoshi Sato, Toyotaro Suzumura, Satoshi Matsuoka
  • Organizer: GPU TechnologyConference 2013 (GTC2013)
  • Place of Presentation: San Jose, USA
  • Year and Date: 2013-03-18
• [Presentation] A GPU Implementation of Generalized Graph Processing Algorithm GIM-V (2012)
  • Author(s): Koichi Shirahata, Hitoshi Sato, Toyotaro Suzumura, and Satoshi Matsuoka
  • Organizer: In Proceedings of the 3rd International Workshop on Parallel Algorithm and Parallel Software (IWPAPS 2012)
  • Place of Presentation: Beijing, China
  • Year and Date: 2012-09-28
• [Presentation] 汎用グラフ処理モデルGIM-Vの複数GPUによる大規模計算とデータ転送の最適化 (2012)
  • Author(s): 白幡晃一、佐藤仁、鈴村豊太郎、松岡聡
  • Organizer: 情報処理学会研究報告 2012-HPC-133
  • Place of Presentation: 有馬ビューホテルうらら(兵庫県)
• [Presentation] Hadoop on the Tsubame2.0 Supercomputer (2011)
  • Author(s): Hitoshi Sato and Satoshi Matsuoka
  • Organizer: In Proceedings of the 6th International Conference on Ubiquitous Information and Technologies & Applications
  • Place of Presentation: Seoul, Korea
  • Year and Date: 2011-12-15
• [Presentation] GPGPU を用いた高速大規模グラフ処理に向けて (2011)
  • Author(s): 白幡晃一、佐藤仁、鈴村豊太郎、松岡聡
  • Organizer: 情報処理学会研究報告 2011-HPC-130
  • Place of Presentation: かごしま県民交流センター(鹿児島県)
• [Presentation] 細粒度 I/O を考慮したオンデマンド階層型データストアの実現にむけて (2011)
  • Author(s): 佐藤仁、松岡聡
  • Organizer: 情報処理学会研究報告 2011-HPC-130
  • Place of Presentation: かごしま県民交流センター(鹿児島県)
• [Presentation] 大規模並列ファイルシステムに対するワークフローアプリケーションの I/O 性能解析 (2011)
  • Author(s): 斎藤貴文、佐藤仁、松岡聡
  • Organizer: 情報処理学会研究報告 2011-HPC-130
  • Place of Presentation: かごしま県民交流センター(鹿児島県)
• [Presentation] Hadoop on the Tsubame2.0 Supercomputer (2011)
  • Author(s): Hitoshi Sato, Satoshi Matsuoka
  • Organizer: the 6th International Conference on Ubiquitous Information and Technologies & Applications
  • Place of Presentation: Coex(韓国ソウル)
• [Presentation] A GPU Implementation of Generalized Graph Processing Algorithm GIM-V
  • Author(s): Koichi Shirahata
  • Organizer: In Proceedings of the 3rd International Workshop on Parallel Algorithm and Parallel Software (IWPAPS 2012)
  • Place of Presentation: Delft, the Netherlands
• [Presentation] A Scalable Implementation of a MapReduce-based Graph Algorithm for Large-scale Heterogeneous Supercomputers
  • Author(s): Koichi Shirahata
  • Organizer: GPU Technology Conference 2013 (GTC2013)
  • Place of Presentation: San Jose, USA
• [Presentation] 大規模ヘテロ型スーパーコンピュータ向けデータ並列処理フレームワークの設計と実装
  • Author(s): 佐藤 仁
  • Organizer: 情報処理学会研究報告 2012-HPC-138
  • Place of Presentation: 芦原温泉, 福井県
• [Presentation] A GPU Implementation of Generalized Graph Processing Algorithm GIM-V
  • Author(s): Koichi Shirahata
  • Organizer: The 3rd International Workshop on Parallel Algorithm and Parallel Software (IWPAPS 2012)
  • Place of Presentation: Beijing, China
• [Remarks] HadoopGPU拡張公開ページ
  • URL: https://github.com/koichi626/hadoop-gpu.git
• [Remarks] Hadoop GPU拡張 公開ページ
  • URL: https://github.com/koichi626/hadoop-gpu.git