• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Developing an algebraic model and program development platform for large-scale graph processing

Research Project

Project/Area Number 26280020
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypePartial Multi-year Fund
Section一般
Research Field Software
Research InstitutionThe University of Electro-Communications

Principal Investigator

Iwasaki Hideya  電気通信大学, 大学院情報理工学研究科, 教授 (90203372)

Co-Investigator(Kenkyū-buntansha) 胡 振江  国立情報学研究所, アーキテクチャ科学研究系, 教授 (50292769)
松崎 公紀  高知工科大学, 情報学群, 教授 (30401243)
江本 健斗  九州工業大学, 大学院情報工学研究院, 准教授 (00587470)
Project Period (FY) 2014-04-01 – 2019-03-31
Project Status Completed (Fiscal Year 2018)
Budget Amount *help
¥17,030,000 (Direct Cost: ¥13,100,000、Indirect Cost: ¥3,930,000)
Fiscal Year 2017: ¥2,600,000 (Direct Cost: ¥2,000,000、Indirect Cost: ¥600,000)
Fiscal Year 2016: ¥5,590,000 (Direct Cost: ¥4,300,000、Indirect Cost: ¥1,290,000)
Fiscal Year 2015: ¥5,200,000 (Direct Cost: ¥4,000,000、Indirect Cost: ¥1,200,000)
Fiscal Year 2014: ¥3,640,000 (Direct Cost: ¥2,800,000、Indirect Cost: ¥840,000)
Keywordsプログラミング方法論 / グラフ並列処理 / 領域特化言語 / 代数データ型 / 関数型言語 / 大規模並列処理 / 代数的データ型
Outline of Final Research Achievements

In this research, we developed a system named Fregel as an open source software that encourages the programmer to develop efficient parallel applications for large-scale graph processing. Technical contributions of this research can be summarized as follows. First, we abstracted and formalized the synchronous vertex-centric computation as a higher-order function. Second, on the basis of the functional model, we proposed Fregel, a functional domain specific language for declarative-style programming on large graphs. Third, we developed a Fregel compiler that transforms a Fregel program into programs for existing vertex-centric frameworks such as Giraph and Pregel+. This compiler applies optimization methods for a target Fregel program by removing unnecessary communications between vertices on the basis of algebraic properties of the programs. Fourth, we confirmed that Fregel programs compiled by the Fregel compiler can be executed with reasonable performance through some experiments.

Academic Significance and Societal Importance of the Research Achievements

本研究は,グラフ中に潜む代数的データ型に基づく構造に注目したモデル化を行っている点に,従来の研究にはほとんどみられない学術的な特徴がある.本研究の成果により,従来はプログラマが手で記述していたグラフデータの並列処理に関して,処理の自然な記述を可能とする領域特化言語と,気軽な利用を可能とする並列実行系が一体となり,プログラマの負担が大きく軽減される.さらにプログラムの代数的な性質に基づく最適化処理をコンパイラが内部で自動的に行うため,プログラマは効率のための細部に囚われる必要はなくなる.Webに代表される巨大なグラフデータに対する処理への需要が大きくなっている今日,その意義は大きい.

Report

(6 results)
  • 2018 Annual Research Report   Final Research Report ( PDF )
  • 2017 Annual Research Report
  • 2016 Annual Research Report
  • 2015 Annual Research Report
  • 2014 Annual Research Report
  • Research Products

    (44 results)

All 2019 2018 2017 2016 2015 2014 Other

All Journal Article (8 results) (of which Int'l Joint Research: 3 results,  Peer Reviewed: 8 results,  Open Access: 6 results,  Acknowledgement Compliant: 1 results) Presentation (33 results) (of which Int'l Joint Research: 12 results) Remarks (3 results)

  • [Journal Article] Eliminating unnecesarry communications in the vertex-centric graph processing by the Fregel compiler.2019

    • Author(s)
      加藤直斗, 岩崎英哉
    • Journal Title

      Computer Software

      Volume: 36 Issue: 2 Pages: 2_28-2_46

    • DOI

      10.11309/jssst.36.2_28

    • NAID

      130007666997

    • ISSN
      0289-6540
    • Year and Date
      2019-04-24
    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] A DSL for graph parallel programming with vertex subsets2019

    • Author(s)
      Kento Emoto, Fumihisa Sadahira
    • Journal Title

      The Journal of Supercomputing

      Volume: 75 Issue: 7 Pages: 1-18

    • DOI

      10.1007/s11227-019-02821-w

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] A Generator of Hadoop MapReduce Programs that Manipulate One-dimensional Arrays2017

    • Author(s)
      Reina Miyazaki, Kiminori Matsuzaki, Shigeyuki Sato
    • Journal Title

      Journal of Information Processing

      Volume: 印刷中

    • NAID

      130005990949

    • Related Report
      2016 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Towards Systematic Parallelization of Graph Transformations over Pregel2016

    • Author(s)
      Le Dec Tung, Zhenjiang Hu
    • Journal Title

      International Journal of Parallel Programming (IJPP)

      Volume: 印刷中 Issue: 2 Pages: 320-339

    • DOI

      10.1007/s10766-016-0418-5

    • Related Report
      2017 Annual Research Report 2016 Annual Research Report 2015 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] Let High-level Graph Queries Be Parallel Efficient: An Approach Over Structural Recursion On Pregel2016

    • Author(s)
      Chong Li, Le-Duc Tung, Xiaodong Meng, Zhenjiang Hu
    • Journal Title

      Journal of Information Processing

      Volume: 24 Issue: 6 Pages: 928-936

    • DOI

      10.2197/ipsjjip.24.928

    • NAID

      130005282160

    • ISSN
      1882-6652
    • Related Report
      2016 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] A Partial-tree-based Approach for XPath Query on Large XML Trees2016

    • Author(s)
      Wei Hao, Kiminori Matsuzaki
    • Journal Title

      Journal of Information Processing

      Volume: 24 Pages: 425-438

    • NAID

      130005138316

    • Related Report
      2015 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] Parallel Tree Accumulations on MapReduce2016

    • Author(s)
      Kiminori Matsuzaki, Reina Miyazaki
    • Journal Title

      International Journal of Parallel Programming

      Volume: 44 Issue: 3 Pages: 466-485

    • DOI

      10.1007/s10766-015-0355-8

    • NAID

      110009850861

    • Related Report
      2015 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] An Automatic Fusion Mechanism for Variable-Length List Skeletons in SkeTo2014

    • Author(s)
      Kento Emoto, Kiminori Matsuzaki
    • Journal Title

      International Journal of Parallel Programming

      Volume: 42(4) Issue: 4 Pages: 546-563

    • DOI

      10.1007/s10766-013-0263-8

    • NAID

      120005728994

    • Related Report
      2014 Annual Research Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Presentation] グラフ並列分散処理のための計算モデルを選択可能な領域特化言語2019

    • Author(s)
      秋葉柊哉, 岩崎英哉
    • Organizer
      情報処理学会 第60回プログラミング・シンポジウム
    • Related Report
      2018 Annual Research Report
  • [Presentation] 頂点主体並列分散処理のメッセージ送信遅延による通信数削減2019

    • Author(s)
      加藤 直斗, 岩崎英哉
    • Organizer
      日本ソフトウェア科学会 第21回プログラミングおよびプログラミング言語ワークショップ(PPL2019) (ポスター)
    • Related Report
      2018 Annual Research Report
  • [Presentation] FregelからGraphXへのコンパイルにおけるSMTソルバを用いた不要な通信の削減2019

    • Author(s)
      小西 篤志, 江本 健斗
    • Organizer
      日本ソフトウェア科学会 第21回プログラミングおよびプログラミング言語ワークショップ(PPL2019) (ポスター)
    • Related Report
      2018 Annual Research Report
  • [Presentation] Composing Optimization Techniques for Vertex-Centric Graph Processing via Communication Channels2019

    • Author(s)
      Yongzhe Zhang, Zhenjiang Hu
    • Organizer
      33rd IEEE International Parallel and Distributed Processing Symposium (IPDPS 2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Incrementalization of Vertex-Centric Programs2019

    • Author(s)
      Timothy A. K. Zakian, Ludovic A. R. Capelli, Zhenjiang Hu
    • Organizer
      33rd IEEE International Parallel and Distributed Processing Symposium (IPDPS 2019)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Optimizing Declarative Parallel distributed Graph Processing by using Constraint Solvers2018

    • Author(s)
      Akimasa Morihata, Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, Hideya Iwasaki
    • Organizer
      14th International Symposium on Functional and Logic Programming (FLOPS 2018)
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 頂点主体グラフ並列処理のためのGiraph用クラスライブラリ2018

    • Author(s)
      駒村 春野, 岩崎 英哉
    • Organizer
      第59回プログラミング・シンポジウム
    • Related Report
      2017 Annual Research Report
  • [Presentation] Fregelコンパイラにおける不要な値送受信の削減2018

    • Author(s)
      加藤 直斗, 岩崎 英哉
    • Organizer
      第20回プログラミングおよびプログラミング言語ワークショップ (PPL 2018) (カテゴリ 1)
    • Related Report
      2017 Annual Research Report
  • [Presentation] 関数型言語の初学者のためのHaskellビジュアルプログラミング環境2018

    • Author(s)
      近松 万由子,岩崎 英哉, 中野 圭介
    • Organizer
      第20回プログラミングおよびプログラミング言語ワークショップ (PPL 2018) (カテゴリ 1)
    • Related Report
      2017 Annual Research Report
  • [Presentation] Implementing distributed backpropagation algorithm using Apache Giraph2017

    • Author(s)
      Amogh Rathore, Hideya Iwasaki
    • Organizer
      第19回プログラミングおよびプログラミング言語ワークショップ (PPL 2017) (カテゴリ1)
    • Place of Presentation
      華やぎの章 慶山 (山梨県笛吹市)
    • Year and Date
      2017-03-08
    • Related Report
      2016 Annual Research Report
  • [Presentation] 頂点部分集合変数を備えた大規模グラフ計算用領域特化言語2017

    • Author(s)
      定平 典久, 江本 健斗
    • Organizer
      第19回プログラミングおよびプログラミング言語ワークショップ (PPL 2017) (カテゴリ1)
    • Place of Presentation
      華やぎの章 慶山 (山梨県笛吹市)
    • Year and Date
      2017-03-08
    • Related Report
      2016 Annual Research Report
  • [Presentation] Pregel+プログラムの性能解析2017

    • Author(s)
      寺口 守, 佐藤 重幸, 松崎 公紀
    • Organizer
      第19回プログラミングおよびプログラミング言語ワークショップ (PPL 2017) (ポスター)
    • Place of Presentation
      華やぎの章 慶山 (山梨県笛吹市)
    • Year and Date
      2017-03-08
    • Related Report
      2016 Annual Research Report
  • [Presentation] Spark GraphXへのFregelコンパイラ2017

    • Author(s)
      中島 拓, 江本 健斗
    • Organizer
      第19回プログラミングおよびプログラミング言語ワークショップ (PPL 2017) (ポスター)
    • Place of Presentation
      華やぎの章 慶山 (山梨県笛吹市)
    • Year and Date
      2017-03-08
    • Related Report
      2016 Annual Research Report
  • [Presentation] 一次元配列を操作するプログラムに対するHadoop MapReduceプログラム生成系2017

    • Author(s)
      宮崎 玲奈, 松崎 公紀, 佐藤 重幸
    • Organizer
      情報処理学会第112回プログラミング研究会
    • Place of Presentation
      沖縄県男女共同参画センター (沖縄県那覇市)
    • Year and Date
      2017-01-10
    • Related Report
      2016 Annual Research Report
  • [Presentation] Keeping Control Away from Computation: A Computation Control Layer over the Vertex-centric Graph Processing Model2017

    • Author(s)
      Onofre Coll Ruiz, Kiminori Matsuzaki
    • Organizer
      10th International Symposium on High-Level Parallel Programming and Applications (HLPP 2017)
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 頂点主体並列グラフ処理の制約解消器による効率化2017

    • Author(s)
      森畑 明昌, 江本 健斗, 松崎 公紀, 胡 振江, 岩崎 英哉
    • Organizer
      日本ソフトウェア科学会第34回大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] 複数の頂点主体グラフ計算フレームワーク向けの中間表現とコード生成器2017

    • Author(s)
      松崎 公紀, 岩崎 英哉, 江本 健斗, 胡 振江, 森畑 明昌
    • Organizer
      日本ソフトウェア科学会第34回大会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Palgol: A High-Level DSL for Vertex-Centric Graph Processing with Remote Data Access2017

    • Author(s)
      Yongzhe Zhang, Hsiang-Shang Ko, Zhenjiang Hu
    • Organizer
      15th Asian Symposium on Programming Languages and Systems
    • Related Report
      2017 Annual Research Report
    • Int'l Joint Research
  • [Presentation] s6raph: Vertex-centric Graph Processing Framework with Functional Interface2016

    • Author(s)
      Onofre Coll Ruiz, Kiminori Matsuzaki, and Shigeyuki Sato
    • Organizer
      Proceedings of the 5th ACM SIGPLAN Workshop on Functional High-Performance Computing (FHPC 2016)
    • Place of Presentation
      Nara, Japan
    • Year and Date
      2016-09-22
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Think like a vertex, behave like a function! a functional DSL for vertex-centric big graph processing2016

    • Author(s)
      Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, Akimasa Morihata, Hideya Iwasaki
    • Organizer
      Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming (ICFP 2016)
    • Place of Presentation
      Nara, Japan
    • Year and Date
      2016-09-19
    • Related Report
      2016 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 大規模グラフ並列処理のための関数型領域特化言語 Fregel とその評価2016

    • Author(s)
      江本 健斗, 松崎 公紀, 胡 振江, 森畑 明昌, 岩崎 英哉
    • Organizer
      日本ソフトウェア科学会第33回大会講演論文集
    • Place of Presentation
      東北大学 (宮城県仙台市)
    • Year and Date
      2016-09-07
    • Related Report
      2016 Annual Research Report
  • [Presentation] Describing Pregel Algorithms with Non-adjacent Data Communication2016

    • Author(s)
      Yongzhe Zhang, Hsiang-Shang Ko, Zhenjiang Hu
    • Organizer
      日本ソフトウェア科学会第33回大会講演論文集
    • Place of Presentation
      東北大学 (宮城県仙台市)
    • Year and Date
      2016-09-07
    • Related Report
      2016 Annual Research Report
  • [Presentation] データ依存を持つ繰り返し計算アルゴリズムからのHadoop MapReduceプログラムの生成2016

    • Author(s)
      宮崎 玲奈, 松崎 公紀
    • Organizer
      日本ソフトウェア科学会第33回大会講演論文集
    • Place of Presentation
      東北大学 (宮城県仙台市)
    • Year and Date
      2016-09-07
    • Related Report
      2016 Annual Research Report
  • [Presentation] 大規模グラフ並列処理のための関数型領域特化言語2016

    • Author(s)
      江本 健斗,松崎 公紀,胡 振江,森畑 明昌,岩崎 英哉
    • Organizer
      第18回プログラミングおよびプログラミング言語ワークショップ (PPL 2016)
    • Place of Presentation
      岡山県玉野市
    • Year and Date
      2016-03-07
    • Related Report
      2015 Annual Research Report
  • [Presentation] A Functional DSL for Large Scale Graph Processing (Poster)2016

    • Author(s)
      Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, Akimasa Morihata, Hideya Iwasaki
    • Organizer
      13th International Symposium on Functional and Logic Programming (FLOPS 2016)
    • Place of Presentation
      Kochi, Japan
    • Year and Date
      2016-03-04
    • Related Report
      2015 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A functional approach to PregelA functional approach to Pregel (Poster)2015

    • Author(s)
      Onofre Coll Ruiz, Kiminori Matsuzaki
    • Organizer
      13th Asian Symposium on Programming Languages and Systems (APLAS 2015)
    • Place of Presentation
      Pohong, Korea
    • Year and Date
      2015-11-30
    • Related Report
      2015 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Processing UnQL Graph QueriProcessing UnQL Graph Queries with Pregel2015

    • Author(s)
      Chong Li, Le-Duc Tung, Xiaodong Meng, Zhenjiang Hu
    • Organizer
      日本ソフトウェア科学会第32会大会
    • Place of Presentation
      東京
    • Year and Date
      2015-09-08
    • Related Report
      2015 Annual Research Report
  • [Presentation] Towards Systematic Parallelization of Graph Transformations over Pregel2015

    • Author(s)
      Le Dec Tung, Zhenjiang Hu
    • Organizer
      8th International Symposium on High-level Parallel Programming and Applications (HLPP 2015)
    • Place of Presentation
      Pisa, Italy
    • Year and Date
      2015-07-02
    • Related Report
      2015 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Functional Models of Hadoop MapReduce with Application to Scan2015

    • Author(s)
      Kiminori Matsuzaki
    • Organizer
      8th International Symposium on High-level Parallel Programming and Applications (HLPP 2015)
    • Place of Presentation
      Pisa, Italy
    • Year and Date
      2015-07-02
    • Related Report
      2015 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pregel meets UnCAL: a Systematic Framework for Transforming Big Graphs2015

    • Author(s)
      Le-Duc Tung
    • Organizer
      Ph.D. Synposium, 31st IEEE International Conference on data Engineering (ICDE 2015)
    • Place of Presentation
      Seoul, Korea
    • Year and Date
      2015-04-13 – 2015-04-17
    • Related Report
      2014 Annual Research Report
  • [Presentation] Pregel meets UnCAL: a Systematic Framework for Transforming Big Graphs2015

    • Author(s)
      Le Dec Tung
    • Organizer
      31st International Conference on Data Engineering Workshop (ICDEW 2015)
    • Place of Presentation
      Seoul, Korea
    • Year and Date
      2015-04-13
    • Related Report
      2015 Annual Research Report
    • Int'l Joint Research
  • [Presentation] A Verified Generate-Test-Aggregate Coq Library for Parallel Programs Extraction2014

    • Author(s)
      Kento Emoto, Frederic Loulergue, Julien Tesson
    • Organizer
      Interactive Theorem Proving - 5th International Conference (ITP 2014)
    • Place of Presentation
      Vienna, Austria
    • Year and Date
      2014-07-14 – 2014-07-17
    • Related Report
      2014 Annual Research Report
  • [Presentation] Parallel Tree Accumulations on MapReduce2014

    • Author(s)
      Kiminori Matsuzaki, Reina Miyazaki
    • Organizer
      7th International Symposium on High-level Parallel Programming and Applications (HLPP 2014)
    • Place of Presentation
      Amsterdam, Netherlands
    • Year and Date
      2014-07-03 – 2014-07-04
    • Related Report
      2014 Annual Research Report
  • [Remarks] Fregel Project

    • URL

      https://ipl.cs.uec.ac.jp/~iwasaki/Fregel/

    • Related Report
      2018 Annual Research Report
  • [Remarks] Fregel プロジェクト

    • URL

      https://ipl.cs.uec.ac.jp/~iwasaki/Fregel/index-j.html

    • Related Report
      2018 Annual Research Report
  • [Remarks] Fregel Project

    • URL

      http://ipl.cs.uec.ac.jp/~iwasaki/Fregel/

    • Related Report
      2017 Annual Research Report 2016 Annual Research Report

URL: 

Published: 2014-04-04   Modified: 2020-03-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi