A Parallel Processing Scheme for big data targeting a computer with many CPU cores and a large capacity main memory.

2016 Fiscal Year Final Research Report

• Project/Area Number: 15K12006
• Research Category: Grant-in-Aid for Challenging Exploratory Research
• Allocation Type: Multi-year Fund
• Research Field: Software
• Research Institution: University of Tsukuba
• Principal Investigator: KATO kazuhiko 筑波大学, システム情報系, 教授 (90224493)
• Project Period (FY): 2015-04-01 – 2017-03-31
• Keywords: ビッグデータ / メニーコア / 並列処理 / B木 / B+木

Outline of Final Research Achievements

We have conducted research and development on a high-speed processing method for big data processing targeting a computer with many CPU cores and a large capacity main memory. Many of useful big data processing has mathematical properties to derive parallelism. By using that, we proposed a parallel processing scheme to data partition and processing, and the intermediate results are merged in an almost lock-free manner, which enables full-parallelism. We implemented the scheme using the Scala programming language. We conducted experiments using real machines and confirmed that high-speed big data parallel processing can be performed efficiently with the scheme.

Free Research Field

システムソフトウエア