Efficient Data Staging into The Big Data Analytics Platform

2017 Fiscal Year Final Research Report

• Project/Area Number: 16K21675
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Multimedia database; High performance computing
• Research Institution: National Institute of Advanced Industrial Science and Technology
• Principal Investigator: Tanimura Yusuke 国立研究開発法人産業技術総合研究所, 情報・人間工学領域, 主任研究員 (80415710)
• Project Period (FY): 2016-04-01 – 2018-03-31
• Keywords: データストレージ / ビッグデータ解析 / データステージング / 資源管理 / クラウド

Outline of Final Research Achievements

Data staging between the big data analytics platform (the main computing system) and the backend storage, pre- and post- processing in the backend to achieve efficient data staging, and appropriate scheduling to avoid performance interference by concurrent data staging was studied. The prototype system by using Spark and Alluxio was designed and implemented, and then I/O (staging) performance including storage-side processing was evaluated. The basic results would enable multi-tenant operation of the big data analytics platform and more effective use of the backend storage.

Free Research Field

ストレージシステム