| 研究課題/領域番号 |
16K16038
|
|---|
| 研究機関 | 会津大学 |
|---|
研究代表者 |
李 鵬 (李鵬) 会津大学, コンピュータ理工学部, 准教授 (30735915)
|
|---|
| 研究期間 (年度) |
2016-04-01 – 2018-03-31
|
| キーワード | big data processing / cloud |
|---|
| 研究実績の概要 |
In FY2016, we develop an intelligent software platform for storing and processing big spatial-temporal data. First, we construct a hierarchy indexing structure based on 3-dimensional R-tree and distribute the R-tree and its associated data to multiple nodes. Second, we propose a traffic-aware task placement to minimize job completion time of MapReduce jobs on Spark. We develop an optimization framework by jointly considering both data and task placement in the MapReduce model. Finally, we study the randomness in MapReduce job execution and propose a novel optimization framework to guarantee predictable job completion time.
|
| 現在までの達成度 (区分) |
現在までの達成度 (区分)
1: 当初の計画以上に進展している
理由
In FY2016, we achieve our research goals by addressing many challenges in system performance optimization. In the beginning, we build the system and it works well in a small-scale cluster (less than 10 machines). However, when we deploy the system into a larger cluster, its performance is unsatisfied. Therefore, we make many research efforts on performance optimization by proposing new algorithms for job scheduling, traffic and storage management.
|
| 今後の研究の推進方策 |
In FY2017, we will continue to study the ML-engine as we proposed in the original research plan. First, we adopt the machine learning technology to identify and predict data skew. Specifically, we first classify the space into several clusters according to the known data skew in history. Second, we adopt deep learning technology to extract data access pattern based on activity traces collected from services/application layer. Specifically, we create a multi-layer artificial neural network model. Then, we use a bottom-to-up process to obtain activation probabilities for all hidden units in the network. After that, a top-to-bottom process obtains good initial weights with minimum error. Finally, we integrate all components by defining clean interfaces and optimize message flow among them.
|
