Development of Data Management Framework Integrating Stream Processing and Analytical Data Processing

2014 Fiscal Year Final Research Report

• Project/Area Number: 24700111
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Media informatics/Database
• Research Institution: National Institute of Advanced Industrial Science and Technology
• Principal Investigator: YUI Makoto 独立行政法人産業技術総合研究所, 情報技術研究部門, 主任研究員 (10586712)
• Project Period (FY): 2012-04-01 – 2015-03-31
• Keywords: 機械学習 / ビッグデータ / データベース / 関係データベース / オンライン学習 / 確率的勾配降下法

Outline of Final Research Achievements

We proposed a database-Hadoop hybrid approach to scalable machine learning where batch-learning is performed on the Hadoop platform, while incremental-learning is performed on PostgreSQL.
We conducted a series of exterimental evaluation using a commercial advertisement dataset provided in the KDD Cup 2012, Track 2. The experimental results show that our scheme has a superior training speed compared with state-of-the-art scalable machine learning frameworks, 5 and 7.65 times faster than Vowpal Wabbit and Bismarck, respectively, for a regression task.

Free Research Field

データベース学