• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Development of Data Management Framework Integrating Stream Processing and Analytical Data Processing

Research Project

Project/Area Number 24700111
Research Category

Grant-in-Aid for Young Scientists (B)

Allocation TypeMulti-year Fund
Research Field Media informatics/Database
Research InstitutionNational Institute of Advanced Industrial Science and Technology

Principal Investigator

YUI Makoto  独立行政法人産業技術総合研究所, 情報技術研究部門, 主任研究員 (10586712)

Project Period (FY) 2012-04-01 – 2015-03-31
Project Status Completed (Fiscal Year 2014)
Budget Amount *help
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2014: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000)
Fiscal Year 2013: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000)
Fiscal Year 2012: ¥1,820,000 (Direct Cost: ¥1,400,000、Indirect Cost: ¥420,000)
Keywords機械学習 / ビッグデータ / データベース / 関係データベース / オンライン学習 / 確率的勾配降下法 / MapReduce / 並列処理
Outline of Final Research Achievements

We proposed a database-Hadoop hybrid approach to scalable machine learning where batch-learning is performed on the Hadoop platform, while incremental-learning is performed on PostgreSQL.
We conducted a series of exterimental evaluation using a commercial advertisement dataset provided in the KDD Cup 2012, Track 2. The experimental results show that our scheme has a superior training speed compared with state-of-the-art scalable machine learning frameworks, 5 and 7.65 times faster than Vowpal Wabbit and Bismarck, respectively, for a regression task.

Report

(4 results)
  • 2014 Annual Research Report   Final Research Report ( PDF )
  • 2013 Research-status Report
  • 2012 Research-status Report
  • Research Products

    (11 results)

All 2015 2014 2013 2012 Other

All Journal Article (2 results) (of which Peer Reviewed: 2 results,  Acknowledgement Compliant: 1 results) Presentation (8 results) (of which Invited: 2 results) Remarks (1 results)

  • [Journal Article] Apache Hiveを用いたスケーラブルな機械学習機構の構築2015

    • Author(s)
      油井誠, 小島功
    • Journal Title

      情報処理学会論文誌: データベース

      Volume: 8 Pages: 73-87

    • NAID

      110009886573

    • Related Report
      2014 Annual Research Report
    • Peer Reviewed / Acknowledgement Compliant
  • [Journal Article] A Database-Hadoop Hybrid Approach to Scalable Machine Learning2013

    • Author(s)
      Makoto Yui, Isao Kojima
    • Journal Title

      Proc. IEEE 2nd International Congress on Big Data, July 2013.

      Volume: - Pages: 1-8

    • DOI

      10.1109/bigdata.congress.2013.10

    • Related Report
      2013 Research-status Report
    • Peer Reviewed
  • [Presentation] Hivemall: Apache Hiveを用いたスケーラブルな機械学習ライブラリ2014

    • Author(s)
      油井誠
    • Organizer
      第26回コンピュータシステム・シンポジウム(ComSys2014)
    • Place of Presentation
      芝浦工業大学 豊洲キャンパス(東京都)
    • Year and Date
      2014-11-19 – 2014-11-20
    • Related Report
      2014 Annual Research Report
    • Invited
  • [Presentation] Hivemall: Apache Hiveを用いたスケーラブルな機械学習基盤2014

    • Author(s)
      油井誠
    • Organizer
      第20回先端的データベースとWeb技術動向講演会 (ACM SIGMOD 日本支部第57回支部大会)
    • Place of Presentation
      リコーITソリューションズ株式会社 本社事業所42F大会議室(東京都)
    • Year and Date
      2014-10-04
    • Related Report
      2014 Annual Research Report
    • Invited
  • [Presentation] Hivemall: Scalable Machine Learning Library for Apache Hive2014

    • Author(s)
      Makoto Yui
    • Organizer
      Hadoop Summit 2014
    • Place of Presentation
      San Jose Convention Center(San Jose, CA, USA)
    • Year and Date
      2014-06-09 – 2014-06-11
    • Related Report
      2014 Annual Research Report
  • [Presentation] Hivemall: Scalable Machine Learning Library for Apache Hive2014

    • Author(s)
      Makoto Yui
    • Organizer
      Hadoop summit 2013
    • Place of Presentation
      San Jose, CA, USA
    • Related Report
      2013 Research-status Report
  • [Presentation] A Database-Hadoop Hybrid Approach to Scalable Machine Learning2013

    • Author(s)
      Makoto Yui, Isao Kojima
    • Organizer
      IEEE 2nd International Congress on Big Data
    • Place of Presentation
      Santa Clara, CA, USA
    • Related Report
      2013 Research-status Report
  • [Presentation] Hivemall: Hive scalable machine learning library2013

    • Author(s)
      Makoto Yui, Isao Kojima
    • Organizer
      NIPS 2013 Workshop on Machine Learning Open Source Software: Towards Open Workflows
    • Place of Presentation
      Lake Tahoe, Nevada, USA
    • Related Report
      2013 Research-status Report
  • [Presentation] A Hybrid Approach to Linked Data Query Processing with Time Constraints2013

    • Author(s)
      Steven Lynden , Isao Kojima , Akiyoshi Matono , Akihito Nakamura , Makoto Yui
    • Organizer
      The 6th Workshop on Linked Data on the Web (LDOW2013)
    • Place of Presentation
      Rio de Janeiro, Brazil
    • Related Report
      2012 Research-status Report
  • [Presentation] MapReduceによる確率的勾配降下法を用いた広告クリック率予測の実践2012

    • Author(s)
      後藤 康路、油井 誠、横山 昌平、小島 功、石川 博
    • Organizer
      第155回データベースシステム研究発表会
    • Place of Presentation
      東京都・秋葉原
    • Related Report
      2012 Research-status Report
  • [Remarks] Hivemall: Hive scalable machine learning library

    • URL

      https://github.com/myui/hivemall

    • Related Report
      2013 Research-status Report

URL: 

Published: 2013-05-31   Modified: 2019-07-29  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi