• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Preliminary study of an inexpensive implementation methodology for an all-sky oriented astronomical data archive system powered by Hadoop for huge observational multi-wavelength data set

Research Project

Project/Area Number 15K17501
Research Category

Grant-in-Aid for Young Scientists (B)

Allocation TypeMulti-year Fund
Research Field Computational science
Research InstitutionFukuoka University

Principal Investigator

Eguchi Satoshi  福岡大学, 理学部, 助教 (40647202)

Project Period (FY) 2015-04-01 – 2018-03-31
Project Status Completed (Fiscal Year 2017)
Budget Amount *help
¥1,950,000 (Direct Cost: ¥1,500,000、Indirect Cost: ¥450,000)
Fiscal Year 2016: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Fiscal Year 2015: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Keywordsバーチャル天文台 / ビッグデータ / 分散コンピューティング / クラウドコンピューティング / Hadoop / Hive / データベース / ALMA望遠鏡 / アルゴリズム
Outline of Final Research Achievements

The size of astronomical observational data has been exponentially inflating due to the hugeness and complexity of modern telescopes and their instruments. Public cloud computing would be an attractive solution for the data explosion to astronomers without sufficient financial support thanks to its flexibility and inexpensiveness. To this end, I investigated the feasibility through the implementation of a simple astronomical database running on Hadoop, a software framework for distributed computing, and Hive, Hadoop-based database software with an SQL-like query language. I found that we should (1) choose clouds enabling the users to get arbitrary computing resources depending on the complexity of problems at the time, instead of virtual private servers providing limited and fixed resources at a lower annual cost, (2) adopt the Tez engine and the ORC file format for the Hive database, (3) partition the Hive table based on the level 6 HEALPix labeling.

Report

(4 results)
  • 2017 Annual Research Report   Final Research Report ( PDF )
  • 2016 Research-status Report
  • 2015 Research-status Report
  • Research Products

    (7 results)

All 2017 2016 2015

All Journal Article (3 results) (of which Peer Reviewed: 1 results) Presentation (4 results) (of which Int'l Joint Research: 2 results)

  • [Journal Article] Compton thick absorber in type 1 quasar 3C 345 revealed by Suzaku and Swift/BAT2017

    • Author(s)
      Eguchi Satoshi
    • Journal Title

      Monthly Notices of the Royal Astronomical Society

      Volume: 468 Issue: 4 Pages: 4529-4538

    • DOI

      10.1093/mnras/stx754

    • Related Report
      2017 Annual Research Report 2016 Research-status Report
    • Peer Reviewed
  • [Journal Article] Pre-feasibility Study of Astronomical Data Archive Systems Powered by Public Cloud Computing and Hadoop Hive2017

    • Author(s)
      Satoshi Eguchi
    • Journal Title

      Astronomical Data Analysis Software and Systems XXVI

      Volume: 印刷中

    • Related Report
      2016 Research-status Report
  • [Journal Article] Blade Runner -What kind objects are there in the JVO ALMA Archive?-2016

    • Author(s)
      Satoshi Eguchi, Yuji Shirasaki, Christopher Zapart, Masatoshi Ohishi, Yoshihiko Mizumoto, Wataru Kawasaki, Tsuyoshi Kobayashi, George Kosugi
    • Journal Title

      Astronomical Data Analysis Software and Systems XXV

      Volume: 印刷中

    • Related Report
      2015 Research-status Report
  • [Presentation] 「すざく」およびSwift/BATで明らかになった1型クエーサー3C 345の光学的に非常に分厚い吸収体2017

    • Author(s)
      江口 智士
    • Organizer
      日本天文学会
    • Related Report
      2017 Annual Research Report
  • [Presentation] Pre-feasibility Study of Astronomical Data Archive Systems Powered by Public Cloud Computing and Hadoop Hive2016

    • Author(s)
      Satoshi Eguchi
    • Organizer
      Astronomical Data Analysis Software and Systems XXVI
    • Place of Presentation
      Stazione Marittima Conference Centre, Trieste, Italy
    • Year and Date
      2016-10-17
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] Blade Runner -What kind objects are there in the JVO ALMA Archive?-2015

    • Author(s)
      Satoshi Eguchi
    • Organizer
      Astronomical Data Analysis Software and Systems XXV
    • Place of Presentation
      Rydges World Square Sydney, Australia
    • Year and Date
      2015-10-25
    • Related Report
      2015 Research-status Report
    • Int'l Joint Research
  • [Presentation] JVO ALMAアーカイブの天体同定(I)2015

    • Author(s)
      江口 智士
    • Organizer
      日本天文学会2015秋季年会
    • Place of Presentation
      甲南大学
    • Year and Date
      2015-09-09
    • Related Report
      2015 Research-status Report

URL: 

Published: 2015-04-16   Modified: 2019-03-29  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi