• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

On Dataset Diversity for Achiving High Reliability of Machine Learning Software

Research Project

Project/Area Number 18H03224
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 60050:Software-related
Research InstitutionNational Institute of Informatics

Principal Investigator

Nakajima Shin  国立情報学研究所, 大学共同利用機関等の部局等, 名誉教授 (60350211)

Project Period (FY) 2018-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥12,220,000 (Direct Cost: ¥9,400,000、Indirect Cost: ¥2,820,000)
Fiscal Year 2020: ¥3,900,000 (Direct Cost: ¥3,000,000、Indirect Cost: ¥900,000)
Fiscal Year 2019: ¥3,770,000 (Direct Cost: ¥2,900,000、Indirect Cost: ¥870,000)
Fiscal Year 2018: ¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000)
Keywordsソフトウェア工学 / ソフトウェア・テスティング / ニューラル・ネットワーク / ディペンダビリティ
Outline of Final Research Achievements

We proposed a new metamorphic testing method applicable to checking the correctness of machine learning programs constituting the core mechanism of machine learning frameworks. The contributions include a test generation method employing the notion of semantic noises, metamorphic relations referring to active neuron states, and a testing framework to combine the statistical hypothesis testing method and the metamorphic testing.

Academic Significance and Societal Importance of the Research Achievements

深層ニューラルネットワークの技術は高度な信頼性を求められるシステムに応用され、不具合が生じると社会的な影響が大きいことから、品質評価方法の確立が求められている。学樹的には、セマンティックノイズによるデータセット多様性というアイデアから、メタモルフィック・テスティングを深層ニューラルネットワーク訓練学習基盤の検査に応用する方法を示したことである。

Report

(4 results)
  • 2021 Final Research Report ( PDF )
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report
  • Research Products

    (25 results)

All 2021 2020 2019 2018 Other

All Int'l Joint Research (2 results) Journal Article (8 results) (of which Int'l Joint Research: 1 results,  Peer Reviewed: 8 results) Presentation (13 results) (of which Int'l Joint Research: 2 results,  Invited: 2 results) Book (1 results) Remarks (1 results)

  • [Int'l Joint Research] Swinburne University of Technology(オーストラリア)

    • Related Report
      2019 Annual Research Report
  • [Int'l Joint Research] Swinburne University of Technology(オーストラリア)

    • Related Report
      2018 Annual Research Report
  • [Journal Article] AI Extension of SQuaRE Data Quality Model2021

    • Author(s)
      Shin Nakajima, Takako Nakatani
    • Journal Title

      Proc. IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C)

      Volume: - Pages: 306-313

    • DOI

      10.1109/qrs-c55045.2021.00053

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Software Testing with Statistical Partial Oracles - Application to Neural Networks Software -2021

    • Author(s)
      Shin Nakajima
    • Journal Title

      Proc. The 10th International Workshop on SOFL + MSVL for Reliability and Security (SOFL+MSVL2020)

      Volume: - Pages: 175-192

    • DOI

      10.1007/978-3-030-77474-5_12

    • ISBN
      9783030774738, 9783030774745
    • Related Report
      2020 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Distortion and Faults in Machine Learning Software2020

    • Author(s)
      Shin Nakajima
    • Journal Title

      Proc. The 9th International Workshop on SOFL + MSVL for Reliability and Security (SOFL+MSVL 2019)

      Volume: - Pages: 29-41

    • DOI

      10.1007/978-3-030-41418-4_3

    • ISBN
      9783030414177, 9783030414184
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Generating Biased Dataset for Metamorphic Testing of Machine Learning Programs2019

    • Author(s)
      Shin Nakajima, T.Y. Chen
    • Journal Title

      Proc. The 31st IFIP International Conference on Testing Software and Systems (IFIP-ICTSS 2019)

      Volume: - Pages: 56-64

    • DOI

      10.1007/978-3-030-31280-0_4

    • ISBN
      9783030312794, 9783030312800
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Int'l Joint Research
  • [Journal Article] Quality Evaluation Assurance Levels for Deep Neural Networks Software2019

    • Author(s)
      Shin Nakajima
    • Journal Title

      Proc. The 24th International Conference on Technologies and Applications of Artificial Intelligence (TAAI 2019)

      Volume: - Pages: 1-6

    • DOI

      10.1109/taai48200.2019.8959916

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Dataset Diversity for Metamorphic Testing of Machine Learning Software2019

    • Author(s)
      Shin Nakajima
    • Journal Title

      Post-Proc. 8th SOFL+MSVL (LNCS)

      Volume: - Pages: 21-38

    • DOI

      10.1007/978-3-030-13651-2_2

    • ISBN
      9783030136505, 9783030136512
    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Software Testing under Dataset Diversity2018

    • Author(s)
      中島震
    • Journal Title

      Computer Software

      Volume: 35 Issue: 2 Pages: 2_26-2_32

    • DOI

      10.11309/jssst.35.2_26

    • NAID

      130007410585

    • ISSN
      0289-6540
    • Year and Date
      2018-04-24
    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Quality Assurance of Machine Learning Software2018

    • Author(s)
      Shin Nakajima
    • Journal Title

      Proc. IEEE 7th Global Conference on Consumer Electronics

      Volume: - Pages: 601-604

    • DOI

      10.1109/gcce.2018.8574766

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Presentation] ニューラルネットワーク・ソフトウェアの頑健性検査2020

    • Author(s)
      中島震
    • Organizer
      情報処理学会 第206回ソフトウェア工学研究発表会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 統計的な部分オラクルによるテスティング方法2020

    • Author(s)
      中島震
    • Organizer
      日本ソフトウェア科学会第37回大会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 敵対的なセマンティック・ノイズの実行時検知2020

    • Author(s)
      中島震
    • Organizer
      情報処理学会 第205回ソフトウェア工学研究会
    • Related Report
      2020 Annual Research Report
  • [Presentation] 訓練済み機械学習モデル歪みの定量指標2020

    • Author(s)
      中島震
    • Organizer
      電子情報通信学会 ソフトウェア・サイエンス研究会
    • Related Report
      2019 Annual Research Report
  • [Presentation] 機械学習ソフトウェア・テスティングの技術動向2020

    • Author(s)
      中島震
    • Organizer
      電子情報通信学会 システム数理と応用研究会
    • Related Report
      2019 Annual Research Report
  • [Presentation] AIビジネスリスク軽減への価値共創アプローチ2020

    • Author(s)
      中島震
    • Organizer
      日本ソフトウェア科学会ソフトウェア工学の基礎ワークショップ
    • Related Report
      2019 Annual Research Report
  • [Presentation] ファズ・データセットを用いたメタモルフィック・テスティング ~ 機械学習ソフトウェアの検査 ~2019

    • Author(s)
      中島震
    • Organizer
      日本ソフトウェア科学会第36回大会
    • Related Report
      2019 Annual Research Report
  • [Presentation] モデルの歪みと機械学習プログラムの欠陥2019

    • Author(s)
      中島震
    • Organizer
      情報処理学会第202回ソフトウェア工学研究発表会
    • Related Report
      2019 Annual Research Report
  • [Presentation] 機械学習ソフトウェアの品質評価保証レベル2019

    • Author(s)
      中島震
    • Organizer
      電子情報通信学会ソフトウェア・サイエンス研究会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 機械学習ソフトウェアの品質:製品, サービス, プラットフォーム2018

    • Author(s)
      中島震
    • Organizer
      電子情報通信学会知能ソフトウェア工学研究会
    • Related Report
      2018 Annual Research Report
  • [Presentation] Dataset Diversity for Metamorphic Testing of Machine Learning Software2018

    • Author(s)
      Shin Nakajima
    • Organizer
      8th International Workshop SOFL+MSVL
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Quality Assurance of Machine Learning Software2018

    • Author(s)
      Shin Nakajima
    • Organizer
      IEEE 7th Global Conference on Consumer Electronics
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] 機械学習ソフトウェアの品質保証とは2018

    • Author(s)
      中島震
    • Organizer
      機械学習工学研究会キックオフシンポジウム
    • Related Report
      2018 Annual Research Report
    • Invited
  • [Book] ソフトウェア工学から学ぶ機械学習の品質問題2020

    • Author(s)
      中島震
    • Total Pages
      179
    • Publisher
      丸善出版
    • ISBN
      9784621305737
    • Related Report
      2020 Annual Research Report
  • [Remarks] Researchmap

    • URL

      https://researchmap.jp/nkjm/

    • Related Report
      2019 Annual Research Report

URL: 

Published: 2018-04-23   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi