Development of evaluation techniques for machine learning systems for software bug prediction

2023 Fiscal Year Final Research Report

• Project/Area Number: 20K11749
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Review Section: Basic Section 60050:Software-related
• Research Institution: Okayama University
• Principal Investigator: Monden Akito 岡山大学, 環境生命自然科学学域, 教授 (80311786)
• Project Period (FY): 2020-04-01 – 2024-03-31
• Keywords: ソフトウェア開発データ / ソフトウェアバグ予測 / ソフトウェアメトリクス / 機械学習

Outline of Final Research Achievements

In the evaluation of machine learning systems, it is important to (1) evaluate the quality of training data and (2) evaluate the performance of system output. For (1), we defined a data inconsistency measure, Similar Case Inconsistency Level (SCIL). Through evaluation experiments, we showed that the less inconsistent the dataset is, the better the prediction performance of the resulting machine learning model tends to be. For (2), we defined the expected values of performance measures for a two-class classification problem based on the neg/pos ratio of the dataset. Application experiments showed that there are cases in which conventional evaluation measures cannot correctly evaluate the prediction performance, indicating the usefulness of the proposed measures.

Free Research Field

ソフトウェア工学

Academic Significance and Societal Importance of the Research Achievements

本研究の成果によって，ソフトウェア開発データを対象とした機械学習システムにおいて，学習データを事前に評価すること，および，性能評価をより適切に行うことが可能となり，ソフトウェア工学分野のさらなる発展に寄与できると期待される．また，提案方法は，機械学習を利用する様々な分野への応用が期待される．