Differential Testing Techniques Towards Large-scale Deployment of Deep Learning Systems

2020 Fiscal Year Annual Research Report

• Project/Area Number: 19K24348
• Research Institution: Kyushu University
• Principal Investigator: 馬 雷 九州大学, システム情報科学研究院, 学術研究員 (70842061)
• Project Period (FY): 2019-08-30 – 2021-03-31
• Keywords: 深層学習システム / ソフトウェア品質保障 / システム開発と展開 / 機械学習工学 / ソフトウェアテスト

Outline of Annual Research Achievements

深層学習(DL)は、多くの最先端のアプリケーション分野で競争力のある知能と大きな成功を収めてきた。GPU搭載サーバー/クラウドから小型デバイス(携帯電話など)へのDLシステムの導入と適用の需要が急増している。しかし、DLシステムを小型デバイスに効果的に移行して展開することは非常に困難であり、現在、DLシステムの展開に対して品質保証の手法がまだ欠けている。
本研究では、深層学習(DL)システム展開の品質保証のための自動差動テストフレームワーク (Differential Testing Framework)を構築することを目的としている。この研究は当初研究計画に従って、順調に目標に完成した。
具体的には、(i) 多様的なDLシステム展開中の品質を評価するための差動テスト基準の設計した；DLシステム展開中に導入されたソフトウェアの欠陥を検出するための自動的な差動テストフレームワークの開発した；(ii)実用的な小型モバイル機器や組込みシステムにお ける提案手法の系統的評価となっている。本研究の進展により、(iii)DLシステム展開中の品質保証環境が整い、より信頼性の高いDLシステム展開プ ロセスを支援することが検証されました。本件研究結果はこれからも大規模DLシステム開発と展開の段階、さらに実際の応用に役に立つことが期待できる。

Research Products

• [Int'l Joint Research] University of Calgary(カナダ)
  • Country Name: CANADA
  • Counterpart Institution: University of Calgary
• [Int'l Joint Research] Concordia university(カナダ)
  • Country Name: CANADA
  • Counterpart Institution: Concordia university
• [Int'l Joint Research] Nanyang Technological University(シンガポール)
  • Country Name: SINGAPORE
  • Counterpart Institution: Nanyang Technological University
• [Int'l Joint Research] Peking University(中国)
  • Country Name: CHINA
  • Counterpart Institution: Peking University
• [Presentation] Effective Hybrid System Falsification Using Monte Carlo Tree Search Guided by QB-Robustness (2021)
  • Author(s): Zhenya Zhang, Deyun Lyu, Paolo Arcaini, Lei Ma, Ichiro Hasuo and Jianjun Zhao
  • Organizer: The 33rd International Conference on Computer-Aided Verification, 2021 (CAV 2021, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding (2021)
  • Author(s): Maryam V. Pour, Li Zhuo, Lei Ma and Hadi Hemmati
  • Organizer: IEEE International Conference on Software Testing, Verification and Validation (ICST 2021, CORE Rank A)
  • Int'l Joint Research
• [Presentation] Decision-Guided Weighted Automata Extraction from Recurrent Neural Networks (2021)
  • Author(s): Xiyue Zhang, Xiaoning Du, Xiaofei Xie, Lei Ma, Yang Liu, Meng Sun
  • Organizer: Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI 2021, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] EfficientDeRain: Learning Pixel-wise Dilation Filtering for High-Efficiency Single-Image (2021)
  • Author(s): Qing Guo, Jingyang Sun, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Wei Feng, Yang Liu, and Jianjun
  • Organizer: Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI 2021, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] Watch out! Motion is Blurring the Vision of Your Deep Neural Networks (2020)
  • Author(s): Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Jian Wang, Bing Yu, Wei Feng, Yang Liu
  • Organizer: Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo (2020)
  • Author(s): Zi Peng, Jinqiu Yang, Tse-Hsun Chen, Lei Ma
  • Organizer: The 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] MARBLE: Model-Based Robustness Analysis of Stateful Deep Learning Systems (2020)
  • Author(s): Xiaoning Du, Yi Li, Xiaofei Xie, Lei Ma, Yang Liu, Jianjun Zhao
  • Organizer: The 35th IEEE/ACM International Conference on Automated Software Engineering. (ASE 2020, CORE Rank A)
  • Int'l Joint Research
• [Presentation] Cats Are Not Fish: Deep Learning Testing Calls for Out-Of-Distribution Awareness (2020)
  • Author(s): David Berend, Xiaofei Xie, Lei Ma, Lingjun Zhou, Yang Liu, Chi Xu, Jianjun Zhao
  • Organizer: The 35th IEEE/ACM International Conference on Automated Software Engineering. (ASE 2020, CORE Rank A)
  • Int'l Joint Research
• [Presentation] Few-Shot Guided Mix for DNN Repairing (2020)
  • Author(s): Xuhong Ren, Bing Yu, Hua Qi, Felix Juefei-Xu, Zhuo Li, Wanli Xue, Lei Ma and Jianjun Zhao
  • Organizer: The 36th IEEE International Conference on Software Maintenance and Evolution (CORE Rank A)
  • Int'l Joint Research
• [Presentation] DeepSonar: Towards Effective and Robust Detection of AI-Synthesized Fake Voices (2020)
  • Author(s): Run Wang,Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, Yang Liu
  • Organizer: Proceedings of the 28th ACM International Conference on Multimedia (ACM MM, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] Amora: Black-box Adversarial Morphing Attack (2020)
  • Author(s): Run Wang,Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, Yang Liu
  • Organizer: Proceedings of the 28th ACM International Conference on Multimedia (ACM MM, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] Towards characterizing adversarial defects of deep learning software from the lens of uncertainty (2020)
  • Author(s): Xiyue Zhang, Xiaofei Xie, Lei Ma, Xiaoning Du, Qiang Hu, Yang Liu, Jianjun Zhao, Meng Sun
  • Organizer: The 42nd International Conference on Software Engineering, 12 pages, 23-29 May 2020, Seoul, South Korea (ICSE’20, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] Generating Adversarial Examples for Holding Robustness of Source Code Processing Models (2020)
  • Author(s): Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma, Yang Liu, Zhi Jin
  • Organizer: The 34th AAAI Conference on Artificial Intelligence, New York, USA, Feb 7-12, 2020. (AAAI'20, CORE Rank A*)
  • Int'l Joint Research
• [Presentation] Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning (2020)
  • Author(s): Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, Yang Liu
  • Organizer: The 34th AAAI Conference on Artificial Intelligence, New York, USA, Feb 7-12, 2020. (AAAI'20, CORE Rank A*)
  • Int'l Joint Research