Development and clinical application of personalized IoT system to control the risk of mental and physical disorders of workers

• Project/Area Number: 20H00569
• Research Category: Grant-in-Aid for Scientific Research (A)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Medium-sized Section 59:Sports sciences, physical education, health sciences, and related fields
• Research Institution: The University of Tokyo
• Principal Investigator: Yamamoto Yoshiharu 東京大学, 大学院教育学研究科(教育学部), 教授 (60251427)
• Co-Investigator(Kenkyū-buntansha): 北島 剛司 藤田医科大学, 医学部, 教授 (40360234) ; 吉内 一浩 東京大学, 医学部附属病院, 准教授 (70313153) ; 中村 亨 大阪大学, 大学院基礎工学研究科, 特任教授(常勤) (80419473)
• Project Period (FY): 2020-04-01 – 2023-03-31
• Project Status: Completed (Fiscal Year 2022)
• Budget Amount: ¥45,370,000 (Direct Cost: ¥34,900,000、Indirect Cost: ¥10,470,000); Fiscal Year 2022: ¥11,440,000 (Direct Cost: ¥8,800,000、Indirect Cost: ¥2,640,000); Fiscal Year 2021: ¥17,030,000 (Direct Cost: ¥13,100,000、Indirect Cost: ¥3,930,000); Fiscal Year 2020: ¥16,900,000 (Direct Cost: ¥13,000,000、Indirect Cost: ¥3,900,000)
• Keywords: Micro-randomized trial / Affective computing / 睡眠安定性 / AIoT / 健康リスクの制御 / IoTデバイス / 心身の不調 / Just-in-Time適応型介入 / Micro-Randomized Trial / 健康のリスク制御

Outline of Research at the Start

勤労者における日常生活下の健康リスク（心身の不調）に着目し、just-in-timeな介入（必要なものを、必要な時に、必要なだけ行う介入）を大規模に実現するためのIoTクラウドシステムを活用した研究を行う。申請者らが開発してきたIoTクラウドシステムおよび実時間介入技術を用いて大規模実証実験を行い、収集したビッグデータを機械学習等の手法で分析することによって個人に最適な介入手法を導き出し、その最適化された介入手法が症状の改善やコントロールといった健康リスクの制御に役立つかを検討する。

Outline of Final Research Achievements

The aim of this research project was to identify the vulnerable state of mental health among Japanese employees and to modify their health-related behaviors in real-world settings. By using digital devices, including a smartphone application and a wearable activity monitor, we measured employees' moods and bio-signals (i.e., physical activity, heart rate, and voice) in real-world settings. Then, we developed an AI model to predict the self-reported moods from the emotionally neutral voice data. As a result, the model could predict employees' moods reasonably. In addition, we conducted micro-randomized trial targeting instability in habitual sleep behaviors, which is known as potential factor contributing to mood disturbances. It was found that habitual sleep duration was significantly stabilized during the trial especially among the employees with unstable habitual sleep behaviors.

Academic Significance and Societal Importance of the Research Achievements

本研究では(1)感情中立的発話データに基づく気分推定技術の開発、および (2)睡眠習慣の安定化を目的としたMicro-randomized trialの実証実験を行った。これらの研究開発は、国際的にも実証例の少ない、あるいは、前人未到の試みであり、不調の検知・制御に関する先駆的な取り組みであったと言える。また、上述の研究成果は全て日常生活下で得られたデータを基盤として得られたものであることから、real-worldでの健康リスクの予兆検知・一次予防の実現可能性を担保するものである。

Research Products

• [Journal Article] The Voice of the Body: Why AI Should Listen to It and an Archive (2023)
  • Author(s): Qian Kun、Hu Bin、Yamamoto Yoshiharu、Schuller Bjoern W.
  • Journal Title: Cyborg and Bionic Systems Volume: 4 Pages: 1-3
  • DOI: 10.34133/cbsystems.0005
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Exploring interpretable representations for heart sound abnormality detection (2023)
  • Author(s): Wang Zhihua、Qian Kun、Liu Houguang、Hu Bin、Schuller Bjoern W.、Yamamoto Yoshiharu
  • Journal Title: Biomedical Signal Processing and Control Volume: 82 Pages: 104569-104569
  • DOI: 10.1016/j.bspc.2023.104569
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] The Effects of Objective Push-Type Sleep Feedback on Habitual Sleep Behavior and Momentary Symptoms in Daily Life: mHealth Intervention Trial Using a Health Care Internet of Things System (2022)
  • Author(s): Takeuchi Hiroki、Suwa Kaori、Kishi Akifumi、Nakamura Toru、Yoshiuchi Kazuhiro、Yamamoto Yoshiharu
  • Journal Title: JMIR mHealth and uHealth Volume: 10 Issue: 10 Pages: e39150-e39150
  • DOI: 10.2196/39150
  • Peer Reviewed / Open Access
• [Journal Article] Altered distribution of resting periods of daily locomotor activity in patients with delayed sleep phase disorder (2022)
  • Author(s): Hirose Marina、Nakamura Toru、Watanabe Akiko、Esaki Yuichi、Koike Shigefumi、Yamamoto Yoshiharu、Iwata Nakao、Kitajima Tsuyoshi
  • Journal Title: Frontiers in Psychiatry Volume: 13 Pages: 1-9
  • DOI: 10.3389/fpsyt.2022.933690
  • Peer Reviewed / Open Access
• [Journal Article] Psychological Field Versus Physiological Field: From Qualitative Analysis to Quantitative Modeling of the Mental Status (2022)
  • Author(s): Hu Bin、Qian Kun、Dong Qunxi、Luo Yuejia、Yamamoto Yoshiharu、Schuller Bjorn W.
  • Journal Title: IEEE Transactions on Computational Social Systems Volume: 9 Issue: 5 Pages: 1275-1281
  • DOI: 10.1109/tcss.2022.3204144
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Learning Multimodal Representations for Drowsiness Detection (2021)
  • Author(s): Qian Kun、Koike Tomoya、Nakamura Toru、Schuller Bjorn、Yamamoto Yoshiharu
  • Journal Title: IEEE Transactions on Intelligent Transportation Systems Volume: in press Issue: 8 Pages: 1-10
  • DOI: 10.1109/tits.2021.3105326
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Seasonal Sleep Variations and Their Association With Meteorological Factors: A Japanese Population Study Using Large-Scale Body Acceleration Data (2021)
  • Author(s): Li Li、Nakamura Toru、Hayano Junichiro、Yamamoto Yoshiharu
  • Journal Title: Frontiers in Digital Health Volume: 3
  • DOI: 10.3389/fdgth.2021.677043
  • Peer Reviewed / Open Access
• [Journal Article] Age and gender differences in objective sleep properties using large-scale body acceleration data in a Japanese population (2021)
  • Author(s): Li Li、Nakamura Toru、Hayano Junichiro、Yamamoto Yoshiharu
  • Journal Title: Scientific Reports Volume: 11 Issue: 1
  • DOI: 10.1038/s41598-021-89341-x
  • Peer Reviewed / Open Access
• [Journal Article] COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis (2021)
  • Author(s): Schuller Bjorn W.、Schuller Dagmar M.、Qian Kun、Liu Juan、Zheng Huaiyuan、Li Xiao
  • Journal Title: Frontiers in Digital Health Volume: 3 Pages: 1-1
  • DOI: 10.3389/fdgth.2021.564906
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Computer Audition for Fighting the SARS-CoV-2 Corona Crisis ? Introducing the Multi-task Speech Corpus for COVID-19 (2021)
  • Author(s): Qian Kun、Schmitt Maximilian、Zheng Huaiyuan、Koike Tomoya、Han Jing、Liu Juan、Ji Wei、Duan Junjun、Song Meishu、Yang Zijiang、Ren Zhao、Liu Shuo、Zhang Zixing、Yamamoto Yoshiharu、Schuller Bjorn W.
  • Journal Title: IEEE Internet of Things Journal Volume: - Issue: 21 Pages: 1-1
  • DOI: 10.1109/jiot.2021.3067605
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Can Machine Learning Assist Locating the Excitation of Snore Sound? A Review (2021)
  • Author(s): Qian Kun、Janott Christoph、Schmitt Maximilian、Zhang Zixing、Heiser Clemens、Hemmert Werner、Yamamoto Yoshiharu、Schuller Bjorn W.
  • Journal Title: IEEE Journal of Biomedical and Health Informatics Volume: 25 Issue: 4 Pages: 1233-1246
  • DOI: 10.1109/jbhi.2020.3012666
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Artificial Intelligence Internet of Things for the Elderly: From Assisted Living to Health-Care Monitoring. (2021)
  • Author(s): Kun Qian, Zixing Zhang, Yoshiharu Yamamoto, and Bjorn W. Schuller.
  • Journal Title: IEEE Signal Processing Magazine Volume: 38
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Can Appliances Understand the Behaviour of Elderly via Machine Learning? A Feasibility Study (2020)
  • Author(s): Qian Kun、Koike Tomoya、Yoshiuchi Kazuhiro、Schuller Bjorn W.、Yamamoto Yoshiharu
  • Journal Title: IEEE Internet of Things Journal Volume: - Issue: 10 Pages: 1-1
  • DOI: 10.1109/jiot.2020.3045009
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Presentation] Learning optimal time-frequency representations for heart sound: A comparative study. (2023)
  • Author(s): Wang, Z., Z. Bao, K. Qian, B. Hu, B. W. Schuller, and Y. Yamamoto.
  • Organizer: Proceedings of the 9th Conference on Sound and Music Technology, Springer Nature Singapore, pp. 93-104, 2023.
  • Int'l Joint Research
• [Presentation] A federated learning paradigm for heart sound classification. (2022)
  • Author(s): Qiu, W., K. Qian, Z. Wang, Y. Chang, Z. Bao, B. Hu, B. W. Schuller, and Y. Yamamoto.
  • Organizer: Proceedings of 44th Annual International Conference of IEEE Engineering in Medicine and Biology Society (EMBC 2022), pp. 1045-1048, 2022.
  • Int'l Joint Research
• [Presentation] Heart sound classification based on residual shrinkage networks. (2022)
  • Author(s): Zhu, L., K. Qian, Z. Wang, B. Hu, Y. Yamamoto, and B. W. Schuller.
  • Organizer: Proceedings of 44th Annual International Conference of IEEE Engineering in Medicine and Biology Society (EMBC 2022), pp. 4469-4472, 2022.
  • Int'l Joint Research