Document reading analysis - towards smart documents

• Project/Area Number: 16K16089
• Research Category: Grant-in-Aid for Young Scientists (B)
• Allocation Type: Multi-year Fund
• Research Field: Perceptual information processing
• Research Institution: Osaka Prefecture University
• Principal Investigator: Augereau Olivier 大阪府立大学, 工学(系)研究科(研究院), 研究員 (10772436)
• Project Period (FY): 2016-04-01 – 2018-03-31
• Project Status: Completed (Fiscal Year 2017)
• Budget Amount: ¥4,030,000 (Direct Cost: ¥3,100,000、Indirect Cost: ¥930,000); Fiscal Year 2017: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000); Fiscal Year 2016: ¥2,470,000 (Direct Cost: ¥1,900,000、Indirect Cost: ¥570,000)
• Keywords: eye tracking / reading analysis / reading understanding / smart documents / smart document / adaptive document / 視覚メディア処理 / document analysis / reader analysis

Outline of Final Research Achievements

The aim of this research project is to create smart documents that can adapt themselves to the readers. Some sensors such as eye trackers (Tobii, SMI, pupil, JINS MEME) and physiological sensors (E4 wristband) are used to analyze information about the user while reading a document. For example we estimate the reader comprehension, confusion, concentration and emotion. At the same time, the document that the reader is interacting with is analyzed. The information about the document and the user behavior are combined to proceed a "mutual analysis". By knowing the difficulty of a document and the skill of the reader, the system can select an easier version of the document. Another application is to recommend a user to review specific parts of a lesson based on the confidence and correctness of his answers.

Research Products

• [Journal Article] Assessing Cognitive Workload on Printed and Electronic Media Using Eye-Tracker and EDA Wristband (2018)
  • Author(s): Brishtel, Iuliia and Ishimaru, Shoya and Augereau, Olivier and Kise, Koichi and Dengel, Andreas
  • Journal Title: Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion Volume: 1 Pages: 1-2
  • DOI: 10.1145/3180308.3180354
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Wordometer Systems for Everyday Life (2017)
  • Author(s): Olivier Augereau, Charles Lima Sanches, Koichi Kise, Kai Kunze
  • Journal Title: PACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume: 1 Issue: 4 Pages: 1-21
  • DOI: 10.1145/3161601
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Identification of Reader Specific Difficult Words by Analyzing Eye Gaze and Document Content (2017)
  • Author(s): Garain, Utpal and Pandit, Onkar and Augereau, Olivier and Okoso, Ayano and Kise, Koichi
  • Journal Title: Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on Volume: 1 Pages: 1346-1351
  • DOI: 10.1109/icdar.2017.221
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Using the eye gaze to predict document reading subjective understanding (2017)
  • Author(s): Sanches, Charles Lima and Augereau, Olivier and Kise, Koichi
  • Journal Title: Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on Volume: 8
  • DOI: 10.1109/icdar.2017.377
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Estimation of Confidence Based on Eye Gaze: An Application to Multiple-choice Questions (2017)
  • Author(s): Yamada, Kento and Kise, Koichi and Augereau, Olivier
  • Journal Title: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers Volume: 1 Pages: 217-220
  • DOI: 10.1145/3123024.3123138
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Vertical error correction of eye trackers in nonrestrictive reading condition (2016)
  • Author(s): Charles Lima Sanches, Olivier Augereau, Koichi Kise
  • Journal Title: IPSJ Transactions on Computer Vision and Applications Volume: 8 Issue: 1
  • DOI: 10.1186/s41074-016-0008-x
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Presentation] Towards an Automated Estimation of English Skill via TOEIC Score Based on Reading Analysis (2016)
  • Author(s): Olivier Augereau, Hiroki Fujiyoshi and Koichi Kise
  • Organizer: International Conference on Pattern Recognition
  • Place of Presentation: Cancun
  • Year and Date: 2016-12-05
  • Int'l Joint Research
• [Presentation] Emotional arousal estimation while reading comics based on physiological signal analysis (2016)
  • Author(s): Mizuki Matsubara, Olivier Augereau, Charles Lima Sanches, Koichi Kise
  • Organizer: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding
  • Place of Presentation: Cancun
  • Year and Date: 2016-12-04
  • Int'l Joint Research
• [Presentation] Manga content analysis using physiological signals (2016)
  • Author(s): Charles Lima Sanches, Olivier Augereau, Koichi Kise
  • Organizer: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding
  • Place of Presentation: Cancun
  • Year and Date: 2016-12-04
  • Int'l Joint Research
• [Presentation] Estimation of english skill with a mobile eye tracker (2016)
  • Author(s): Olivier Augereau, Kai Kunze, Hiroki Fujiyoshi, Koichi Kise
  • Organizer: UbiComp Adjunct 2016
  • Place of Presentation: Heidelberg
  • Year and Date: 2016-09-12
  • Int'l Joint Research