2017 Fiscal Year Research-status Report
Participatory Sensing and Felicitous Recommending of Venues
| Project/Area Number |
16K16058
|
|---|
| Research Institution | National Institute of Informatics |
|---|
Principal Investigator |
ュ イ 国立情報学研究所, コンテンツ科学研究系, 特任助教 (00754681)
|
|---|
| Project Period (FY) |
2016-04-01 – 2019-03-31
|
| Keywords | マルチモーダルディープラーニング / ディープハッシング / 文書及び画像間クロスモーダル検索 |
|---|
| Outline of Annual Research Achievements |
We take Wikipedia featured articles and photos for venues as basic knowledge to learn a deep correlation model for fine-grained venue discovery from Foursquare photos. Specifically, we are interested in the challenging research problem of venue discovery from multimodal dataset: given a photo (with a rough position) as an input, the system returns its exact venue name (i.e., in which venue the photo was taken), category, and textual description.
This work has demonstrated the first study on visual venue discovery over an integrated venue-related multimodal dataset. We proposed a novel framework for deep correlation learning to realize fine-grained venue discovery. In particular, we apply a deep learning model, deep canonical correlation analysis (DCCA), to learn the correlations between venue photos and venue descriptions obtained from Wikipedia and Foursquare. Particularly, our contribution is three-fold: i) A novel dataset for venue-related multimodal contents is created based on integrating venue photos and descriptions from Wikipedia and Foursquare to solve fine-grained venue discovery with the aim of academic research, ii) An end-to-end deep network with two branches CNN and Doc2vec is trained, which converts different views into the same space and maximizes their correlations there, and iii) Extensive experiments verify the practicability of the DCCA model for fine-grained venue discovery, where DCCA outperforms state-of-the-art methods such as KCCA [4]. Some dataset are available on http://research.nii.ac.jp/_yiyu/VenueNet.htm for the research purpose.
|
| Current Status of Research Progress |
Current Status of Research Progress
2: Research has progressed on the whole more than it was originally planned.
Reason
The current research progress is going well. Some initial results have been published in the international conferences. We are working more tasks which are related to venue-related cross-modal deep hashing for efficient retrieval.
|
| Strategy for Future Research Activity |
There is still much room for improvements,
as follows: i) Based on our dataset, some topics will be investigated, e.g. cross-modal retrieval and image question answering where visual objects are described by natural language with different levels of understanding. ii) We still keep enlarging the number of image-text pairs for venues to investigate more interesting questions, for example, is there any correlation between visual objects and categories? iii) We will try to incorporate more data domains such as checkins and tips for personalized venue recommendation. iv) We will investigate more deep learning methods such as long short term memory(LSTM) for processing Wikipedia articles.
|
| Causes of Carryover |
Last year, I planned to attend ACMMM17 held in Mountain View, CA USA. Since I had to prepare for my courses, I did not have time to go for USA. This year, we have submitted the papers to ACMMM18 and plan to submit the paper to ICDM18. So, I would like to use this budget to support me or my student to attend ACMMM18 held in Seoul, Korea or attend IEEE ICDM2018 held in Singapore.
|
Research Products
(2 results)
