Multi-functional CNN by Piping up Single Functions for Emergence of New Functions

• Project Area: Correspondence and Fusion of Artificial Intelligence and Brain Science
• Project/Area Number: 17H06026
• Research Category: Grant-in-Aid for Scientific Research on Innovative Areas (Research in a proposed research area)
• Allocation Type: Single-year Grants
• Review Section: Complex systems
• Research Institution: The University of Electro-Communications
• Principal Investigator: 柳井 啓司 電気通信大学, 大学院情報理工学研究科, 教授 (20301179)
• Project Period (FY): 2017-04-01 – 2019-03-31
• Project Status: Completed (Fiscal Year 2018)
• Budget Amount: ¥12,220,000 (Direct Cost: ¥9,400,000、Indirect Cost: ¥2,820,000); Fiscal Year 2018: ¥6,110,000 (Direct Cost: ¥4,700,000、Indirect Cost: ¥1,410,000); Fiscal Year 2017: ¥6,110,000 (Direct Cost: ¥4,700,000、Indirect Cost: ¥1,410,000)
• Keywords: 深層学習 / 畳み込みネットワーク / 同時学習 / 人工知能

Outline of Annual Research Achievements

本研究では，単一のCNN （Convolutional Neural Network, 畳み込みネットワーク）に複数のタスクのための機能を同時に学習させ，学習した独立の機能の組み合わせ・重ね合わせによって，事前に学習した個別の単一機能とは異なる，新しい機能を実現し，Neural Networkがより人間の脳に近い汎用的な能力を持っていることを実証することを目的とする．

2年目は，1年目の画像を入力して画像を出力するEncoder-Decoderネットワークのマルチファンクション学習の研究としてスタイル変換と領域分割の組み合わせ，を発展させて，より多くの画像変換タスクを1つのネットワーク学習可能とすることを実施した．特に，"PiggyBack"と呼ばれる方法を応用して１つのEncoder-Decoderネットワークに領域分割，coloring, スタイル変換を同時および逐次に学習することを実現した．

Research Progress Status

平成30年度が最終年度であるため、記入しない。

Strategy for Future Research Activity

平成30年度が最終年度であるため、記入しない。

Research Products

• [Journal Article] Simultaneous Estimation of Dish Locations and Calories with Multi-Task Learning (2019)
  • Author(s): Takumi Ege and Keiji Yanai
  • Journal Title: IEICE Transactions on Information and Systems Volume: E102.D Issue: 7 Pages: 1240-1246
  • DOI: 10.1587/transinf.2018CEP0004
  • NAID: 130007671316
  • ISSN: 0916-8532, 1745-1361
  • Year and Date: 2019-07-01
  • Peer Reviewed / Open Access
• [Journal Article] Webly-Supervised Food Detection with Foodness Proposal (2019)
  • Author(s): Wataru Shimoda and Keiji Yanai
  • Journal Title: IEICE Transactions on Information and Systems Volume: E102.D Issue: 7 Pages: 1230-1239
  • DOI: 10.1587/transinf.2018CEP0001
  • NAID: 130007671323
  • ISSN: 0916-8532, 1745-1361
  • Year and Date: 2019-07-01
  • Peer Reviewed / Open Access
• [Journal Article] Weakly Supervised Semantic Segmentation Using Distinct Class Specific Saliency Maps (2019)
  • Author(s): Wataru Shimoda and Keiji Yanai
  • Journal Title: Computer Vision and Image Understanding Volume: - Pages: 102712-102712
  • DOI: 10.1016/j.cviu.2018.08.006
  • Peer Reviewed
• [Journal Article] Image-search-based Food Calorie Estimation with CNN Feature Learning (2018)
  • Author(s): 會下拓実, 下田和, 柳井啓司
  • Journal Title: 電子情報通信学会論文誌D 情報・システム Volume: J101-D Issue: 8 Pages: 1099-1109
  • DOI: 10.14923/transinfj.2017IUP0009
  • ISSN: 1880-4535, 1881-0225
  • Year and Date: 2018-08-01
  • Peer Reviewed
• [Journal Article] Image-Based Food Calorie Estimation Using Recipe Information (2018)
  • Author(s): Takumi Ege and Keiji Yanai
  • Journal Title: IEICE Transactions on Information and Systems Volume: E101.D Issue: 5 Pages: 1333-1341
  • DOI: 10.1587/transinf.2017MVP0027
  • NAID: 130006729787
  • ISSN: 0916-8532, 1745-1361
  • Year and Date: 2018-05-01
  • Peer Reviewed
• [Journal Article] CNN 特徴量学習に基づく画像検索による食事画像カロリー量推定 (2018)
  • Author(s): 會下拓実, 柳井啓司
  • Journal Title: 電子情報通信学会論文誌 D Volume: 8
  • Peer Reviewed
• [Journal Article] Neural Style Vectorを用いた絵画画像のスタイル検索 (2017)
  • Author(s): 松尾真, 柳井啓司
  • Journal Title: 電子情報通信学会論文誌 D Volume: 8 Pages: 742-749
  • Peer Reviewed
• [Presentation] A Large-scale Analysis of Regional Tendency of Twitter Photos Using Only Image Features (2019)
  • Author(s): Tetsuya Nagano, Takumi Ege, Wataru Shimoda and Keiji Yanai
  • Organizer: Proc. of IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR)
  • Int'l Joint Research
• [Presentation] Image-Based Estimation of Real Food Size for Accurate Food Calorie Estimation (2019)
  • Author(s): Takumi Ege, Yoshikazu Ando, Ryosuke Tanno, Wataru Shimoda and Keiji Yanai
  • Organizer: Proc. of IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR)
  • Int'l Joint Research
• [Presentation] 米飯画像の実寸推定に基づく面積を考慮したカロリー量推定 (2019)
  • Author(s): 會下 拓実，柳井 啓司
  • Organizer: データ工学と情報マネジメントに関するフォーラム (DEIM)
• [Presentation] 深度付き画像と深層学習による食事カロリー量推定システムの開発 (2019)
  • Author(s): 安蒜 祥和，會下 拓実，柳井 啓司
  • Organizer: データ工学と情報マネジメントに関するフォーラム (DEIM)
• [Presentation] 画像変換ネットワークによる連続学習 (2019)
  • Author(s): 松本 晨人，柳井 啓司
  • Organizer: データ工学と情報マネジメントに関するフォーラム (DEIM)
• [Presentation] 位置情報付きTwitter画像を用いた世界の食事傾向分析 (2019)
  • Author(s): 岡本 開夢，柳井 啓司
  • Organizer: データ工学と情報マネジメントに関するフォーラム (DEIM)
• [Presentation] Conditional GANによる化粧顔画像変換 (2019)
  • Author(s): 五味 京祐，柳井 啓司
  • Organizer: データ工学と情報マネジメントに関するフォーラム (DEIM)
• [Presentation] 変化領域の推測による弱教師あり領域分割の精度向上 (2019)
  • Author(s): 下田 和, 柳井 啓司
  • Organizer: 電子情報通信学会 パターン認識・メディア理解研究会（PRMU）
• [Presentation] 深層学習による太陽画像からの太陽黒点数の推定 (2019)
  • Author(s): 樋口陽光, 會下拓実, 柳井啓司
  • Organizer: 電子情報通信学会 総合大会
• [Presentation] Multi-task Learning of Dish Detection and Calorie Estimation (2018)
  • Author(s): Takumi Ege and Keiji Yanai
  • Organizer: Proc. of International Workshop on Multimedia Assisted Dietary Management (MADIMA)
  • Int'l Joint Research
• [Presentation] Food Category Transfer with Conditional Cycle GAN and a Large-scale Food Image Dataset (2018)
  • Author(s): Daichi Horita, Ryosuke Tanno, Wataru Shimoda, Keiji Yanai
  • Organizer: Proc. of International Workshop on Multimedia Assisted Dietary Management (MADIMA)
  • Int'l Joint Research
• [Presentation] Food Image Generation using A Large Amount of Food Images with Conditional GAN: RamenGAN and RecipeGAN (2018)
  • Author(s): Yoshifumi Ito, Wataru Shimoda and Keiji Yanai
  • Organizer: Proc. of International Workshop on Multimedia Assisted Dietary Management (MADIMA)
  • Int'l Joint Research
• [Presentation] Magical Rice Bowl: Real-time Food Category Changer (2018)
  • Author(s): Ryosuke Tanno, Daichi Horita, Wataru Shimoda and Keiji Yanai
  • Organizer: ACM Multimedia, Demo Paper
  • Int'l Joint Research
• [Presentation] AR DeepCalorieCam V2: Food Calorie Estimation with CNN and AR-based Actual Size Estimation (2018)
  • Author(s): Ryosuke Tanno, Takumi Ege, Keiji Yanai
  • Organizer: ACM Symposium on Virtual Reality Software and Technology (VRST)
  • Int'l Joint Research
• [Presentation] CNN-based Photo Transformation for Improving Attractiveness of Ramen Photos (2018)
  • Author(s): Daichi Horita, Jaehyeong Cho, Takumi Ege, Keiji Yanai
  • Organizer: ACM Symposium on Virtual Reality Software and Technology (VRST)
  • Int'l Joint Research
• [Presentation] Word-Conditioned Image Style Transfer (2018)
  • Author(s): Yu Sugiyama and Keiji Yanai
  • Organizer: Proc. of ACCV Workshop on AI Aesthetics in Art and Media (AIAM)
  • Int'l Joint Research
• [Presentation] Font Style Transfer Using Neural Style Transfer and Unsupervised Cross-domain Transfer (2018)
  • Author(s): Atsushi Narusawa, Wataru Shimoda, and Keiji Yanai
  • Organizer: Proc. of ACCV Workshop on AI Aesthetics in Art and Media
  • Int'l Joint Research
• [Presentation] Real-Time Image Classification and Transformation Apps on iOS by "Chainer2MPSNNGraph" (2018)
  • Author(s): Yuki Izumi, Daichi Horita, Ryosuke Tanno and Keiji Yanai
  • Organizer: Proc. of NIPS WS on Machine Learning on the Phone and other Consumer Devices (MLPCD)
  • Int'l Joint Research
• [Presentation] Continual Learning for an Encoder-Decoder CNN Using "Piggyback" (2018)
  • Author(s): Asato Matsumoto and Keiji Yanai
  • Organizer: Proc. of NIPS Continual Learning Workshop
  • Int'l Joint Research
• [Presentation] CNN-based Food Transformation on HoloLens (2018)
  • Author(s): Shu Naritmo, Ryosuke Tanno, Takumi Ege, and Keiji Yanai
  • Organizer: Proc. of International Workshop on Interface and Experience Design with AI for VR/AR (DAIVAR)
  • Int'l Joint Research
• [Presentation] 深層学習による質感文字生成 (2018)
  • Author(s): 成沢 淳史, 下田 和, 柳井 啓司
  • Organizer: 人工知能学会全国大会
• [Presentation] 画像内容を考慮した質感表現に基づく画像変換 (2018)
  • Author(s): 杉山 優, 柳井 啓司
  • Organizer: 人工知能学会全国大会
• [Presentation] 大量のTwitter画像を用いたConditional Cycle GANによる食事写真カテゴリ変換 (2018)
  • Author(s): 堀田 大地, 成冨 志優, 丹野 良介, 下田 和, 柳井 啓司
  • Organizer: 人工知能学会全国大会
• [Presentation] 教師情報に含まれるノイズに堅牢な弱教師あり領域分割手法 (2018)
  • Author(s): 下田 和, 柳井 啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] CNNによる料理検出とカロリー量推定のマルチタスク学習 (2018)
  • Author(s): 會下 拓実, 柳井 啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] 単語情報を利用した画像の質感転送 (2018)
  • Author(s): 杉山 優, 柳井 啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] Chainer2MPSGraph: 高速深層学習 モバイルアプリ作成のためのモデルコンバータ (2018)
  • Author(s): 泉 裕貴, 堀田 大地, 丹野 良介, 柳井 啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] 画像マイニングを用いた Conditional Cycle GAN による食事画像変換 (2018)
  • Author(s): 堀田 大地, 丹野 良介, 下田 和, 柳井 啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] CNNを用いた質感文字生成 (2018)
  • Author(s): 成沢淳史, 下田和, 柳井啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] AR 技術とモバイル深層学習を活用した 食事カロリー量推定 (2018)
  • Author(s): 丹野良介, 會下拓実, Jaehyeong Cho, 柳井啓司
  • Organizer: 画像の認識・理解シンポジウム(MIRU)
• [Presentation] AR DeepCalorieCam: An iOS App for Food Calorie Estimation with Augmented Reality (2018)
  • Author(s): Ryosuke Tanno and Keiji Yanai
  • Organizer: the International Multimedia Modeling Conference (MMM)
  • Int'l Joint Research
• [Presentation] Predicting Segmentation Easiness from the Consistency for Weakly-Supervised Segmentation (2017)
  • Author(s): Wataru Shimoda and Keiji Yanai
  • Organizer: Asian Conference on Pattern Recognition (ACPR)
  • Int'l Joint Research
• [Presentation] Estimating Food Calories for Multiple-dish Food Photos (2017)
  • Author(s): Takumi Ege and Keiji Yanai
  • Organizer: Asian Conference on Pattern Recognition (ACPR)
  • Int'l Joint Research
• [Presentation] Image-Based Food Calorie Estimation Using Knowledge on Food Categories, Ingredients and Cooking Directions (2017)
  • Author(s): Takumi Ege and Keiji Yanai
  • Organizer: ACM Multimedia Thematic Workshops on Understanding
  • Int'l Joint Research
• [Presentation] Comparison of Two Approaches for Direct Food Calorie Estimation (2017)
  • Author(s): Takumi Ege and Keiji Yanai
  • Organizer: International Workshop on Multimedia Assisted Dietary Management (MADIMA)
  • Int'l Joint Research
• [Presentation] Partial Style Transfer Using Weakly-Supervised Semantic Segmentation (2017)
  • Author(s): Shin Matsuo, Wataru Shimoda and Keiji Yanai
  • Organizer: ICME Workshop on Multimedia Artworks Analysis (MMArt)
  • Int'l Joint Research
• [Presentation] Conditional Fast Style Transfer Network (2017)
  • Author(s): Keiji Yanai and Ryosuke Tanno
  • Organizer: ACM International Conference on Multimedia Retrieval (ICMR)
  • Int'l Joint Research
• [Presentation] Simultaneous Estimation of Food Categories and Calories with Multi-task CNN (2017)
  • Author(s): Takumi Ege and Keiji Yanai
  • Organizer: IAPR International Conference on Machine Vision Applications (MVA)
  • Int'l Joint Research
• [Presentation] Unseen Style Transfer Based on a Conditional Fast Style Transfer Network, Proc. International Conference on Learning Representation Workshop Track (ICLR WS) (2017)
  • Author(s): Keiji Yanai
  • Organizer: 2017
  • Int'l Joint Research
• [Presentation] Twitter Photo Geo-Localization Using Both Textual and Visual Features (2017)
  • Author(s): Shin Matsuo, Wataru Shimoda and Keiji Yanai
  • Organizer: International Conference on Multimedia Big Data (BIGMM)
  • Int'l Joint Research
• [Presentation] Learning Food Image Similarity for Food Image Retrieval (2017)
  • Author(s): Wataru Shimoda and Keiji Yanai
  • Organizer: International Conference on Multimedia Big Data (BIGMM)
  • Int'l Joint Research