Research on a Multi-modal Registration Model for Video Matching and Its Advanced Applications

• Project/Area Number: 20H00607
• Research Category: Grant-in-Aid for Scientific Research (A)
• Allocation Type: Single-year Grants
• Section: 一般
• Review Section: Medium-sized Section 61:Human informatics and related fields
• Research Institution: Osaka University
• Principal Investigator: 槇原 靖 大阪大学, 産業科学研究所, 教授 (90403005)
• Project Period (FY): 2020-04-01 – 2024-03-31
• Project Status: Granted (Fiscal Year 2023)
• Budget Amount: ¥44,850,000 (Direct Cost: ¥34,500,000、Indirect Cost: ¥10,350,000); Fiscal Year 2023: ¥10,270,000 (Direct Cost: ¥7,900,000、Indirect Cost: ¥2,370,000); Fiscal Year 2022: ¥10,270,000 (Direct Cost: ¥7,900,000、Indirect Cost: ¥2,370,000); Fiscal Year 2021: ¥12,480,000 (Direct Cost: ¥9,600,000、Indirect Cost: ¥2,880,000); Fiscal Year 2020: ¥11,830,000 (Direct Cost: ¥9,100,000、Indirect Cost: ¥2,730,000)
• Keywords: コンピュータビジョン / パターン認識 / バイオメトリクス

Outline of Research at the Start

本研究では、空間・時間・色（輝度）を含む多元的な位置合わせに基づく映像照合の枠組みを構築する。特に、空間・時間・色の変形と変形前の映像を分離して推定するモデルや、変形場自体を後段の映像照合に活用する手法、位置合わせ過程を考慮した判断根拠の提示手法を開発する。また、提案手法の有効性を、人物映像照合実験等を通して検証する。

Outline of Annual Research Achievements

P1) 解きほぐし表現学習による空間・時間・色の多元位置合わせモデル
P1-2) 解きほぐし表現のネットワーク構造と損失関数の設計：2021年度は，非同期の多視点映像を時間的に位置合わせするためのネットワーク構造を設計し、非同期多視点人物映像を対象とした、人体モデル当てはめの問題に取り組んだ。2022年度は、更に、人物映像が一部欠落した場合に空間的な位置合わせを行うネットワーク構造を設計することで、より適用範囲の広い人物モデル当てはめ手法へと拡張した。特に，人物の欠損部分によって推定結果が不安定化するカメラパラメータに対して，人物の足下や頭頂などをアンカー点として制約することで安定化する枠組みを実装した．
P2) 空間・時間・色の多元変形場を活用した映像照合
P2-1) 映像照合ネットワークの構築：上記の空間欠損データに対して推定される高精度な3次元人物モデル系列を入力とする歩容認証ネットワークを設計した。特に、欠損割合による特徴の変化を抑制するための特徴変換器を導入することで、上部欠落や下部欠落、また、それらが時間的に一定である場合や変化する場合など，様々な空間的欠損パターンに対しても高精度な認証が可能であることを示した。
P2-2) 変形場自体を手がかりとした映像照合：部分欠落を含む歩行シルエットを入力とした歩容認証手法において，欠落割合の程度によって相違度の分布が異なることから，空間変形の一つである欠落割合を手がかりとした相違度正規化ネットワークを設計し，大規模歩行映像データベースを用いて，その有用性を確認した。

Current Status of Research Progress

2: Research has progressed on the whole more than it was originally planned.

Reason

空間的欠落を含む映像に対する照合問題に取り組むことで，当初計画にある項目の内容を実施できたため．

Strategy for Future Research Activity

当該年度の成果を元に、次年度は以下の項目の研究を推進する。
P1) 解きほぐし表現学習による空間・時間・色の多元位置合わせモデル
P1-3) 解きほぐし表現を利用したデータ拡張
画像認識で頻繁に用いられる並進やスケーリングといった単純なデータ拡張とは異なり、解きほぐし表現によるデータ拡張は、実際のデータの組み合わせによって学習サンプルを生成できる点が優れている。本研究の場合、具体的には、ある映像ペアから得られた変形場を、別の映像ペアから生成された変形前の映像に作用させることで、様々な空間・時間・色変形を伴う映像ペアを生成する。拡張された学習データを用いて 、映像照合の精度評価を行う。
P3) 位置合わせ過程を考慮した判断根拠の提示
GradCam等の深層学習向けの判断根拠の可視化手法を、位置合わせ過程を考慮した本手法用に拡張する。具体的には、変形前の位置の合った映像ペア上で相違の原因となった箇所を可視化すると共に、空間変形場、時間変形場（時間伸縮関数）、色変形場（色変換関数）の上でも、相違の原因となった箇所を可視化する。これにより、本手法の多元位置合わせモデルによる映像照合のいずれの過程で相違が生じたかを明らかにする。被験者による主観評価を通して、従来の位置ずれを伴う映像上での可視化との比較実験を行う。

Research Products

• [Int'l Joint Research] 南方科技大学/深セン大学(中国)
• [Int'l Joint Research] エジプト日本科学技術大学(エジプト)
• [Int'l Joint Research] University of Southampton(英国)
• [Int'l Joint Research] 南京理工大学/南方科技大学(中国)
• [Int'l Joint Research] 日本エジプト科学技術大学(エジプト)
• [Journal Article] Occlusion-Aware Human Mesh Model-Based Gait Recognition (2023)
  • Author(s): Xu Chi、Makihara Yasushi、Li Xiang、Yagi Yasushi
  • Journal Title: IEEE Transactions on Information Forensics and Security Volume: 18 Pages: 1309-1321
  • DOI: 10.1109/tifs.2023.3236181
  • Peer Reviewed / Open Access
• [Journal Article] Batch Hard Contrastive Loss and Its Application to Cross-View Gait Recognition (2023)
  • Author(s): Aljazaerly Mohamad Ammar Alsherfawi、Makihara Yasushi、Muramatsu Daigo、Yagi Yasushi
  • Journal Title: IEEE Access Volume: 11 Pages: 31177-31187
  • DOI: 10.1109/access.2023.3262271
  • Peer Reviewed / Open Access
• [Journal Article] Multi-View Large Population Gait Database With Human Meshes and Its Performance Evaluation (2022)
  • Author(s): Li Xiang、Makihara Yasushi、Xu Chi、Yagi Yasushi
  • Journal Title: IEEE Transactions on Biometrics, Behavior, and Identity Science Volume: 4 Issue: 2 Pages: 234-248
  • DOI: 10.1109/tbiom.2022.3174559
  • Peer Reviewed / Open Access
• [Journal Article] Investigating strategies towards adversarially robust time series classification (2022)
  • Author(s): Abdu-Aguye Mubarak G.、Gomaa Walid、Makihara Yasushi、Yagi Yasushi
  • Journal Title: Pattern Recognition Letters Volume: 156 Pages: 104-111
  • DOI: 10.1016/j.patrec.2022.01.023
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Detecting Lower MMSE Scores in Older Adults Using Cross-Trial Features From a Dual-Task With Gait and Arithmetic (2021)
  • Author(s): Wu Shuqiong、Matsuura Taku、Okura Fumio、Makihara Yasushi、Zhou Chengju、Aoki Kota、Mitsugami Ikuhisa、Yagi Yasushi
  • Journal Title: IEEE Access Volume: 9 Pages: 150268-150282
  • DOI: 10.1109/access.2021.3126067
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Early Detection of Low Cognitive Scores from Dual-task Performance Data Using a Spatio-temporal Graph Convolutional Neural Network (2021)
  • Author(s): Wu Shuqiong、Okura Fumio、Makihara Yasushi、Aoki Kota、Niwa Masataka、Yagi Yasushi
  • Journal Title: The 43rd Annual Int. Conf of the IEEE Engineering in Medicine and Biology Society (EMBC 2021) Volume: - Pages: 1895-1901
  • DOI: 10.1109/embc46164.2021.9630304
  • Peer Reviewed / Open Access
• [Journal Article] End-to-end Model-based Gait Recognition using Synchronized Multi-view Pose Constraint (2021)
  • Author(s): Li Xiang、Makihara Yasushi、Xu Chi、Yagi Yasushi
  • Journal Title: Prof. of the 1st Int. Workshop on Human-centric Trustworthy Computer Vision: From Research to Applications Volume: - Pages: 4106-4115
  • DOI: 10.1109/iccvw54120.2021.00456
  • Peer Reviewed / Open Access
• [Journal Article] HID 2021: Competition on Human Identification at a Distance 2021 (2021)
  • Author(s): Yu Shiqi、Huang Yongzhen、Wang Liang、Makihara Yasushi、Garcia Reyes Edel B.、Zheng Feng、Rahman Ahad Md Atiqur、Lin Beibei、Yang Yuchao、Xiong Haijun、Huang Binyuan、Zhang Yuxuan
  • Journal Title: Prof. of the 5th Int. Joint Conf. on Biometrics (IJCB 2021) Volume: 1 Pages: 1-7
  • DOI: 10.1109/ijcb52358.2021.9484377
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Estimation of Gait Relative Attribute Distributions using a Differentiable Trade-off Model of Optimal and Uniform Transports (2021)
  • Author(s): Makihara Yasushi、Hayashi Yuta、Shehata Allam、Muramatsu Daigo、Yagi Yasushi
  • Journal Title: Prof. of the 5th Int. Joint Conf. on Biometrics (IJCB 2021) Volume: - Pages: 1-8
  • DOI: 10.1109/ijcb52358.2021.9484362
  • Peer Reviewed / Open Access
• [Journal Article] Learn to Walk Across Ages: Motion Augmented Multi-age Group Gait Video Translation (2021)
  • Author(s): Y. Zhang, Y. Makihara, D. Muramatsu, J. Zhang, L. Niu, L. Zhang, Y. Yagi
  • Journal Title: IEEE Access Volume: 9 Pages: 40550-40559
  • DOI: 10.1109/access.2021.3061684
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] End-to-End Model-Based Gait Recognition (2021)
  • Author(s): Li Xiang、Makihara Yasushi、Xu Chi、Yagi Yasushi、Yu Shiqi、Ren Mingwu
  • Journal Title: Proc. of the 15th Asian Conf. on Computer Vision Volume: 1 Pages: 3-20
  • DOI: 10.1007/978-3-030-69535-4_1
  • ISBN: 9783030695347, 9783030695354
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Real-Time Gait-Based Age Estimation and Gender Classification from a Single Image (2021)
  • Author(s): C. Xu, Y. Makihara, R. Liao, H. Niitsuma, X. Li, Y. Yagi, J. Lu
  • Journal Title: Proc. of the IEEE Winter Conf. on Applications of Computer Vision 2021 (WACV 2021) Volume: 1 Pages: 1-11
  • Peer Reviewed / Open Access
• [Journal Article] Wearable Sensor-based Gait Analysis for Age and Gender Estimation (2020)
  • Author(s): M.A.R. Ahad, T.T. Ngo, A.D. Antar, M. Ahmed, T. Hossain, D. Muramatsu, Y. Makihara, S. Inoue, Y. Yagi
  • Journal Title: Sensors, MDPI Volume: 20 Issue: 8 Pages: 1-24
  • DOI: 10.3390/s20082424
  • NAID: 120007145039
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Gait recognition invariant to carried objects using alpha blending generative adversarial networks (2020)
  • Author(s): Xiang Li, Yasushi Makihara, Chi Xu, Yasushi Yagi, Mingwu Ren
  • Journal Title: Pattern Recognition Volume: 105 Pages: 1-12
  • DOI: 10.1016/j.patcog.2020.107376
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Performance Evaluation of Model-based Gait on Multi-view Very Large Population Database with Pose Sequences (2020)
  • Author(s): W. An, S. Yu, Y. Makihara, X. Wu, C. Xu, Y. Yu, R. Liao, Y. Yagi
  • Journal Title: IEEE Trans. on Biometrics, Behavior, and Identity Science Volume: 2 Issue: 4 Pages: 421-430
  • DOI: 10.1109/tbiom.2020.3008862
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Gait Recognition via Semi-supervised Disentangled Representation Learning to Identity and Covariate Features (2020)
  • Author(s): Li Xiang、Makihara Yasushi、Xu Chi、Yagi Yasushi、Ren Mingwu
  • Journal Title: Proc. the 33rd IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2020) Volume: 1 Pages: 13309-13319
  • DOI: 10.1109/cvpr42600.2020.01332
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Gait Recognition from a Single Image Using a Phase-Aware Gait Cycle Reconstruction Network (2020)
  • Author(s): Xu Chi、Makihara Yasushi、Li Xiang、Yagi Yasushi、Lu Jianfeng
  • Journal Title: Proc. of the 16th European Conf. on Computer Vision (ECCV 2020) Volume: 1 Pages: 386-403
  • DOI: 10.1007/978-3-030-58529-7_23
  • ISBN: 9783030585280, 9783030585297
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] How Confident Are You in Your Estimate of a Human Age? Uncertainty-aware Gait-based Age Estimation by Label Distribution Learning (2020)
  • Author(s): Sakata Atsuya、Makihara Yasushi、Takemura Noriko、Muramatsu Daigo、Yagi Yasushi
  • Journal Title: Proc. of the 4th Int. Joint Conference on Biometrics (IJCB 2020) Volume: 1 Pages: 1-10
  • DOI: 10.1109/ijcb48548.2020.9304914
  • Peer Reviewed / Open Access
• [Journal Article] DeformGait: Gait Recognition under Posture Changes using Deformation Patterns between Gait Feature Pairs (2020)
  • Author(s): XU Chi、Adachi Daisuke、Makihara Yasushi、Yagi Yasushi、Lu Jianfeng
  • Journal Title: Proc. of the 4th Int. Joint Conference on Biometrics (IJCB 2020) Volume: 1 Pages: 1-10
  • DOI: 10.1109/ijcb48548.2020.9304902
  • Peer Reviewed / Open Access
• [Presentation] Multi-view Large Population Gait Database with Human Meshes and Its Performance Evaluation (2023)
  • Author(s): X. Li, Y. Makihara, C. Xu, Y. Yagi
  • Organizer: The 26th SANKEN International Symposium
  • Int'l Joint Research
• [Presentation] Recognizing Human Identity, Age, and Aesthetic Attributes from Gait (2022)
  • Author(s): Y. Makihara
  • Organizer: The 9th IIAE Int. Conf. on Intelligent Systems and Image Processing (ICISIP 2022)
  • Int'l Joint Research / Invited
• [Presentation] Recognizing Human Identity, Age, and Aesthetic Attributes from Gait (2022)
  • Author(s): Y. Makihara
  • Organizer: The 17th IEEE Computer Society Workshop on Biometrics
  • Int'l Joint Research / Invited
• [Presentation] 歩行映像解析とその医療・健康分野への応用 (2022)
  • Author(s): 槇原 靖
  • Organizer: 学術情報メディアセンターセミナー「高齢社会を支えるバイオメカニクス技術」
  • Invited
• [Presentation] 人物検出および照合の全体最適化による歩容認証 (2022)
  • Author(s): 沖村達平, 槇原靖, 八木康史
  • Organizer: 研究報告コンピュータビジョンとイメージメディア（CVIM）
• [Presentation] 人物検出及び照合の全体最適化による歩容認証 (2022)
  • Author(s): 沖村達平, 槇原靖, 八木康史
  • Organizer: 第25回 画像の認識・理解シンポジウム (MIRU2022)
• [Presentation] End-to-end Model-based Gait Recognition using Synchronized Multi-view Pose Constraint (2022)
  • Author(s): X. Li, Y. Makihara, C. Xu, Y. Yagi
  • Organizer: The 25th SANKEN International Symposium
  • Int'l Joint Research
• [Presentation] Real-Time Gait-Based Age Estimation and Gender Classification from a Single Image (2022)
  • Author(s): C. Xu, Y. Makihara, R. Liao, H. Niitsuma, X. Li, Y. Yagi, J. Lu
  • Organizer: The 25th SANKEN International Symposium
  • Int'l Joint Research
• [Presentation] Video-based Gait Analysis and Its Applications (2022)
  • Author(s): Y. Makihara
  • Organizer: The 11th Int. Workshop on Image Media Quality and its Applications (IMQA 2022)
  • Int'l Joint Research / Invited
• [Presentation] Early Detection of Low Cognitive Scores from Dual-task Performance Data Using a Spatio-temporal Graph Convolutional Neural Network (2021)
  • Author(s): S. Wu, F. Okura, Y. Makihara, K. Aoki, M. Niwa, and Y. Yagi
  • Organizer: The 43rd Annual Int. Conf of the IEEE Engineering in Medicine and Biology Society (EMBC 2021)
  • Int'l Joint Research
• [Presentation] End-to-end Model-based Gait Recognition using Synchronized Multi-view Pose Constraint (2021)
  • Author(s): X. Li, Y. Makihara, C. Xu, Y. Yagi
  • Organizer: The 1st Int. Workshop on Human-centric Trustworthy Computer Vision: From Research to Applications,
  • Int'l Joint Research
• [Presentation] HID 2021: Competition on Human Identification at a Distance 2021 (2021)
  • Author(s): S. Yu, Y. Huang, L. Wang, Y. Makihara, E.G. Reyes, F. Zheng, M.A.R. Ahad, B. Lin, Y. Yang, X. Haijun, B. Huang, Y. Zhang
  • Organizer: The 5th Int. Joint Conf. on Biometrics (IJCB 2021)
  • Int'l Joint Research
• [Presentation] Estimation of Gait Relative Attribute Distributions using a Differentiable Trade-off Model of Optimal and Uniform Transports (2021)
  • Author(s): Y. Makihara, Y. Hayashi, A. Shehata, D. Muramatsu, Y. Yagi
  • Organizer: The 5th Int. Joint Conf. on Biometrics (IJCB 2021)
  • Int'l Joint Research
• [Presentation] Estimation of Gait Relative Attribute Distributions using a Differentiable Trade-off Model of Optimal and Uniform Transports (2021)
  • Author(s): Y. Makihara, Y. Hayashi, A. Shehata, D. Muramatsu, Y. Yagi
  • Organizer: 第11回バイオメトリクスと認識・認証シンポジウム
• [Presentation] 部分欠落および観測視点変化を含む歩容認証の頑健性評価 (2021)
  • Author(s): 辻 翔吾, 槇原 靖, 八木 康史
  • Organizer: コンピュータビジョンとイメージメディア（CVIM）2021年5月研究会
• [Presentation] Video-based Gait Analysis and Its Applications (2021)
  • Author(s): Y. Makihara
  • Organizer: nt. Conf. on Big Data, IoT and Machine Learning (BIM 2021)
  • Int'l Joint Research / Invited
• [Presentation] Video-based Gait Analysis and Its Applications (2021)
  • Author(s): Y. Makihara
  • Organizer: The 6th Int. Symposium on Artificial Intelligence and Robotics (ISAIR 2021)
  • Int'l Joint Research / Invited
• [Presentation] On Gait Relative Attributes (2021)
  • Author(s): Y. Makihara
  • Organizer: 5th Int. Workshop on Symbolic-Neural Learning (SNL-2021)
  • Int'l Joint Research / Invited
• [Presentation] End-to-end Model-based Gait Recognition (2021)
  • Author(s): X. Li, Y. Makihara, C. Xu, Y. Yagi, S. Yu, M. Ren
  • Organizer: The 15th Asian Conf. on Computer Vision
  • Int'l Joint Research
• [Presentation] Real-Time Gait-Based Age Estimation and Gender Classification from a Single Image (2021)
  • Author(s): C. Xu, Y. Makihara, R. Liao, H. Niitsuma, X. Li, Y. Yagi, J. Lu
  • Organizer: The IEEE Winter Conf. on Applications of Computer Vision 2021 (WACV 2021)
  • Int'l Joint Research
• [Presentation] Video-based Gait Analysis and Its Applications (2020)
  • Author(s): Y. Makihara
  • Organizer: The 3rd Int. Conf. on Multimedia Analysis and Pattern Recognition
  • Int'l Joint Research / Invited
• [Presentation] Gait Recognition via Semi-supervised Disentangled Representation Learning to Identity and Covariate Features (2020)
  • Author(s): Xiang Li, Yasushi Makihara, Chi Xu, Yasushi Yagi, Mingwu Ren
  • Organizer: The 33rd IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2020)
  • Int'l Joint Research
• [Presentation] Gait Recognition from a Single Image using a Phase-Aware Gait Cycle Reconstruction Network (2020)
  • Author(s): C. Xu, Y. Makihara, X. Li, Y. Yagi, J. Lu
  • Organizer: The 16th European Conf. on Computer Vision (ECCV 2020)
  • Int'l Joint Research
• [Presentation] How Confident Are You in Your Estimate of a Human Age? Uncertainty-aware Gait-based Age Estimation by Label Distribution Learning (2020)
  • Author(s): A. Sakata, Y. Makihara, N. Takemura, D. Muramatsu, Y. Yagi
  • Organizer: The 4th Int. Joint Conference on Biometrics (IJCB 2020)
  • Int'l Joint Research
• [Presentation] DeformGait: Gait Recognition under Posture Changes using Deformation Patterns between Gait Feature Pairs (2020)
  • Author(s): C. Xu, D. Adachi, Y. Makihara, Y. Yagi, J. Lu
  • Organizer: The 4th Int. Joint Conference on Biometrics (IJCB 2020)
  • Int'l Joint Research
• [Patent(Industrial Property Rights)] 周期画像復元装置及び方法、識別装置及び方法、検証装置及び方法、特徴抽出装置、訓練方法、並びに位相推定装置 (2020)
  • Inventor(s): 八木 康史、槇原 靖、XU CHI、LI XIANG
  • Industrial Property Rights Holder: 八木 康史、槇原 靖、XU CHI、LI XIANG
  • Industrial Property Rights Type: 特許
  • Filing Date: 2020