Cross-Modal Video Retrieval Model Based on Video-Text Dual Alignment

Cross-modal video retrieval remains a major challenge in natural language processing due to the natural semantic divide between video and text. Most approaches use a single encoder to extract video and text features separately, and train video-text pairs by means of contrastive learning, but this global alignment of video and text is prone to neglecting more fine-grained features of both. In addition, some studies focus only on profiling the video description text, ignoring the correlation relationship with the video. Therefore, this paper proposes a video retrieval method based on video-text alignment, which realizes both global and fine-grained alignment between video and text. For global alignment, the video and text are aligned by a single encoder and after linear projection; for fine-grained alignment, the video encoder is trained to align the video and text by masking some semantic information in the text. By experimentally comparing with multiple existing methods on MSR-VTT and MSVD datasets, the model achieves R@1 (recall at 1) metrics of 51.5% and 52.4% on MSR-VTT and MSVD datasets, respectively, which indicates that the proposed model can improve the efficiency of cross-modal video retrieval.

@article{Che2024,
  title     = {Cross-Modal Video Retrieval Model Based on Video-Text Dual Alignment},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {15},
  number    = {2},
  year      = {2024},
  publisher = {The Science and Information Organization},
  author    = {Zhanbin Che and Huaili Guo},
  doi       = {10.14569/IJACSA.2024.0150232},
  url       = {https://doi.org/10.14569/IJACSA.2024.0150232}
}