The document proposes a method called TxV for retrieving videos using natural language text queries. TxV combines various textual and visual features from inputs like BERT, CLIP and ResNet using an identity layer. It also uses a Dual Softmax Inference technique to revise the initial text-video similarities using background queries. Experimental results on datasets like MSR-VTT, IACC.3 and V3C1 showed that TxV outperforms state-of-the-art methods, with improvements in metrics like mean extended inferred average precision and recall. The approach provides an efficient way to combine multiple features for the text-video retrieval task.

1. Are all combinations equal?
Combining textual and visual features with
multiple space learning for text-based
video retrieval
Damianos Galanopoulos, Vasileios Mezaris
Information Technologies Institute-CERTH
ECCV2022 Workshop on AI for Creative Video Editing and Understanding

2. Problem statement
● Retrieving videos using natural-language text
○ “I would like to find a video where some kids are playing basketball in an open field”
Proposed Method
● We propose the TxV model
● Combination of various textual and visual features
● Utilization of prior text-video similarities for revising the ones
computed by our network

3. TxV model
bow
w2v
bert
clip
resnet152-11k
resnext101-wsl
clip vit-b/32
Identity
layer

4. Dual Softmax Inference (DSinf)
● Inspired from the Dual Softmax loss
● Text-video similarities revision
● Fixed set of predefined background queries

5. Experimental Results
● AVS datasets:
○ Training on MSR-VTT, TGIF, ActivityNet Captions and Vatex
○ Evaluation on IACC.3 and V3C1 evaluated on 2016-2018 and
2019-2021 queries V3C1 evaluated on TRECVID AVS
○ Mean extended inferred average precision (MXinfAP)
● MSR-VTT datasets:
○ MSR-VTT full and 1k-A splits
○ Recall R@k; Median rank (Medr); Mean average precision (mAP)

6. Experimental results
Results and SotA comparisons on the IACC.3 and V3C1 datasets

7. Experimental results
Results and SotA comparisons on the MSR-VTT full and 1k-A splits.

8. Experimental results
Different strategies on background queries selection when using DSinf

9. Conclusion
● We proposed the TxV model for the text-video retrieval task
● Efficient combination of multiple textual and visual features
● A dual softmax operation at the retrieval stage boosts the
performance
● Experimental results and comparisons on different datasets
document the value of our approach

10. Contact details
Damianos Galanopoulos, Vasileios Mezaris
Information Technologies Institute-CERTH
dgalanop@iti.gr, bmezaris@iti.gr
www.iti.gr/~bmezaris
This work was supported by the EU Horizon 2020 programme under grant
agreements H2020-101021866 CRiTERIA and H2020-832921 MIRROR.