Semantic reasoning in zero example video event retrieval

Semantic reasoning in zero example video event retrieval

de Boer, M.H.T.
Lu, Y.J.
Zhang, H.
Schutte, K.
Ngo, C.W.
Kraaij, W.

2017

Searching in digital video data for high-level events, such as a parade or a car accident, is challenging when the query is textual and lacks visual example images or videos. Current research in deep neural networks is highly beneficial for the retrieval of high-level events using visual examples, but without examples it is still hard to (1) determine which concepts are useful to pre-train (Vocabulary challenge) and (2) which pre-trained concept detectors are relevant for a certain unseen high-level event (Concept Selection challenge). In our article, we present our Semantic Event Retrieval System which (1) shows the importance of high-level concepts in a vocabulary for the retrieval of complex and generic high-level events and (2) uses a novel concept selection method (i-w2v) based on semantic embeddings. Our experiments on the international TRECVID Multimedia Event Detection benchmark show that a diverse vocabulary including high-level concepts improves performance on the retrieval of high-level events in videos and that our novel method outperforms a knowledge-based concept selection method. © 2017 ACM.

2016 ICT 2015 Observation, Weapon & Protection Systems
DSC - Data Science II - Intelligent Imaging
TS - Technical Sciences
Content-based visual information retrieval
Multimedia event detection
Semantics
Zero shot
Accidents
Benchmarking
Deep neural networks
Image retrieval
Knowledge based systems
Multimedia systems
Semantics
Concept selection
Content-based visual information retrieval
Digital video data
Knowledge based
Multimedia event detections
Retrieval systems
Semantic reasoning
Zero shot
Computer graphics

http://resolver.tudelft.nl/uuid:f0979246-3772-44f5-a205-e11cc17185b0

782443

1551-6857

ACM Transactions on Multimedia Computing, Communications and Applications, 13 (4)

60

article