Skip to Content

Sketch-based Video Browser (or Video Hunter)

Type: 
Multimedia
One line description: 
An interactive video retrieval tool for known-item search tasks.
HTML link: 
project/sbvb
Annotation: 

Sketch-based Video Browser (or Video Hunter) is a new interactive video retrieval tool focusing on known-item search (KIS) tasks. In KIS tasks, users know about some video scene, know it is contained in a collection, but do not know where it is located. Therefore, users have to search/browse the collection with advanced techniques enabling query initialization, result visualization and browsing. Given such support, known-item search is not restricted just to ideal query formulation. The Video Hunter tool has participated in the Video Browser Showdown, winning the competition in 2014 and 2015. In both years 2016 and 2017, the Video Hunter tool achieved the third place.

The Video Hunter tool focuses on intuitive interactive interface, enabling users to materialize a query idea, supporting various forms of visualizations and browsing.

Query formulation

In order to materialize a known-item, users can upload an example image, provide keywords and/or draw a multi-sketch with color/edge features. The tool employs state-of-the-art deep convolutional neural networks to process queries based on example query objects (using DeCAF features) and keywords (annotations based on network classifications). The multi-sketch retrieval relies on position-color feature signatures and edge histograms.

Visualization

The visualization component for results supports two modes. The first mode displays the list of keyframes ordered with respect to the relevance to a query. The keyframes are accompanied with the temporal context, i.e., keyframes preceding/following the matched keyframe in the particular video. Basically, the results are displayed in the mode with one matched keyframe per one line with the temporal context. However, more results per line are enabled. The second mode displays a larger set of keyframes as a color-sorted map, without a temporal context. This second option is suitable for first phases of the retrieval where users first try to find a candidate query object. Beside this two main modes, the tool supports also interactive navigation summaries and compact visualizations of groups of similar keyframes.

Browsing

Vertical scrolling of the pages of ranked results is a standard way to improve recall. Once users find a promising scene or video, inspection of the temporal neighborhood is required. The Video Hunter tool enables horizontal browsing for each result line (extension of the temporal context) and also browsing using the interactive navigation summary.

Download the tool

References

  1. C. Cobarzan, K. Schoeffmann, W. Bailer, W. Hurst, A. Blažek, J. Lokoč, S. Vrochidis, K. U. Barthel, L. Rossetto. Interactive Video Search Tools: A Detailed Analysis of the Video Browser Showdown 2015, Multimedia tools and applications, Springer, 76(4): 5539-5571, 2017
  2. Adam Blazek, Jakub Lokoc, David Kubon: Video Hunter at VBS 2017. MMM, 2017: 493-498
  3. Adam Blazek, David Kubon, Jakub Lokoc: Known-Item Search in Video Databases with Textual Queries. SISAP 2016: 117-124
  4. David Kubon, Adam Blazek, Jakub Lokoc, Tomás Skopal: Multi-sketch Semantic Video Browser. MMM (2) 2016: 406-411
  5. Adam Blazek, Jakub Lokoc, Filip Matzner, Tomás Skopal: Enhanced Signature-Based Video Browser. MMM (2) 2015: 243-248
  6. Jakub Lokoc, Adam Blazek, Tomás Skopal: Signature-Based Video Browser. MMM (2) 2014: 415-418
  7. Adam Blazek, Jakub Lokoc, Tomás Skopal: Video Retrieval with Feature Signature Sketches. SISAP 2014: 25-36

 



by Dr. Radut.