Columbia University

Technology Ventures

Indexing and searching algorithm for video content on consumer video sites

Technology #cu12269

This technology describes a video search engine for identifying scenes, events, locations and objects featured in on-line videos. The audio and visual components of a video stream are separately analyzed to create characteristic elements referred to as audio-visual atoms. Visual data is partitioned into short sections that are analyzed to identify objects and motion that yield information regarding the video’s contents. Multiple sections can be compared and combined to identify the most significant visual atoms. Characteristic sounds identified in the audio data are decomposed and simplified to create audio atoms. The combined audio-visual atoms can then form a characteristic signature of the video contents that can be compared to a database of video data to identify categories that best describe the video’s contents.

Identification of videos based upon audio-visual atoms enables content-based video search services

Analysis and automated sorting of videos using audio-visual atoms enables search engines to locate videos based upon their actual content rather than by user-provided descriptions, comments, or labels associated with the videos.

The technology was demonstrated by training the algorithm using part of Kodak’s consumer benchmark video collection and successfully using it to produce accurate content descriptions of the remaining videos in the collection.

Lead Inventor:

Shih-Fu Chang, Ph.D.


  • Search engine classification of on-line consumer videos.
  • Contextual search for video content on personal computers.
  • Automatic contextual sorting of videos.
  • Automatic identification of offensive or banned video material.


  • Enables automated indexing and categorization of videos by content rather than existing description or label.
  • Provides quantitative data for tracking trends in on-line video sites.
  • Combines audio and video data for more thorough classifications.

Patent information:

Patent Issued (US 8,135,221)

Tech Ventures Reference: IR CU12269

Related Publications: