Columbia University

Technology Ventures

Video classification system automatically identifies, characterizes, and searches for event information in unconstrained video

Technology #cu14303

Over 100 hours of video are uploaded to the Internet every minute, providing a vast amount of information for activities such as data mining and training computer-learning algorithms. This technology is a system that applies a classification algorithm to an unconstrained video file. It builds upon the IBM-Columbia multimedia event recounting (MER) system and solves the problem of automated video browsing and text summarization. The algorithm takes a video file as input and recognizes clusters that are close in time and correspond to a particular event. Textual outputs are generated which classify the visual, action, and audial characteristics of each segment in the video. These textual outputs can be used to automatically classify vast amounts of unconstrained video information. Thus, this technology may help video information become amenable to both simple querying and advanced data mining.

Semantic ontology-based processing pipeline produces above average clarity of explanation in one-sixth the video time

Currently, the only way videos can be indexed is through user-inputted text captions. If a video does not contain this extra information, it cannot be indexed. Using a set of design decisions based on the close connection between the ontology of semantic classifiers and the functional aspects of natural language, this technology can automatically summarize video files in one-sixth the usual time and produce text captions useful for understanding what a video is truly about.

The algorithm is informed throughout by human psychology and user studies, and has been validated with user input and training videos.

Lead Inventor:

John R. Kender, Ph.D.


  • Automated video classification
  • Video hosting sites like Youtube, Netflix, Vimeo, and Hulu can verify the content of uploaded videos
  • Training of video detection software and learning algorithms
  • Advertisers can determine what message is being conveyed to viewers
  • Make videos more accessible to the vision impaired
  • Language learning software that automatically matches video and words
  • Mine data in vast amounts of video
  • Automatically create trailers for a video
  • Archive historical media
  • Automatic select shots for action movie


  • Allows review of videos in 1/6 of the usual time
  • Simple text description based on human preference

Patent information:

Patent Pending (WO/2005/072239)

Tech Ventures Reference: IR CU14303

Related Publications: