Columbia University

Technology Ventures

Graph-based multimedia annotation system facilitates rapid retrieval and exploration of large image and video collections

Technology #m10-016

The recent explosion in multimedia usage has resulted in a high demand for algorithms and applications to search large collections of images and videos. This technology describes the Columbia Transductive Annotation by Graph (TAG) system. TAG is designed to facilitate rapid retrieval and exploration of large image and video databases. It incorporates label propagation techniques and intuitive graphic user interfaces that allow users to quickly browse and annotate a small number of images and videos, then generate refined labels for all remaining data in the collection. This technology can be applied to domains such as intelligence, surveillance, consumer, biomedical, and web, providing a powerful tool to save search time and increase working efficacy.

TAG system reduces burden of manual labeling while quickly allowing users to accurately and efficiently sort multimedia

While traditional approaches based on automatic image classification usually require a sufficiently large number of labeled samples to train classifiers (i.e. supervised learning), the TAG system minimizes the burden of manual labeling on users. The method is able to start with a small number of user-provided labels and propagate them to the remaining data to generate optimal ways of ranking, classifying, and presenting the multimedia data sets. The objective is to optimally leverage the available user input (as few as one or two samples per class) and propagate such information in the most effective way to all the remaining data in the collection.

Lead Inventor:

Shih-Fu Chang, Ph.D.


  • Art collections
  • Photograph archives
  • Retail catalogs
  • Medical diagnosis
  • Crime prevention
  • Home security
  • Intellectual property
  • Architectural and engineering design
  • Geographical information and remote sensing systems


  • Minimizes the burden of manual labeling on users
  • Refined labels are generated in real-time or near-real time
  • Allows users to accurately and efficiently browse large multimedia databases

Patent information:

Patent Pending (WO/2010/075408) Patent Pending (WO/2009/155281)

Tech Ventures Reference: IR M10-016

Related Publications: