Columbia University

Technology Ventures

Augmented and Virtual Reality assistant for remote object manipulation

Technology #cu15254

Certain tele-assistant applications require a remote expert to direct and instruct a local user with object manipulation tasks. Effectively relaying object-oriented actions between users is often difficult due to the complex spatial orientations involved. This technology provides virtual representations of objects within a virtual reality platform. Users are able to visually demonstrate and communicate spatial maneuvers to others. With this technology, users are able to communicate spatial information to remote end-users effectively through a robust virtual reality platform.

Virtual replicas for remote object maneuvering in Virtual Reality setting

Existing solutions to remote object manipulation are unable to navigate complex spatial orientations and are limited when the object is momentarily visually obscured. This technology utilizes object tracking and dual object representation at each user location to create a robust platform for virtual representation. The remote user is able to maneuver a virtual replica of the object in virtual reality. The local user also views a virtual representation of the object within the local virtual environment. Object maneuvers at one location are spatially mimicked in virtual representations at the second location. As such, complex object manipulation instructions can be conveyed.

A prototype of this technology has been tested with users and has been preferred over other existing remote assistance tools.

Lead Inventor:

Steven Feiner, Ph.D.


  • Remote surgery
  • Distance learning for education and training
  • 3D enabled technical and customer support


  • Simplifies task of communicating complex spatial directions
  • Allows remote supervision of object manipulation
  • Provides robust object manipulation amidst obscured object visualization

Patent Information:

Patent Pending

Tech Ventures Reference: IR CU15254

Related Publications: