Description:
This technology enables humans to negotiate robot actions instantly through an intuitive augmented reality interface. By visualizing robot intentions and allowing direct constraint input, it removes ambiguity and accelerates collaboration. The result is faster, clearer, and more flexible human-robot teamwork in dynamic, real-world environments.
Background:
Human–robot teams struggle with ambiguous and low-bandwidth communication methods such as gestures, voice commands, and kinesthetic teaching, all of which suffer from occlusion, noise, processing delays, and high training burdens. Existing planners typically rely on static models and cannot incorporate live human preferences, forcing disruptive manual replanning. These limitations hinder smooth, dynamic, and scalable coordination between humans and autonomous robots, particularly in complex and time-sensitive environments.
Technology Overview:
The invention uses an augmented reality interface to display robots’ intended actions and trajectories as aligned virtual overlays, allowing users to impose constraints by interacting with objects in the scene. These interactions are converted into Answer Set Programming constraints through an Action Restrictor, which the ASP planner integrates to generate updated symbolic action sequences for multi-robot teams. A multi-threaded runtime executes and monitors plans while continuously updating and replanning based on AR-derived feedback, supporting real-time negotiation between humans and robots.
Advantages:
• Enables precise visualization of robot intent for low-ambiguity communication
• Delivers real-time, high-bandwidth interaction through AR overlays
• Reduces training burden by relying on intuitive AR gestures
• Supports dynamic replanning by injecting constraints directly into an ASP planner
• Coordinates multiple robots simultaneously through parallel plan execution
• Operates across common AR platforms without proprietary hardware requirements
• Improves safety by previewing potentially hazardous robot actions in AR
• Increases operational efficiency through reduced task delays and waiting time
Applications:
• Industrial and warehouse automation
• Construction task coordination and site maintenance
• Agriculture and resource extraction robotics
• Military and defense mission support
• Search and rescue operations
• Healthcare service robot coordination
• Human-human teaming (construction or search and rescue scenes)
Intellectual Property Summary:
• United States 11,958,183 Utility Filed 09/18/2020 Issue date 04/16/2024 Status Patented
Stage of Development:
Prototype
Licensing Status:
This technology is available for licensing.
Licensing Potential:
Strong potential for robotics companies, industrial automation providers, and AR platform developers seeking intuitive, real-time collaboration systems that enhance efficiency, safety, and scalability in human-robot teaming.
Additional Information:
Information available upon request.
Inventors:
Kishan Chandan, Shiqi Zhang
Alternate NCS Title: Real-Time AR Negotiation for Human-Robot Teaming