XR & VR Teleoperation

Becoming the robot

In XR teleoperation, a human operator puts on a virtual or augmented reality headset and picks up hand controllers, or uses hand tracking built into the headset itself. The headset streams video from cameras mounted on the robot's head directly to the operator's eyes, and as the operator turns their head, the robot's head and camera view turn with it in real time. The effect is a first-person presence inside the robot's body rather than watching it from the outside.

The operator's hand and arm movements are tracked by the headset's controllers or by camera-based hand tracking, and those movements are streamed to the robot's arms and grippers so they move in step. Instead of pressing buttons or nudging a joystick to command an abstract motion, the operator simply reaches, grabs, and turns their wrist, and the robot's arm does the same. This is what people mean when they say the operator becomes the robot for the duration of the session.