Human video is one of the richest sources of examples for robot learning. But where the camera sits changes what a model can learn from it. First-person video, also called egocentric, is recorded from the worker's own head. Third-person video, also called exocentric, is recorded from a fixed or handheld camera that watches the worker.

What first-person video captures well

  • Hands and objects up close. Grasps, tool use and contact points stay in view, because the camera points where the person is looking.
  • A viewpoint close to the robot's. Head and wrist cameras on robots see the workspace from a similar position, which narrows the gap between training data and deployment.
  • Attention. Where the person looks next is built into the footage. The camera turns towards the next object before the hand reaches it.
  • Natural work at scale. A camera on the head does not need setting up per task, so people can record whole shifts of ordinary work.

What third-person video captures well

  • Whole-body movement. Posture, walking and reaching across a room are visible, which matters for mobile and humanoid robots.
  • Spatial layout. A wide, stable view shows where things are relative to each other and to the person.
  • Several people at once. Collaboration and hand-offs between workers are easier to follow.
  • A steady frame. Without head movement, object motion in the frame is real object motion.

Where each falls short

First-person video moves constantly with the head, the body is mostly out of frame, and hands can drop out of view when they reach into a sink or behind an object. Third-person video often loses the hands behind the body, shows them too small to read a grip, and sees the scene from a position no robot camera will occupy.

Good annotation can offset some of the first-person weaknesses. We flag camera motion on every clip so training can filter out head movement, require the acting hand to be visible at the start and end of every clip, and cut clips before large head movements begin.

How to choose

  • If your robot's job is mostly manipulation at a bench, counter or machine, first-person footage is usually the closer match.
  • If it has to move around, work alongside people or plan whole-body motion, third-person views add information egocentric video cannot.
  • Many teams use both: first-person for the hands and the task, third-person for context and body movement.
  • Either way, the annotation matters as much as the camera. Unsegmented hours of either view teach a model far less than consistently labelled clips.

We collect and annotate first-person video at real workplaces and homes. See sample footage on our data collection page.