Egocentric data is video recorded from the point of view of the person doing the work, usually with a camera mounted on the head. Instead of watching someone wash a bowl from across the room, you see the sink, the bowl and both hands exactly as the person saw them.

For robotics, that viewpoint matters. A robot with a head or wrist camera sees the world from roughly the same place a person does when working with their hands. First-person footage of people doing real tasks is one of the most direct ways to show a model what manipulation looks like from that position.

Why robotics teams use it

  • The hands are the subject. Hand-object contact, grip changes and tool use sit in the centre of the frame, where a third-person camera often loses them behind the body.
  • It shows real conditions. Clutter, poor lighting, worn equipment and interruptions are part of the footage, not cleaned out of it as they would be in a lab demonstration.
  • It scales with ordinary work. People already wash dishes, stack paper and fold laundry all day. Recording that work produces far more variety than staging each demonstration.
  • It covers many environments. The same capture setup works in a kitchen, a print shop or a warehouse, so a dataset can grow across settings without new hardware.

Raw footage is not training data yet

An hour of head-mounted video is a continuous stream. A model needs it broken into meaningful units and labelled. In practice that means three layers of annotation:

  1. Temporal segmentation. The video is cut into short clips, each containing one atomic action such as picking up a sponge or rinsing a plate.
  2. Language descriptions. Each clip gets a natural-language description of what happens, ideally for each hand separately and grounded in the specific objects involved.
  3. Structured attributes. Fields such as which hand acts, how much the camera moved and what kind of scene it is, so clips can be filtered and weighted during training.

We describe how we run each of these stages, and the rules behind them, in how to build a VLA training dataset.

What makes egocentric video hard to label

  • Actions are short and dense. A single minute of washing dishes can contain around twenty distinct actions.
  • Two hands do different things. One hand often holds or steadies while the other acts. Collapsing both into one sentence loses the structure a bimanual policy needs.
  • The camera moves with the head. Apparent object motion is often just the person looking around, which can mislead a model if it is not flagged.
  • Hands leave the frame. They disappear into a sink, a cabinet or out of view, and the annotator has to decide whether the action is still clear.

How to judge an egocentric dataset

Whether you are buying footage, commissioning collection or evaluating a public dataset, these questions separate usable data from volume:

  • Does the camera position, resolution, frame rate and field of view match what your robot will see?
  • How many distinct environments and tasks does it cover, and are they the ones your robot will work in?
  • Is it segmented into single actions with consistent boundaries, or delivered as long unlabelled video?
  • Are descriptions specific enough to identify the object, the tool and what each hand is doing?
  • Who reviewed the labels, how many times, and is there a written rule set you can read?
  • What rights do you get to use, retrain on and redistribute the data?

Where Wortel fits

We have annotated more than 30,000 hours of egocentric video for robotics, and we now collect first-person footage ourselves at real workplaces and homes in Pakistan. You can see sample clips from our first sites on the data collection page, or read how we deliver robotics training data.