Vision-language-action (VLA) models learn to connect what they see, what they are told and what they do. Human video can supply the first two in large quantities, but only if it is cut and described with real discipline. This is the process we use on a live VLA program annotating a 300,000-hour corpus of egocentric video.
1. Define the unit of work
Everything follows from one decision: what counts as a single record. Ours is simple. One clip holds one action, performed by one hand or by both hands working together, and no clip is longer than ten seconds. Fix this before anyone labels anything, because every later stage depends on it.
2. Write the rule set before you label
Reviewers should check work against a published standard, not their own taste. These are the core segmentation rules we hold every annotator to:
- One clip, one action. Two independent actions by the same hand never share a clip.
- The operating hand is visible from the wrist up in both the start and the end frame.
- The hand appears in at least 90% of the clip and is never out of view for a second or more.
- No clip runs past ten seconds. Long repeated actions are split at natural pauses.
- No valid action in the video is skipped.
Real footage breaks clean rules, so the rule set also needs written exceptions. Actions shorter than half a second get padded to 0.5 seconds or merged. Identical objects moved continuously within two seconds can share a clip. A hand briefly hidden in a sink or cabinet keeps the clip intact if the intent is clear. And the end frame is cut before any large head movement that follows the action.
3. Segment first, describe second
A bad boundary silently corrupts the description written on top of it. We review every boundary twice before any description work starts. Segmentation is done in a timeline tool where annotators scrub the source, set boundaries frame by frame and loop each segment to confirm the cut. Boundary accuracy at 0.5-second resolution cannot be judged by watching at normal speed.
4. Describe each hand separately
"Washes the bowl" is not enough supervision. A policy has to resolve which bowl, with which tool and where. And when one hand steadies while the other acts, one sentence for both throws away the structure a bimanual model needs. So each hand gets its own object-grounded description:
5. Add structured attributes
Free text is not enough on its own. Two structured fields make the data far easier to train on. Camera motion (small or large) lets you filter or down-weight clips where apparent object movement is really head movement. Scene, chosen from a controlled vocabulary such as sink, counter or shelf, gives you a clean conditioning signal instead of thousands of unnormalised strings.
6. Review in two layers
Both segmentation and description go through two reviews. The internal review is attributed: the reviewer knows whose work it is and sends feedback and coaching back to that annotator. The external review is run by a separate team, blind to who did the work, and it judges the batch rather than the person. An external rejection sends a large part of the batch back for re-work, not just the failing clip. That cost is what keeps internal review honest. Every record therefore passes four human reviews before delivery.
7. Keep every correction
When a reviewer rewrites a description, keep both versions: the corrected text and the original beneath it. Preserved originals turn the review queue into a labelled record of annotator error, which you can use for per-annotator quality scores, targeted retraining and sharper guidelines. Once a clip is approved, lock it.
The record you end up with
If you want to see this on your own footage, we run pilots on about two hours of video. Read more about our robotics training data service.