What this environment asks for
Home kitchens are the densest source of bimanual manipulation there is. A single minute of washing up holds around twenty distinct actions, performed by two hands doing different things, under a camera that moves with the operator's head.
Recordings are cut to frame-accurate boundaries, one action per clip, and each hand is described independently against the object it is acting on. Camera motion and scene are captured as structured attributes rather than left implicit in the text.
How the work runs
- Segmented continuous recordings into atomic-action clips against the published rule set.
- Enforced one action per clip, by one hand or by coordinated hands.
- Required the operating hand to be visible from the wrist up at both boundary frames.
- Described each hand separately, naming the target object with disambiguating attributes.
- Recorded camera motion as Small or Large so head movement can be weighted in training.
- Tagged scene from a controlled vocabulary rather than free text.
- Passed every record through two segmentation reviews and two description reviews.
What makes it hard
- One hand stabilises while the other acts, so collapsing both into one sentence destroys the structure a bimanual policy needs.
- Hands disappear into sinks, cupboards and behind pans for a second at a time.
- Repetitive actions such as stirring have no natural boundary and must be split at pauses.
- Head motion during a task looks like object motion in the frame.
- Cluttered worktops make object reference ambiguous without attributes.
- Steam, water and reflective surfaces obscure the operating hand.
Result
A training-ready set of atomic-action clips with per-hand language, camera motion and scene attributes, gated by four independent human reviews before delivery.
What comes out
Structured records carrying clip index, frame-accurate timecodes, hand enum, two independent descriptions, camera motion, scene and full review state with correction history.
