The right annotation type depends on what your model has to output, not on what looks most detailed. More precise labels cost more time per image, so the aim is the simplest type that gives the model what it needs. These are the types we use most, with examples from our own projects.

Bounding boxes

A rectangle around each object, with a class label. Use boxes when the model needs to find and count things but does not need their exact outline. They are the fastest type to label and suit object detection, counting and tracking. Examples include corn stand counting in dense rows and waste object detection across fourteen classes inside crowded bins.

Polygon segmentation

An outline that follows the object's true edge, or a mask covering every pixel of a class. Use it when shape or area matters: measuring ground cover, separating crop from weed, or replacing one surface in a photo. Examples include ground cover segmentation, where every pixel of a field plot is classed as groundcover, weed, residue, crop or soil, and kitchen surface segmentation, where every cabinet face and worktop becomes its own editable region.

Key points

A single point per object, or a set of points on landmarks. Use key points when position is all you need, especially for very small or densely packed objects where a box adds nothing. Our tomato stand count marked one point per plant across drone orthomosaics to measure emergence and gaps along the rows.

Classification on top of detection

Many projects need a box and a grade: size, ripeness, health or growth stage. Examples include orange ripeness classification and cabbage size classification. The hard part is consistency, because grades like medium or ripe drift between annotators unless the spec defines the boundaries with examples.

Image-level tags

One or more labels for the whole image, with no location. Use tags when the question is about the image as a whole. Our bin fill level project estimated how full a waste bin was across more than ten thousand images.

Video segmentation and description

For video, the time axis is the label. Continuous footage is cut into clips at action boundaries, then each clip is described in language. This is the core of robotics and VLA training data, covered in detail in how to build a VLA training dataset.

How to choose

  • What does the model output: a location, an outline, a point, a grade or a description?
  • How small and how crowded are the objects? Tiny, packed objects often suit points better than boxes.
  • Does area or exact shape feed into a measurement or an edit? If so, you need segmentation.
  • Which edge cases will come up, such as occlusion, look-alike classes or partial objects, and how should each be labelled?

Keeping quality consistent

Whatever the type, quality comes from the spec more than the tool. Write down the classes, the edge cases and examples of each before labelling starts. Score work per annotator so problems are caught early, feed errors back as coaching, and review before delivery. Browse every project in our annotation portfolio, or see how we run data annotation services.