What Is Data Labeling? How AI Annotation Actually Works

What data labeling actually is

Data labeling is the step where raw material, a photo, a sentence, an audio clip, a pair of chatbot replies, gets a tag that tells a model what it's looking at. AWS defines the process plainly as identifying raw data and adding one or more meaningful, informative labels to provide context so a machine learning model can learn from it. ^[4] The output of that process has a specific name in the field.

Most of the AI systems doing real work today, spam filters, self-driving perception stacks, medical-image triage, chat assistants tuned to be helpful rather than merely fluent, are trained with supervised learning, a paradigm that needs input-output pairs to learn from. A folder of a million unlabeled street photos teaches a detection model nothing about where the pedestrians are; the same million photos with a box around every person, car, and cyclist is a dataset the model can learn a task from. Raw, unlabeled text can teach a language model the statistics of language, but it can't teach the model which of two replies a person would prefer, or which scan shows a tumor. That judgment only exists once a human, or a system trained on prior human judgment, writes it down as a label.

The six annotation task types that cover most labeling work

Annotation work splits into a manageable number of task types, and each one calls for a different tool, skill level, and price. Here's the map.

Vision annotation is a useful illustration of how cost scales with content rather than item count. When Microsoft's research team built the COCO dataset, still a standard benchmark for object detection, they used crowd-worker interfaces built specifically for category detection, instance spotting, and segmentation to produce 2.5 million labeled object instances across 328,000 images. ^[6] That works out to roughly 7-8 labeled objects per photo on average, and every one of those objects is a separate unit of annotation work, which is why a busy street scene costs far more to label than a single product shot on a white background, even though a spec sheet would call both of them "one image."

Preference ranking deserves a full explanation of its own, covering how comparison pairs are structured, why the data is so scarce, and how alignment quality depends on it; that's the subject of Dayda's dedicated guide to RLHF and preference data. Here, it's enough to note that it's the most labor-intensive task on this list per item, because a rater has to read and judge complete responses rather than tag a word or draw a box. ^[2]

Red-teaming is annotation aimed at breaking a model rather than describing it. Anthropic researchers ran large-scale, crowdworker-driven red-teaming exercises against language models at different sizes and safety-training levels, and released a dataset of 38,961 human-written red-team attacks along with the harm labels attached to them. ^[4] Each attack is itself an annotated example: a person tried to elicit a harmful response, and the outcome was labeled well enough to become training or evaluation data for a safety classifier.

Who actually does the labeling

Almost every labeling program pulls from some mix of four sources, and AWS's own SageMaker documentation happens to describe three of them cleanly, because they map directly onto the workforce types its labeling product lets a buyer choose from: the Amazon Mechanical Turk workforce (open, on-demand crowdsourcing), a vendor-managed workforce (a BPO or specialized labeling company), and a private workforce (an in-house team). ^[8] A fourth source, domain experts, sits outside that framework entirely, because it's recruited for judgment rather than throughput.

• In-house teams. Employees or contractors labeling a company's own data, with full product context and no handoff friction, but limited throughput and a headcount cost that competes with everything else the team wants to build.
• BPO and vendor-managed workforces. Specialized labeling companies, what AWS's workforce documentation calls the vendor-managed path, run trained labeler pools at scale and bill by the hour or the unit. ^[4] This is where most commodity labeling happens: Grand View Research estimates the outsourced segment held an 84.6% share of the $18.63 billion global data labeling market in 2024, dwarfing in-house labeling. ^[8]
• Crowdsourcing platforms. Open marketplaces like Amazon Mechanical Turk let a buyer post small, well-specified tasks to a large, anonymous worker pool. ^[4] It's the cheapest way to get simple, high-volume labels fast, and the weakest fit for anything requiring specialized judgment or consistent guideline interpretation across thousands of strangers.
• Domain experts. For clinical, legal, financial, or highly technical labeling, generalist labelers can't reliably make the judgment calls the task requires. Specialist networks recruit doctors, lawyers, and engineers specifically to label or rank output in their field, priced closer to a professional hourly rate than a commodity labeling rate.

How label quality is actually measured

Labels are only as trustworthy as the process that checks them, and serious labeling operations don't take a single annotator's word for it. Three practices carry most of the weight.

Inter-annotator agreement

The core check is inter-annotator agreement (IAA): have two or more people label the same item independently, then measure how often they agree, corrected for the agreement expected from pure chance. Cohen's kappa handles two raters. Krippendorff's alpha generalizes to any number of raters, tolerates missing labels, and works across nominal, ordinal, interval, and ratio data, which is why it has become a standard tool for real-world annotation pipelines rather than a two-rater classroom example. ^[6] A low alpha isn't just a statistic. It's a diagnostic: it tells a labeling manager that the guidelines are ambiguous, an annotator needs retraining, or the task itself is more subjective than the taxonomy assumed. ^[8]

Gold-standard sets

The second practice is seeding the labeling stream with gold-standard items, examples whose correct label is already known, mixed in without telling the annotator which items they are. An annotator who misses a gold item is failing a case where the right answer is settled, which is a cleaner quality signal than disagreement between two annotators who might both be wrong in the same way.

QA sampling and spot audits

The third practice is ongoing QA sampling: a smaller, senior reviewer pool re-checks a percentage of completed labels on a rolling basis, rather than reviewing everything, which would defeat the point of outsourcing volume in the first place. Buyers evaluating a labeled dataset should ask for the IAA scores, the gold-standard pass rate, and the QA sampling rate as a matter of course. A vendor who can't produce any of the three is asking a buyer to trust the labels on faith.

A worked example: labeling 10,000 images for object detection

Take a concrete case: a retail-vision startup needs 10,000 in-store photos labeled with bounding boxes across five classes (person, shopping cart, empty shelf, price tag, checkout lane) to train an object-detection model. Here's what that project actually involves.

• 1Fix the taxonomy first. Five classes sounds simple, but the team still has to write a guideline that resolves edge cases: does a cart partially hidden behind a person get a box? Does a shelf that's 80% empty count as "empty"? Skipping this step is the single most common reason labeled data comes back inconsistent.
• 2Estimate the real unit of work. A busy retail photo typically contains several objects, not one. At a planning assumption of roughly 5 objects per photo, 10,000 images means about 50,000 individual bounding boxes, and cost and time scale with that number, not the image count. ^[6]
• 3Route the work to a workforce that matches the judgment required. Bounding boxes on visually obvious, well-defined objects are a good fit for a commodity BPO vendor rather than domain experts. A 2023 investigation into an OpenAI-Sama labeling contract documented a vendor bill rate of $12.50 an hour for annotation-style work, six to nine times what the Kenya-based labelers doing the work took home; that figure remains a useful, if dated, anchor for what a buyer pays a BPO vendor per labeler-hour on straightforward annotation. ^[6]
• 4Build the QA layer in before work starts, not after. Hold out 3-5% of images as a gold-standard set with pre-verified boxes, and route a rolling sample of completed work to a senior reviewer for spot audits, rather than discovering quality problems only after the full batch is delivered.
• 5Check agreement before trusting the batch. Send a subset of the same images to two independent annotators and compute Krippendorff's alpha on the results. ^[4] A low score means fixing the guideline and retraining annotators before the remaining images get labeled with the same mistake baked in tens of thousands of times over.
• 6Total the real cost. At a planning throughput of roughly 40-60 boxes labeled per hour, depending on object density and image clarity, 50,000 boxes runs somewhere in the 850-1,250 labeler-hour range; at a $12.50/hour vendor bill rate, that's roughly $10,600-$15,600 in raw labeling cost, before QA review time, project management, and tooling. ^[6]

Why annotation is one of the largest real costs of building a model

Labeling routinely shows up as one of the largest line items in an AI project's budget. Three forces explain why, independent of the specific dollar figures for any given task; a companion Dayda guide breaks those figures down channel by channel.

• Task complexity compounds, it doesn't add. A bounding box on an obvious object takes seconds; the same box on a partially occluded object in a cluttered scene takes much longer, and a full RLHF preference judgment on a long, nuanced response takes longer still. Cost stops scaling linearly with item count the moment real-world messiness enters the picture.
• Expertise is scarce exactly where it matters most. The domains with the most valuable AI use cases, clinical, legal, financial, are also the domains where a generalist labeler's judgment isn't good enough, which forces buyers into a much smaller, much more expensive labor pool.
• Quality control consumes as much time as a second labeling pass. Gold-standard sets, inter-annotator agreement checks, and QA sampling all take labeler and reviewer hours on top of the primary work. Skipping them doesn't save money; it moves the cost downstream to a model trained on bad labels.
• Guidelines get rewritten mid-project, and relabeling is real work. Almost no annotation guideline survives first contact with real data unchanged. Edge cases surface, categories get split or merged, and every revision risks invalidating labels completed under the old rules.

An honest budget for a labeling project accounts for judgment difficulty and QA overhead, not just a per-item rate multiplied by item count. Few projects hit every one of these cost drivers at full force, but most hit at least two.

The shift to AI-assisted, human-in-the-loop labeling

The clearest change in how labeling gets done is that fewer labels are produced by a human starting from a blank canvas. Active learning and pre-labeling, where a model produces an initial label and a human reviews or corrects it rather than creating one from scratch, have moved from research technique to standard product feature.

Amazon's own Ground Truth service is a clear illustration of how this works in practice. Its automated-labeling feature trains a model on a sample of human-labeled data, then uses that model to label the rest of the dataset, sending back to human workers only the items where the model's confidence falls below a set accuracy bar: for bounding boxes, a mean intersection-over-union of 0.6; for semantic segmentation, 0.7. ^[6] The process runs in rounds, labeling a batch, retraining, raising the auto-labeled share, and routing only the hardest remaining items to people. Notably, AWS has since stopped accepting new customers for Ground Truth itself while continuing to support existing ones, even as the active-learning pattern it helped popularize has become close to a default feature across modern labeling platforms. ^[8]

This shift compresses cost and turnaround time for well-defined, high-volume tasks like classification and simple bounding boxes, the same task types AWS's own automated-labeling feature supports today. ^[2] It does far less for the task types that are hardest to automate in the first place: preference ranking, red-teaming, and any domain-expert judgment call, because the entire reason those tasks need a human is that the judgment can't yet be delegated to a model at acceptable accuracy. That gap is the honest state of AI-assisted labeling today: real cost compression at the easy end, and just as much human effort as ever at the hard end.

The model is only as good as its labels

Every claim about a model's capability rests, somewhere upstream, on a labeling decision someone made. A classifier is only as accurate as its training tags. A detection model only sees what its bounding boxes taught it to see. An aligned chat model reflects the preferences humans expressed to it during training. Understanding labeling, the task types, who does it, how it's checked, and what it costs, means understanding one of the few parts of the AI stack that still runs on human judgment and shows no sign of fully automating away.

For companies sitting on real, human-labeled interactions, support resolutions, expert reviews, moderation decisions, that labeling work has often already been done as a byproduct of running the business. That's a different kind of asset than raw, unlabeled data, and it's usually the more valuable one.