How to Measure Data Labeling Quality: Inter-Annotator Agreement Explained

Why raw percent agreement isn't a real quality metric

A team ships a labeling job, pulls a sample, and finds two annotators agreed on 90% of items. That number feels like a passing grade, and it's the number most vendors lead with. It's also close to meaningless on its own, because it never asks how much agreement would have happened anyway if both annotators were labeling at random.

Jacob Cohen made exactly this point in 1960, using a psychiatric diagnosis example: two clinicians independently classifying patients as schizophrenic or not, each doing so purely by chance, would still land on roughly 82% joint "agreement" simply from the arithmetic of overlapping marginal probabilities. ^[2] A dataset with heavily skewed classes, say 90% of items are clearly "not toxic", inflates raw agreement automatically, because two annotators who default to the majority class whenever they're unsure will agree often without actually applying any real judgment.

This is what the field's chance-corrected agreement statistics exist to fix. Three of them cover essentially every practical situation a labeling project runs into: Cohen's kappa for exactly two annotators, Fleiss' kappa for three or more, and Krippendorff's alpha for anything with missing data or a mix of measurement types. Each has a real formula, a real worked calculation, and a real convention for what the resulting number means. What makes training data "high quality"↗ covers agreement as one line in a nine-dimension checklist; this guide is the full mechanics behind that line.

Cohen's kappa: agreement between two annotators

Cohen's kappa is defined as: κ = (Po − Pe) / (1 − Pe), where Po is the observed proportion of items on which the two annotators agree, and Pe is the proportion of agreement expected by chance, computed from each annotator's own marginal rate of using each label. ^[4] The denominator, 1 − Pe, represents the maximum possible improvement over chance; kappa expresses how much of that available headroom the annotators actually achieved.

Here's a fully worked example. Two annotators label the same 20 items for a content-moderation task as either toxic or not toxic. Their labels agree on 18 of the 20 items and disagree on 2:

• 1Observed agreement (Po). The two annotators agree on 7 + 11 = 18 of 20 items, so Po = 18/20 = 0.90.
• 2Chance agreement (Pe). Annotator 1 called 8/20 = 0.40 of items toxic and 12/20 = 0.60 not toxic. Annotator 2 called 8/20 = 0.40 toxic and 12/20 = 0.60 not toxic. If both were labeling independently at those same rates, they'd agree on "toxic" by chance 0.40 × 0.40 = 0.16 of the time, and on "not toxic" by chance 0.60 × 0.60 = 0.36 of the time, giving Pe = 0.16 + 0.36 = 0.52.
• 3Kappa. κ = (0.90 − 0.52) / (1 − 0.52) = 0.38 / 0.48 = 0.79.

Fleiss' kappa: agreement across three or more annotators

Cohen's kappa only works for exactly two annotators labeling the same items. Most real labeling projects run three or more annotators per item specifically to enable majority voting, and Fleiss' kappa extends the same chance-correction logic to that setting. ^[2] The formula: κ = (P̄ − P̄e) / (1 − P̄e), where P̄ is the mean per-item agreement across all items and P̄e is the expected agreement based on how often each category was used overall.

For each item with n raters and k categories, per-item agreement is: Pᵢ = [Σⱼ nᵢⱼ(nᵢⱼ − 1)] / [n(n − 1)], where nᵢⱼ is the number of raters who assigned item i to category j. P̄ is just the average of Pᵢ across all items. P̄e is computed from pⱼ, the overall proportion of all annotator-item assignments that fell into category j, as P̄e = Σⱼ pⱼ². ^[6]

Extend the same 20-item task with a third annotator, one who is a little less consistent, and get this per-item breakdown: 15 of the 20 items are unanimous across all three annotators (11 unanimously "not toxic," 4 unanimously "toxic"), and 5 items split 2-1.

• 1Category totals. Across 20 items × 3 raters = 60 total assignments, "toxic" was assigned 23 times and "not toxic" 37 times, giving pₜₒₓᵢc = 23/60 = 0.383 and pₙₒₜ = 37/60 = 0.617.
• 2Expected agreement. P̄e = 0.383² + 0.617² = 0.147 + 0.381 = 0.527.
• 3Observed agreement. Averaging Pᵢ = [nᵢⱼ(nᵢⱼ−1) summed over categories] / [3×2] across all 20 items (each unanimous item contributes Pᵢ = 1, each 2-1 split contributes Pᵢ = 1/3) gives P̄ = 0.833.
• 4Fleiss' kappa. κ = (0.833 − 0.527) / (1 − 0.527) = 0.306 / 0.473 = 0.65.

Krippendorff's alpha, and what these numbers mean in practice

Neither kappa handles two situations that come up constantly in real annotation pipelines: not every annotator labels every item (common with large annotator pools and item-level routing), and some tasks mix nominal labels with ordinal ratings or continuous scores in the same reliability study. Krippendorff's alpha, developed originally for content analysis, is built to cover both. ^[2]

Alpha's general form is α = 1 − (Do / De), where Do is the observed disagreement among values assigned to units and De is the disagreement expected if labels were assigned by chance given the overall distribution of values. ^[4] When annotators agree perfectly, Do = 0 and α = 1. When agreement is no better than chance, Do = De and α = 0. Krippendorff's own summary of what makes alpha distinct from kappa-family statistics is direct about the scope:

The practical rule of thumb: use Cohen's kappa for a clean two-annotator comparison, Fleiss' kappa for three or more annotators with complete, purely nominal data, and Krippendorff's alpha the moment data goes missing, annotator counts vary item to item, or the label type isn't strictly nominal. Krippendorff's own convention, echoed in follow-up work with Andrew Hayes, treats alpha ≥ 0.800 as the bar for reliable data and alpha ≥ 0.667 as the lowest defensible floor for tentative conclusions; anything below that is conventionally discarded rather than acted on. ^[6]

Reading the number: the Landis and Koch bands

A raw kappa or alpha number is only useful against a shared interpretation scale, and the field's default reference point is Landis and Koch's 1977 paper, which proposed the bands most annotation teams still cite today. ^[2]

These bands are a convention, not a law of statistics, and Landis and Koch themselves proposed them as "reasonable" rather than mathematically derived. ^[2] A widely used clinical-research review of the kappa statistic reproduces the same table and separately argues that healthcare applications should demand a stricter floor, treating anything below 0.60 as inadequate given the cost of a wrong medical label. ^[4] The right floor for a labeling project is a function of what a wrong label costs downstream, not a single universal number.

Gold-standard and honeypot task design

Inter-annotator agreement is a batch-level diagnostic: it tells a team something is wrong after a set of items has already been labeled by multiple people. Gold-standard tasks, also called honeypots or test questions, catch problems while the work is still happening, by seeding pre-answered items into the live queue that look identical to real work.

Appen's implementation is a useful concrete example of how this works in production. Job managers create test questions, items with a known correct answer, and the platform blends them into two modes: Quiz Mode, where a contributor must clear an accuracy threshold on test questions before being allowed to start real work, and Work Mode, where test questions continue to appear randomly mixed into the live queue throughout the job. ^[8] If a contributor's accuracy on these hidden questions drops below a set threshold mid-task, the job manager is notified and can remove that contributor immediately, rather than discovering the problem only after a full batch has already been delivered and paid for.

That last point matters: the same test questions that gate individual annotators in real time also generate the ground-truth comparisons needed to compute agreement statistics against a known answer, rather than only annotator-to-annotator. Amazon SageMaker Ground Truth, while it built its core reliability mechanism around multi-worker annotation consolidation rather than a named "honeypot" feature, uses the same underlying idea: known-answer items checked against worker submissions to estimate worker accuracy going into its consolidation algorithm. ^[2]

Consensus and adjudication: turning multiple labels into one

Once multiple annotators have labeled the same item, something has to reduce those labels to the one that ships. Three approaches cover almost every real workflow, in increasing order of cost and rigor.

• Majority vote. The label most annotators chose wins, ties broken by a rule set in advance (escalate, default to the harder category, or re-annotate). Cheap, fast, and reasonable for low-stakes, high-volume, subjective-but-bounded tasks like sentiment or topic tagging.
• Weighted voting by historical accuracy. Instead of counting every annotator equally, weight each vote by that annotator's estimated reliability, learned from their performance on gold-standard items and their agreement history. This is the approach formalized by Dawid and Skene in 1979: jointly estimate each observer's error rate and the most likely true label using an EM algorithm, without needing ground truth for every item up front. ^[4] Amazon SageMaker Ground Truth's default multi-class consolidation is a direct descendant of this idea, using a variant of Expectation-Maximization to weight each worker's annotations by an estimated per-worker parameter rather than a flat vote. ^[6]
• Expert adjudication. A qualified reviewer, not a crowd vote, makes the final call on items where annotators disagree, or on the entire task from the start. This is the only defensible approach once mislabeling has real consequences: medical, legal, or safety-critical labeling needs a domain expert's judgment, not the statistical opinion of a general crowd, however well its votes are weighted.

The choice among these three should track task difficulty and stakes, not habit. A support-ticket topic tagger and a radiology-report labeling project should not use the same consensus mechanism, even if both technically involve "multiple annotators labeling the same items."

Comparing all four QA mechanisms

None of the methods covered so far is a complete quality system alone. Each catches a different failure mode and misses others, which is why serious labeling operations run them together rather than picking one.

Why quality actually breaks: fatigue, drift, and ambiguous guidelines

Low agreement gets read as an annotator problem by default, and it's often not. Three failure modes account for most real degradation, and only one of them is really about the people doing the labeling.

Fatigue and drift over long sessions

Annotators labeling the same category boundary for hours make progressively more inconsistent judgment calls near the edges, even without any change in effort or attention. This shows up as agreement that's fine early in a session and degrades later in it. The fix is structural: cap session length, rotate task types, and check whether agreement scores correlate with time-on-task before assuming a specific annotator is the problem.

Guideline ambiguity, the most common root cause

Aroyo and Welty's 2015 analysis of crowd annotation practice names this directly as one of several flawed assumptions the field has historically made: that disagreement among annotators signals an error to be corrected, rather than a legitimate reflection of genuine ambiguity or reasonable differences in interpretation. ^[2] A task asking annotators to mark content as "toxic" without defining whether sarcasm, reclaimed slurs, or political criticism count will produce persistent disagreement regardless of who is doing the labeling, because the category itself hasn't been pinned down.

The practical test: pull the specific items driving a low agreement score and read them. If disagreement clusters tightly around one recognizable edge case (sarcastic tone, borderline medical claims, ambiguous entity boundaries), that's a guideline problem, fixable by rewriting the instructions and adding examples for exactly that edge case. If disagreement is spread randomly across otherwise clear-cut items with no discernible pattern, that's more likely a specific annotator's competence or attention.

Distinguishing ambiguous tasks from bad annotators

• Same annotator, consistent pattern across items → likely a genuine competence or attention issue; check their gold-standard accuracy trend over time.
• Same item type, disagreement across many different annotators → almost always a guideline gap, not a people problem.
• High agreement on gold-standard items but low agreement on real items → the annotators understand clear-cut cases fine; the live data contains genuinely harder or more ambiguous cases than the gold set represents.
• Agreement that degrades over the course of a session → fatigue or drift, not a fixed skill gap.

Measuring quality is a system, not a single number

No single statistic proves a labeling project is trustworthy. Cohen's kappa and Fleiss' kappa quantify how much annotators agree beyond chance on nominal data with two or more raters. Krippendorff's alpha extends the same logic to missing data and mixed measurement types. Gold-standard tasks catch individual annotator failures in real time rather than after delivery. Consensus and adjudication workflows, matched to the task's actual stakes, turn multiple raw labels into the one that ships. None of these substitutes for the others, and the comparison table above exists precisely because each one has a specific blind spot the others cover.

The single most useful habit this guide can leave a team with: when agreement drops, read the disagreements before reassigning blame. A kappa of 0.55 concentrated on one ambiguous edge case is a guideline problem with a cheap fix. The same 0.55 spread randomly across clear-cut items is a different problem entirely, and the two require completely different responses. What makes training data "high quality"↗ covers where labeling accuracy fits among the other dimensions that determine whether a dataset is actually usable.