What Is AI Training Data? A Complete Guide

What is AI training data, exactly?

Training data is the raw material a model learns from: the sentences, images, sound waves, video frames, spreadsheet rows, and source code files it's shown during training, before any of it turns into a working model. The model itself is a different kind of object entirely, a fixed set of learned numeric weights, a compressed mathematical summary of the patterns the training data contained. Confusing the two is one of the most common mix-ups in AI coverage. A leaked model checkpoint and a leaked training dataset are not the same kind of asset, and they carry different legal, technical, and commercial consequences.

Every modality does the same underlying job, even though the examples look nothing alike. Feed a model enough well-chosen examples of a pattern, text that follows grammar and reasoning, images that map to object categories, audio that maps to words, and it learns to recognize or produce new examples it never saw during training. What changes across modalities is how the examples get collected, whether they need a human-added label, and how much data a model actually needs before that pattern shows up reliably. Where the data physically comes from, public web crawls, licensed archives, platform logs, brokered deals, is a separate question with its own answer in our map of the AI data supply chain↗. This guide stays on what training data actually is and how models use it once it's already been sourced.

The six data modalities models learn from

Six modalities account for nearly all commercially relevant AI training data. Each has developed its own conventions for scale, collection, and labeling, shaped almost entirely by how expensive it is to gather and annotate examples of that type.

Text still dominates raw token count because language is the cheapest modality to collect and the most useful for teaching general reasoning. Almost every frontier model released since 2023 is multimodal, though, combining several of these rows into one training run rather than picking a single one.

Labeled vs. unlabeled data

Labeled data pairs each example with the answer a model is supposed to produce: an image tagged with its object category, a support ticket tagged with its resolution, a sentence tagged with its sentiment. Unlabeled data is just the raw example with no attached answer; the model has to find useful structure in it on its own, usually by predicting a piece of the example from the rest of it, the next word, a masked pixel, a missing frame.

ImageNet is the clean textbook example of labeled data: its roughly 14.2 million images were sorted by human annotators into more than 21,000 WordNet categories, and that manual sorting is exactly what let early computer-vision models learn supervised object recognition ^[2]. LAION-5B sits in between the two categories: its 5.85 billion image-text pairs do carry a caption, but the caption is just whatever alt text a web author happened to write, never manually verified for accuracy, which is why practitioners usually call it weakly labeled rather than a clean supervised dataset ^[6]. Raw Common Crawl web text carries no labels at all; language models extract structure from it purely by predicting the next token, a self-supervised objective that requires no human annotation up front ^[8].

Pretraining data vs. fine-tuning data vs. evaluation data

A single trained model is usually the product of at least three separate datasets, each doing a different job, built at wildly different scales.

Pretraining data

Pretraining is where a model acquires its general competence in a modality, and it runs on the largest, least-curated dataset in the pipeline, usually unlabeled. Hoffmann et al.'s Chinchilla paper found that model size and training-token count should scale together under a fixed compute budget, and trained a 70-billion-parameter model compute-optimally on roughly 1.4 trillion tokens, about 20 tokens for every parameter, well past what most earlier large models had used ^[2]. Almost nobody outside a handful of AI labs collects a pretraining corpus from scratch. Most builders start from an already-pretrained open-weight model and skip straight to fine-tuning. That's partly a matter of resources, and partly because the supply of fresh pretraining-grade text isn't infinite: Epoch AI estimates only about 300 trillion usable tokens of high-quality, human-generated public text exist in total, and projects that current scaling trends could exhaust that stock sometime between 2026 and 2032 ^[4].

Fine-tuning data

Fine-tuning takes a pretrained model and narrows its behavior toward a specific task, domain, or style, using a dataset that is orders of magnitude smaller and, unlike most pretraining data, usually labeled or human-ranked. OpenAI's InstructGPT work is a documented example of what that looks like in practice: a team of about 40 contractors built a supervised fine-tuning set of roughly 13,000 prompts with human-written responses, plus a separate reward-model dataset of about 33,000 prompts where labelers ranked multiple model outputs against each other, the format generally called RLHF preference data ^[2]. Our dedicated guide covers how RLHF preference data is structured and priced↗, and our fine-tuning volume guide↗ covers exactly how many examples a given technique needs, so this piece won't repeat either in depth.

Evaluation (test) data

Evaluation data measures whether a model actually generalized or just memorized, and the measurement only works if that data was never seen during pretraining or fine-tuning. In practice, that guarantee is harder to keep than it sounds. Lee et al. found that language modeling datasets contain many near-duplicate examples, and that this duplication contaminates over 4% of the validation set of standard benchmarks they examined, which means some of what gets reported as model accuracy reflects memorized text rather than genuine generalization ^[2].

Structured vs. unstructured data

A second, independent axis cuts across all six modalities: whether the data arrives with a predefined schema. Structured data lives in fixed fields, rows and columns, key-value pairs, and is already machine-readable without any interpretation step. A CRM export with deal_stage, close_date, and amount columns is structured the moment it's exported. Unstructured data has no such schema. Free text, images, audio, and video are unstructured by default, because a sentence or a photograph doesn't arrive pre-sorted into fields a database can query.

The practical consequence is what it costs to make the data trainable. Structured data is usually ready to feed into a model after light cleaning. Unstructured data almost always needs an extraction or labeling step first, running text through parsing or classification, running images through object detection, before a model can use it for a supervised task, which is the labeling work described in the previous section. A useful middle category, semi-structured data such as JSON documents, log lines, or HTML with markup, carries partial structure and typically needs the least additional work to become usable.

How much data do different kinds of models actually need?

There's no single number, because the answer depends entirely on which of the three stages above you're asking about. At a glance:

What any of this costs to collect or license is a separate question with its own moving parts: labeling labor, licensing fees, and compliance review. Our cost breakdown↗ covers that in full; this section is only about volume.

A worked example: the data behind a support chatbot

Here's how the distinctions above play out on an actual build, not an abstract one. A startup wants an AI assistant that drafts first-response replies to inbound customer support tickets.

• 1Pretraining (inherited, not built). The team starts from an open-weight LLM already pretrained on a corpus in the FineWeb/Common Crawl family, tens of trillions of unlabeled tokens of general web text ^[4][6]. They never touch this stage directly; it's already baked into the base model they license or download.
• 2Fine-tuning data. They pull 5,000 historical tickets, each labeled with the reply the human agent actually sent and whether the customer later confirmed the issue was resolved. That resolution label is what turns raw transcripts into a supervised fine-tuning set: small, labeled, and domain-specific, the opposite profile of the pretraining corpus above.
• 3Preference data (optional). To sharpen tone, a reviewer ranks pairs of draft replies for 300 tickets, marking which reply reads as more helpful and on-brand. It's the same ranked-comparison format InstructGPT's reward-model dataset used, just several orders of magnitude smaller ^[4], which is normal: a narrow, single-company use case needs nothing close to frontier-lab scale.
• 4Evaluation data. Before any training starts, the team sets aside a random 10% of tickets, about 550, and runs a near-duplicate check against the training 5,000 first, exactly the check that matters, since near-identical tickets (a customer re-sending the same complaint, or a templated response reused across cases) would otherwise leak into both sets and inflate reported accuracy ^[4]. Only after that check does the held-out set get used to measure the fine-tuned model's real resolution accuracy.

The takeaway

AI training data is not one thing. It's at minimum three differently sized, differently purposed datasets, pretraining, fine-tuning, and evaluation, pulled from six modalities that each have their own scale and labeling conventions, and split further by whether any given dataset is structured or not. Getting those distinctions right is what separates a rigorous data strategy, or a rigorous valuation of data you're thinking about selling, from guesswork.

If you're trying to work out where a specific dataset came from in the first place, our map of the AI data supply chain↗ picks up exactly where this guide leaves off.