- 1AI training data is the raw text, images, audio, video, table rows, and code a model learns from, and a single model typically depends on three separate datasets built at different scales: pretraining, fine-tuning, and evaluation.
- 2There's no single 'how much data' number: few-shot prompting can work from 2-50 examples, LoRA/QLoRA fine-tuning typically needs 100-10,000, and genuine domain adaptation can require hundreds of thousands of documents, because the answer depends on technique and task narrowness, not a universal rule.
- 3Public, human-generated text is finite: Epoch AI estimates roughly 300 trillion usable tokens exist and projects models could fully consume that stock somewhere between 2026 and 2032, which is why proprietary and licensed data is rising in value.
- 4Synthetic data is a complement, not a substitute: Gartner projected synthetic data would make up the majority of AI/analytics training data by 2024, but researchers have documented 'model collapse,' a measurable quality loss when models train recursively on model-generated content.
- 5Whether scraped data is legal to train on has no blanket answer. U.S. courts apply a fact-specific fair-use test and the Copyright Office's own report on AI training was still an unfinalized draft as of mid-2025, while licensed or consented data is safe to train on under any theory.
This is a fast-answer FAQ, not a narrative guide: nine of the most common questions people search about AI training data, each answered directly in a few sentences so it stands on its own, with a link to the full deep dive for anyone who wants more. Training data is the raw material a model learns from, split across pretraining, fine-tuning, and evaluation sets; how much you need depends entirely on technique, from a couple dozen examples for few-shot prompting to hundreds of thousands of documents for real domain adaptation; it doesn't always need labels, since pretraining runs mostly unlabeled while fine-tuning and evaluation do; quality is a measurable checklist, not an adjective; scraped data carries real, unsettled legal risk that licensed data doesn't; it comes from six identifiable sources that are shifting from public to proprietary as the open web runs low; and synthetic data augments real data without replacing it. Read the answer that matters to you, or read all nine.
On this page ▾
- What is training data?
- What's the difference between a dataset and training data?
- What's the difference between training data, validation data, and test data?
- How much training data do you need to train or fine-tune a model?
- Does training data need to be labeled?
- What makes training data "high quality"?
- Can you use public or scraped data to train an AI model?
- Where does training data come from?
- Is synthetic data a substitute for real training data?
- One more question?
What is training data?
AI training data is the raw material a model learns from: the sentences, images, audio clips, video frames, table rows, and code files it's shown during training, before any of that becomes a working model. It's a different kind of object than the model itself, which ends up as a fixed set of learned numeric weights, a compressed summary of patterns in the data rather than a copy of it. Training data spans six main modalities, text, image, audio, video, tabular, and code, and a single trained model usually depends on at least three separate datasets built at very different scales: a large, mostly unlabeled pretraining corpus, a much smaller labeled fine-tuning set, and a held-out evaluation set.
- Text: sentences and documents, the cheapest modality to collect at scale
- Image: photos and scans, often paired with a caption or a category label
- Audio: speech and sound recordings, usually transcribed before training
- Video: sequences of frames, the most expensive modality to annotate
- Tabular: structured rows and columns pulled from databases and event logs
- Code: source files used to train code-generation and completion models
For the full breakdown of modalities, scale, and how pretraining, fine-tuning, and evaluation data differ and interact, see What Is AI Training Data? A Complete Guide↗.
What's the difference between a dataset and training data?
A dataset is the container: a bounded, documented collection of records with a defined scope, like "Q1 2026 support tickets" or a named public corpus. Training data is a functional label for whatever subset of one or more datasets gets used to train a model at a given stage. The same underlying dataset can supply training data for one project and validation or evaluation data for another; the records don't change, only the job they're doing changes. The distinction matters most when negotiating a license: a buyer licenses a dataset, a defined asset with agreed terms, and what happens to the rows inside it afterward, training, validation, or evaluation, is a downstream engineering decision the license terms may or may not restrict.
What's the difference between training data, validation data, and test data?
Training data is what the model learns from directly: the examples it sees while its internal parameters get adjusted. Validation data is a separate slice the model never trains on directly; it's checked mid-training to guide tuning decisions like when to stop or which hyperparameters to use, and reusing it too many times lets a model implicitly fit its quirks. Test data is a final, untouched slice held out until training and tuning are completely finished, used once to report how the model will likely perform on genuinely new inputs. Google's own machine learning course illustrates a common example split of roughly 70% training, 15% validation, and 15% test, while being explicit that there's no fixed percentage requirement, only that each set needs enough examples to do its job. [2]
| Dataset | What it's for | Does the model train on it directly? | Typical share |
|---|---|---|---|
| Training set | Adjusts the model's weights/parameters | Yes | ~60–80% |
| Validation set | Guides tuning decisions mid-training | No | ~10–20% |
| Test set | Reports final, unbiased performance, used once | No | ~10–20% |
How much training data do you need to train or fine-tune a model?
There's no single number. The honest answer depends on which technique you're using and how narrow the task is. Rough anchors: few-shot prompting can work from as few as 2 to 50 examples placed directly in the prompt, with no training run at all; LoRA or QLoRA fine-tuning typically lands somewhere between 100 and 10,000 examples; broader instruction-tuning ranges from around 500 to 50,000-plus, and Meta's LIMA study reached strong, GPT-4-competitive results with just 1,000 carefully curated examples. [2] Preference tuning (RLHF or DPO) usually needs thousands to tens of thousands of labeled comparison pairs, while genuine domain adaptation, teaching a model new facts or vocabulary rather than a new format, can require hundreds of thousands of documents or more.
- 1Style or format task (a house tone, a fixed routing taxonomy, a response template): the base model already knows the underlying content, so a LoRA-scale set, roughly a few hundred to a few thousand well-labeled examples, is usually enough.
- 2Domain adaptation (new facts, vocabulary, or reasoning a general model has never seen, like a narrow area of law or clinical coding): expect the realistic floor to be hundreds of thousands of documents, because you're teaching new information, not a new shape.
- 3Preference alignment (teaching the model which of two outputs is better): budget thousands to tens of thousands of labeled comparison pairs, since the goal is capturing human judgment, not adding facts.
“Almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.”
That's the compressed version. The full fine-tuning volume guide↗ walks through the research behind each range and gives a five-step framework for estimating your own number instead of borrowing someone else's.
Does training data need to be labeled?
Whether training data needs labels depends on which stage of training you're feeding it into. Pretraining, where a model builds general competence in a modality, usually runs on unlabeled data: the model predicts a missing piece of each example, the next word, a masked pixel, with no human-added answer attached. Fine-tuning and evaluation generally do need labels: a resolution outcome on a support ticket, a human-ranked pair of responses, a diagnosis code on a clinical note, because that's the signal that teaches or measures a specific behavior. AWS defines data labeling as adding meaningful tags to raw data so a model can learn from it, and calls a properly labeled dataset the ground truth a model is trained and judged against. [4] Most of the raw token count behind large language models is unlabeled text; labeling gets added later, in far smaller quantities.
“A properly labeled dataset that you use as the objective standard to train and assess a given model is often called "ground truth." The accuracy of your trained model will depend on the accuracy of your ground truth.”
For the task types, workforces, and quality checks behind real labeling pipelines, see What Is Data Labeling? How AI Annotation Actually Works↗.
What makes training data "high quality"?
"High quality" means a checkable list of measurable properties: accuracy of labels and facts, diversity of coverage, low duplication, relevance to the target task, balanced representation across categories, freshness, and the absence of toxic content or exposed personal information. Meta's LIMA result is the clearest evidence that curation can outweigh raw volume: a 65-billion-parameter model fine-tuned on just 1,000 hand-curated examples tied or beat GPT-4 in 43% of head-to-head human comparisons. [2] A listing that leads with row count or token count and says nothing about labeling, deduplication, or provenance hasn't told a buyer anything about most of what actually determines whether the data helps.
- Accuracy: do the labels and facts actually match ground truth
- Diversity: does it cover the real range of inputs, not just the easy, common cases
- Deduplication: how much of it is exact or near-duplicate content
- Relevance: how closely each example matches the target task, versus generic filler
- Balance: are categories represented in proportion to the real use case, not just whatever was easiest to collect
- Freshness: how current the facts, prices, or terminology are
- Clean of PII and toxic content: is it safe to train on and ship
For all nine dimensions, how each is measured, and a scoring framework you can run before committing budget to a dataset, see What Makes AI Training Data "High Quality"? The Metrics That Matter↗.
Can you use public or scraped data to train an AI model?
It depends on jurisdiction, licensing, and the specific facts of how the data was acquired. There's no single blanket answer. In the United States, courts apply a fact-specific fair-use test, and multiple 2025 rulings on AI training reached different outcomes depending on exactly how the data was acquired and used, not just how it was later applied. The U.S. Copyright Office has been examining the question directly through a multi-part report on AI and copyright policy; its section on generative AI training was released only as a pre-publication draft in May 2025 and remained unfinalized as of that date, itself a sign the legal picture is genuinely unsettled rather than quietly settled in scrapers' favor. [2] What isn't ambiguous: data you hold a license or documented consent for is safe to train on under any theory, while scraped copyrighted data carries real, case-specific legal risk that no amount of technical cleverness removes.
- Documented consent or a signed license, not just the fact that a page was publicly viewable
- A clear chain of custody: who collected the data, when, and under what stated terms
- No content the source's own terms explicitly restrict from AI training use, since a ToS violation can create separate legal exposure beyond copyright itself
For a full, case-by-case breakdown of the major 2025-2026 rulings and what they mean for a startup evaluating a dataset, see Is It Legal to Train AI on Copyrighted Data? The Law in 2026↗ and the head-to-head Data Licensing vs. Web Scraping↗ decision guide.
Where does training data come from?
It comes from six identifiable sources: open public-web corpora like Common Crawl, proprietary and held-out datasets that publishers and platforms increasingly license directly, user-generated content produced inside products and platforms, data bought and sold through brokers and marketplaces, synthetically generated data, and the broker and marketplace layer that connects supply to demand. Public web text isn't infinite: Epoch AI estimates only about 300 trillion usable tokens of human-generated public text exist, and projects models may fully consume that stock somewhere between 2026 and 2032, which is exactly why the other sources are growing in importance. [2] That shift already shows up in real contracts, not just projections: Google's reported licensing deal with Reddit for its user-generated forum content runs roughly $60 million a year. [4]
| Source | What it is | Trend |
|---|---|---|
| Public web / open corpora | Scraped, openly hosted text like Common Crawl | Finite, being consumed [2] |
| Proprietary & held-out | Closed archives owned by publishers | Rising, direct licensing |
| User-generated platform data | Content produced inside a product, like forum posts or support tickets | Rising, direct platform deals [2] |
| Licensed & purchased | Data bought through brokers and marketplaces | Growing |
| Synthetic | Machine-generated data | Growing fast, complementary [2] |
| Brokers & marketplaces | The layer connecting supply to demand | Formalizing |
For the full map of all six sources, including how winding-down startups feed proprietary supply into this chain, see Where Does AI Training Data Actually Come From? A Map of the Data Supply Chain↗.
Is synthetic data a substitute for real training data?
No. Synthetic data extends and augments real data; it doesn't originate genuinely new information the way human-generated data does. Synthetic examples, generated by a model or simulation rather than recorded from the real world, are cheap to produce and genuinely useful for filling gaps, balancing rare classes, and protecting privacy, and Gartner projected that by 2024 the majority of data used in AI and analytics projects would be synthetically generated. [2] A generator can only resample from patterns it already learned, so researchers have documented model collapse, a measurable, compounding quality loss that appears when models train recursively on model-generated content across generations. [6] For RLHF preference pairs and genuine domain fine-tuning specifically, real human-generated data remains the input that synthetic data can't substitute for.
“Use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.”
For the full decision framework on when to use synthetic, real, or a hybrid of both, see Synthetic vs. Real Data for AI Training: A Decision Guide↗.
One more question?
Every answer above is meant to stand on its own, but none of them is the whole story. Training data touches legal risk, licensing terms, labeling cost, and technical tradeoffs that go deeper than a few sentences can cover, which is exactly what the linked guides above walk through in full. If one thread runs through all nine answers, it's that the question worth asking about any dataset goes past how much do I have and toward where did it come from, who labeled it, and can I document that.
That last question is also the one that determines whether a dataset can be sold or licensed at all. A buyer's diligence process checks provenance and consent before it checks volume, and a seller who can answer it clearly, in writing, moves faster through every negotiation that follows.
Have proprietary data behind one of these questions?
If you're sitting on real, human-generated data from a startup that's winding down or pivoting, support transcripts, platform logs, domain corpora, Dayda can tell you what category it falls into, how it scores on quality, and what qualified buyers would actually pay.
List your data on Dayda→