What Data Do LLMs Like GPT, Claude, and Llama Actually Train On?

Three training stages, three completely different kinds of data

"What does an LLM train on" has three answers, because a modern chat model goes through three distinct training stages and each one consumes a different substance. Conflating them is the single most common error in explanations of this topic, and it makes model cards read as more evasive than they are, because a card that says almost nothing about pretraining sources often says quite a lot about the two later stages.

• 1Pretraining. Trillions of tokens of raw text and, increasingly, images, audio, and video. This is where web crawls, code repositories, book archives, and licensed publisher content go in. It is also the stage labs describe most vaguely. Meta discloses roughly 40 trillion tokens for Llama 4 Scout and roughly 22 trillion for Maverick, which is the most specific pretraining number any of the four majors publishes. ^[6]
• 2Supervised fine-tuning, also called instruction tuning. Curated demonstrations showing the model what a good response to an instruction looks like. This corpus is smaller than the pretraining set by three to five orders of magnitude, and it is where paid contractors and data-labeling vendors do most of their work. Anthropic names "data provided by data-labeling services and paid contractors" as an explicit source category. ^[4] Google's Gemini 3 Pro card lists "different types of instruction tuning data" in its post-training set. ^[6]
• 3Reinforcement learning from feedback. Preference comparisons rather than demonstrations: two or more candidate responses, and a judgment about which is better. Google names "human-preference data" as a post-training input ^[4], and Anthropic states that Claude Opus 4.6's post-training used "reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback." ^[6]

The preference stage is where alignment and much of a model's perceived personality come from, and it runs on a data type that cannot be scraped: structured human judgment about response quality. We cover how that data is collected, structured, and priced in our guide to RLHF and preference data↗. For the rest of this article, when a lab describes "training data" without qualification, assume it means pretraining unless the card says otherwise.

The open corpora that anchored a decade of LLM pretraining

Before frontier labs stopped itemizing their data, and still today for open models, the reference points were a handful of public corpora whose composition is documented down to the sub-dataset. If you want to know what "web-scale pretraining text" concretely consists of, these are the datasets to read, because their papers publish exactly what commercial cards omit.

Two of these corpora exist as direct responses to commercial opacity. Dolma's authors open by observing that commercial models rarely detail their data and that even open models often ship without their training data, then release both the corpus and the toolkit that built it. ^[2] FineWeb's contribution is comparable in spirit: rather than asserting that its filtering pipeline works, its authors publish an ablation for each design choice, so a reader can see what deduplication and quality filtering actually did to model performance. ^[4]

None of this proves that any specific frontier model trained on any specific one of these datasets. It does establish the shape of the thing. When OpenAI writes that GPT-5.6 trained on "information that is publicly available on the internet" ^[2], the concrete referent for that phrase, at the scale involved, is a filtered and deduplicated derivative of large web crawls that looks structurally like C4, FineWeb, or Dolma, whatever the internal pipeline is called.

What OpenAI, Anthropic, Google DeepMind, and Meta actually disclose in 2026

All four publish official documentation for their flagship models. All four describe categories of data and curation practices rather than sources. Where they differ, and the gap is wider than most summaries suggest, is in how many categories they name, whether they attach numbers, and whether they name any specific upstream property.

OpenAI: three categories, no names, no numbers

OpenAI's current disclosure is the shortest of the four and has barely changed across releases. The GPT-5.6 preview system card, published June 26, 2026, states that the model "was trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide or generate," then describes filtering for quality, personal information reduction, and safety classifiers. ^[2] That is the entire training-data section. It names no corpus, no partner, no modality split, and no token count. The August 2025 GPT-5 system card used near-identical language, so this is a settled house position rather than a one-off omission.

Anthropic: five categories, a dated cutoff, and documented crawler behavior

Anthropic's Claude Opus 4.6 system card, published February 2026, is more specific on two axes that matter: it dates the public-web portion, and it describes how that portion was collected.

The same section adds that Anthropic uses a general-purpose web crawler that follows industry-standard practice with respect to robots.txt, does not access password-protected pages or pages requiring sign-in or CAPTCHA verification, and operates transparently enough that site operators can identify it and signal preferences. ^[4] A separate subsection covers crowd workers and the standards Anthropic applies to the data-work platforms it contracts with. What is still absent: which third parties, which internally generated data, what proportion of the mix each category represents, and any token count.

Google DeepMind: the most granular category list of the four

The Gemini 3 Pro model card describes a pretraining set spanning "publicly-available web-documents, text, code, images, audio (including speech and other audio types) and video," then enumerates seven distinct acquisition channels: publicly available datasets that are readily downloadable, data obtained by crawlers, licensed data obtained via commercial licensing agreements, user data from Google products and services subject to Google's terms and user controls, other datasets Google acquires or generates in the course of business operations or directly from its workforce, and AI-generated synthetic data. ^[2] Processing is described as including deduplication, honoring robots.txt, and safety filtering. The February 2026 Gemini 3.1 Pro card does not restate any of this and instead points readers back to the Gemini 3 Pro card, which is worth knowing if you are trying to source a claim about a current Gemini model.

Google is the only one of the four to name commercial licensing and AI-generated synthetic data as separate, explicit pretraining inputs in a model card. It is also the only one whose disclosure includes first-party consumer product data as a named category, which is a meaningfully different provenance story from a web crawl.

Meta: the only lab publishing hard numbers

Meta's Llama 4 model card discloses what the others do not: scale and dates. Llama 4 Scout was pretrained on roughly 40 trillion tokens and Llama 4 Maverick on roughly 22 trillion tokens of multimodal data, with a pretraining cutoff of August 2024. The source description is "a mix of publicly available, licensed data and information from Meta's products and services," and Meta names those products: "publicly shared posts from Instagram and Facebook and people's interactions with Meta AI." ^[8] That is the most concrete named-source disclosure among the four majors, and it comes from the lab that Stanford CRFM's index ranked lowest among the long-tracked companies.

Stanford CRFM's 2025 Foundation Model Transparency Index puts numbers on the pattern. Across 13 companies, the mean score was 41 out of 100, a 17-point decline from 2024, with Data Acquisition and Data Properties among the weakest subdomains. IBM scored highest at 95; xAI and Midjourney tied lowest at 14. ^[4] By the most systematic public measurement available, transparency about training data is actively receding year over year.

Model family by model family: disclosed, undisclosed, and sourcing pattern

This is the comparison to keep. Every cell below comes from the lab's own current model or system card, verified against the primary document rather than secondary reporting.

Read across the rows and the shared pattern is clear. All four majors run the same three-part recipe: filtered public web crawl at the base, a paid layer on top (licensed corpora, contracted human data, or both), and first-party data from their own products where they have it. What separates the rows is which parts of that recipe each lab is willing to name, since the underlying sourcing approaches converge.

The licensed layer: what labs pay for on top of the crawl

Two of the four cards name commercial licensing as an explicit input. Google lists "licensed data obtained via commercial licensing agreements" ^[2], and Meta lists "licensed data" alongside publicly available and product data. ^[4] Neither names a counterparty, so the visible record of who is paid and how much comes almost entirely from the sell side: public companies reporting the revenue, and reporting on deals in progress.

Reddit is the clearest case because it is a public company whose licensing revenue is legible. Columbia Journalism Review's October 2025 analysis reports the Google agreement at $60 million a year and the OpenAI agreement at an estimated $70 million a year. ^[6] Those figures matter here for a structural reason more than a financial one: a Reddit corpus is not a crawl artifact. It is a live feed of human conversation delivered under contract, with terms about what the buyer may do with it, which is a different provenance category than anything a robots.txt-respecting crawler can assemble.

The full landscape of these agreements, including the news publisher deals, stock media licensing, and the shift from training rights toward inference-time licensing, is mapped in our breakdown of AI data licensing deals↗. For the purposes of answering "what do LLMs train on," the load-bearing point is narrower: a meaningful and growing slice of frontier pretraining data now arrives through contracts rather than crawlers, and that slice is invisible in every model card except as the word "licensed."

How much of a modern pretraining mix is actually synthetic?

Synthetic data has moved from a research curiosity to a disclosed production input. Google's Gemini 3 Pro card lists "AI-generated synthetic data" as one of its enumerated training data sources, in the same sentence as crawled and licensed data. ^[2] Anthropic's phrasing, "data generated internally at Anthropic," is broad enough to cover model-generated data without saying so. ^[4] No lab publishes the synthetic share of its mix.

The best public estimate of what a sensible share looks like comes from a 2025 study that pretrained more than 1,000 models across more than 100,000 GPU hours specifically to answer this question. Its findings are unusually actionable. Pretraining on rephrased synthetic data alone was not faster than pretraining on natural web text. Mixing roughly one third rephrased synthetic data with two thirds natural web text produced a 5x to 10x speedup to reach the same validation loss at larger data budgets. Good ratios depended on model size and data budget but converged empirically on about 30% synthetic. Pretraining on textbook-style, purely generated synthetic data alone produced notably higher loss across many downstream domains, and showed the degradation patterns predicted by model collapse. ^[8]

Two conclusions follow for anyone reasoning about where training data comes from. First, synthetic data at production scale currently works as a multiplier on real data: the speedup in that study came from the mixture, and every synthetic-only condition underperformed. Second, the distinction between rephrasing real text and generating text from scratch is doing enormous work, because the two conditions produced opposite results on model collapse. Our guide to synthetic versus real training data↗ covers when each is the right call for a specific project, including the compliance and cost tradeoffs this section skips.

How copyright litigation reshaped what labs will say

The narrowing of disclosure since 2023 tracks the rise of training-data litigation closely enough that the two are hard to separate. The largest resolved example: a U.S. District Court for the Northern District of California granted final approval on July 20, 2026 to a $1.5 billion class settlement in *Bartz v. Anthropic*, covering authors whose works were obtained from pirated book repositories, at roughly $3,000 per work. ^[6] Where a lab acquired a corpus is now a fact with a price attached, which is a strong incentive to describe acquisition in categories rather than sources.

Treat the causal claim as directional rather than proven. Labs also cite competitive sensitivity for withholding data details, and the Transparency Index's 17-point single-year decline spans indicators well beyond data acquisition. ^[2] The legal picture itself remains unsettled across jurisdictions and fact patterns, and we work through the actual rulings and standards in our analysis of whether training on copyrighted data is legal↗.

How to read a model card for training data, in six steps

Applying the above to a card you have not seen before takes about ten minutes. This is the sequence we use when assessing what is actually known about a model's data.

• 1Find the primary document, not the press post. Launch blogs paraphrase; the model or system card PDF is the citable artifact. Google publishes Gemini cards as PDFs on storage.googleapis.com and DeepMind card pages; Anthropic publishes system cards as PDFs on its CDN; OpenAI publishes on its deployment safety hub; Meta publishes model cards in the meta-llama/llama-models GitHub repository.
• 2Check whether the card defers. Point releases frequently point back to the base model for training data. The Gemini 3.1 Pro card, published February 2026, states that it is based on Gemini 3 Pro and refers readers to that card for training dataset information. ^[4] Citing the point-release card for a data claim it does not contain is a common sourcing error.
• 3Separate pretraining from post-training in the text. Look for the two paragraphs. Post-training descriptions naming instruction tuning, preference data, or RLHF tell you nothing about pretraining sources, and vice versa. ^{[4] [6]}
• 4Count the named categories, then ask what each one hides. "Third-party data" and "licensed data" are placeholders for contracts the card will not name. "Internally generated data" typically covers model-generated data. "Product data" means first-party user data, which raises consent questions a crawl does not.
• 5Look for a date and a number. A stated cutoff (Anthropic's May 2025 public-internet cutoff, Meta's August 2024 pretraining cutoff) and a token count (Meta's ~40T for Scout) are the only quantitative anchors any of these cards provide. ^{[4] [6]} Their absence, as in OpenAI's cards, means you cannot size anything.
• 6Check the crawl conduct statement. Whether the card says the crawler honors robots.txt and avoids authenticated pages is the one provenance-quality signal labs consistently volunteer, because it is the one they can make without naming anything. Anthropic and Google both state it explicitly. ^{[6] [8]}

The honest bottom line

LLMs train on filtered web crawl at the base, layered with licensed corpora, contracted human demonstration and preference data, first-party product data where the lab owns a platform, and a growing but bounded share of synthetic data. That recipe is consistent across OpenAI, Anthropic, Google DeepMind, and Meta, and each of them says so in their own words. What none of them says is which sources, in what proportion, from which counterparties. Meta publishes token counts and names two of its own platforms; Google enumerates seven acquisition channels; Anthropic dates the crawl and documents the crawler; OpenAI names three categories and stops. The full source list, in every case, is unpublished.

That gap is why the open corpora still matter as evidence. The Pile, C4, FineWeb, and Dolma are the closest thing available to a ground-truth picture of what web-scale pretraining data physically consists of, and their papers document the filtering and deduplication decisions that commercial pipelines make behind a wall. ^{[2] [4] [6] [8]} Meanwhile the paid layer keeps growing, licensing revenue keeps showing up on the sell side's income statements ^[10], and the industry's measured transparency keeps falling. ^[12] Anyone holding real, provenance-clean data is supplying the one layer of this stack that the crawlers cannot reach and that every card refers to only as "licensed."