- 1The AI data supply chain has six distinct sources: public web corpora, proprietary/held-out datasets, user-generated platform data, licensed and purchased datasets, synthetic data, and the brokers/marketplaces that link them.
- 2Public web text is finite and being consumed: Epoch AI estimates only about 300 trillion usable tokens and projects models may exhaust it between 2026 and 2032, pushing the industry toward closed, proprietary data.
- 3Platform and user-generated data is now monetized directly through licensing deals (Google-Reddit at roughly $60M/yr; OpenAI-News Corp at roughly $250M), proving a real, paid market for held-out data.
- 4Synthetic data is a rapidly growing but still-complementary input. Gartner projected the majority of AI training data would be synthetic, yet human-generated signal remains uniquely valuable for teaching alignment and reasoning.
- 5Winding-down startups are a growing feeder of supply: their proprietary, human-generated, platform-native datasets are scarce, legally packaged, and exactly what AI labs and enterprises are paying for.
The best raw material for training models is human-generated data, and the public internet, once treated as infinite, is being depleted faster than it grows. This guide maps the six real sources of training data that quietly power the industry: open public-web corpora like The Pile and Common Crawl; proprietary held-out datasets that publishers and platforms now license directly; user-generated platform data monetized through landmark deals; licensed and purchased datasets moving through brokers; synthetic data manufactured at scale; and the marketplaces that connect supply to demand. For founders winding down a startup, the map ends in good news: their platform-native, human-generated datasets sit squarely in the scarcest, most in-demand corner of this supply chain.
On this page ▾
A map of the AI data supply chain
Every large language model you've ever used trained on data drawn from the same six pools. Understanding the map matters because the supply chain is shifting under our feet: the cheapest, most familiar source, the public web, is being depleted, while closed, proprietary, and human-generated sources are rising in value. Here's the whole landscape before we zoom into each one.
| Source | What it is | Examples | Trend |
|---|---|---|---|
| Public web / open corpora | Scraped, openly hosted text corpora | Common Crawl, The Pile, RedPajama | Finite; being exhausted [2] |
| Proprietary & held-out | Closed text owned by publishers | News archives, papers, books | Rising; direct licensing [2] |
| User-generated platform data | Content produced inside products | Reddit posts, support tickets | Rising; direct platform deals [2] |
| Licensed & purchased | Data bought from brokers/marketplaces | Labeled datasets, domain corpora | Growing and consolidating [2] |
| Synthetic | Machine-generated data | Model-generated text, simulated data | Exploding in volume [2] |
| Brokers & marketplaces | The layer connecting supply to demand | Data brokers, AI data exchanges | Consolidating and formalizing [2] |
High-quality public web sources
The backbone of nearly every modern model is the open web. The single most-used source is Common Crawl, a nonprofit that has been crawling the internet and making petabytes of page data freely available since 2008. [4] Researchers combined these crawls with curated high-signal sub-corpora, books, code, academic papers, web texts, to build the open datasets behind many models. The canonical example is The Pile, an 800GB English corpus built from 22 diverse sub-sources. [8]
“The Pile is intentionally diverse to aid in training LMs to perform well across many domains, from small-scale legal text to larger-scale code and academic text.”
The critical tension: this source is finite and dilute. After de-duplication and quality filtering, Epoch AI estimates the usable stock of high-quality, human-generated public text at only about 300 trillion tokens, and it's being consumed faster than the web grows. [6] Models increasingly cannibalize earlier models' output, a feedback loop that is exactly why the industry is pivoting toward the closed and human-generated sources below.
Proprietary and held-out datasets
The most dramatic change in the data supply chain is the awakening of proprietary, held-out data. For two decades, publishers let crawlers take their content for free. Now they're charging for it. OpenAI's reported ~$250M multi-year deal with News Corp is the highest-profile example, licensing the publisher's journalism archive for training and display. [6] Similar deals across news, book, and academic publishers have turned held-out text into one of the fastest-growing categories of AI input.
This category is especially valuable because it's source-attributed and high-signal: a licensed news archive has clear provenance, editorial quality, and a defensible legal chain, exactly the properties dilute public-web scrapes lack. [4] For buyers, paying for held-out data trains better models and reduces legal and reputational exposure at the same time.
User-generated platform data
A quieter but structurally huge source is user-generated platform data, the conversations, decisions, and outputs produced by humans inside software. This is where the AI data market meets everyday behavior, and it's now being monetized directly. The reported Google-Reddit deal, valued near $60M per year, licenses Reddit's user content for training and surfaces it in Google search, turning a famously free platform into a paid training-data supplier. [6]
Beyond headline deals, platform-native data, customer-support transcripts, marketplace interactions, clinical notes, legal filings, sales call logs, is the bedrock of reinforcement learning from human feedback (RLHF) and fine-tuning. InstructGPT-style methods showed that aligning models to follow instructions requires human-generated preference data, not more internet text. [6] That makes this source uniquely scarce: it can't be scraped from the public web because it was produced inside a protected platform.
“We use human feedback to fine-tune our supervised learning baseline, labelers both demonstrate desired behavior and rank outputs, to make language models follow instructions more reliably.”
Licensed and purchased datasets
A growing share of training data never comes from the web or from organic platform usage. It's actively bought. This is the domain of data monetization, where organizations with valuable proprietary holdings package, license, and sell them to model builders. McKinsey frames data monetization as a deliberate strategy: turning an internal data asset into direct revenue (selling to third parties) or indirect value (improving products and decisions). [4] For a winding-down startup, this is precisely the lever that turns a cleanup task into a final asset sale.
This segment overlaps heavily with brokers and marketplaces (below), but the key distinction is that licensed data is sold with legal terms: exclusivity, duration, usage scope, and territory, rather than scraped or shared. Because the data carries a clear chain of custody, buyers pay a premium and sellers take on defined obligations, typically bounded by consent and privacy law such as the GDPR's lawful-basis and consent requirements for personal data. [4]
Synthetic data: the manufactured input
The sixth source isn't human at all. Synthetic data is generated by models or simulations rather than collected from reality, everything from a language model writing pseudo-conversations to a simulator producing synthetic sensor logs. This category has grown explosively: Gartner projected that by 2024 the majority of data used for AI and analytics would be synthetically generated, a forecast that signaled how much of the industry's appetite would be met by machines rather than humans. [6]
“Gartner predicts that 60% of the data used for the development of AI and analytics projects will be synthetically generated by 2024.”
The brokers and marketplaces connecting supply to demand
None of this flows automatically. A broker and marketplace layer exists to connect the organizations that hold data with the model builders that need it. The FTC's 2025 report, "Look Behind the Screens," maps a U.S. data broker industry that is consolidating: a handful of large firms acquiring or dominating the space, with the top brokers increasingly positioned at the center of the AI data economy. [4] This consolidation is a direct result of the demand surge documented by the AI Index: training datasets keep growing, and someone has to source them. [6]
This is where winding-down startups feed the supply. When a startup shuts down or pivots, its proprietary, human-generated datasets, support conversations, marketplace interactions, domain logs, are exactly the scarce, high-signal inputs the supply chain above is short on. A managed marketplace like Dayda specializes in packaging those assets for sale: reviewing provenance and consent, de-identifying where required, sampling behind NDAs, qualifying buyers, and negotiating licenses. In Dayda's observed marketplace deals, proprietary human-generated datasets routinely command prices from the low tens of thousands into the low hundreds of thousands, with exclusivity alone frequently multiplying value by 3–5×.
The supply chain is moving toward you
The story of AI training data is a story of migration from public to private. As the open web gets exhausted [4] and model output increasingly dominates public corpora, the marginal value of genuine, human-generated, source-attributed data keeps climbing. Publishers, platforms, and data brokers have all realized their holdings are now pricing assets, and startups that accumulated proprietary datasets sit in the same position.
Understanding the six sources gives you the vocabulary to see where you fit. If your startup generated human data inside a platform, support logs, marketplace activity, clinical or legal workflows, you hold one of the scarcest inputs in the entire chain. The question isn't whether it has value. It's how to package, license, and place it before the window closes.
Turn your data into an AI training asset.
If you're winding down or pivoting and hold proprietary, human-generated data, get a free review of provenance, value, and buyer demand. The team that brokers these deals every day can tell you where your data sits on this map, and what it's worth.
List your data on Dayda→