Structured vs. Unstructured Data for AI Training

What counts as structured data?

Structured data has a predefined schema before you ever look at a single record. Every field has a name, a type, and a fixed position, which is exactly what lets a relational database or a spreadsheet index, filter, and join on it without any interpretation step. A row is already an answer. A customer_id, order_date, and amount column don't need a model to figure out what they mean; the schema already told you. ^[10]

• Relational database rows: CRM records, ERP transactions, billing tables
• Spreadsheets and tabular exports: financial models, inventory sheets, survey response grids
• Structured application logs: fixed-field event logs with a timestamp, an event type, and numeric fields
• Sensor and IoT readings tied to a known schema (device ID, metric, value, timestamp)
• Structured forms: tax filings, insurance claims with fixed fields, medical billing codes

The defining test isn't whether the data looks tidy. It's whether a machine can parse the field's meaning without a model doing any inference. A spreadsheet with a notes column full of free-text sentences is still mostly structured, except for that one column, which is unstructured data riding inside a structured container. That distinction matters more than it sounds, and it comes back later in this guide.

What counts as unstructured data?

Unstructured data has no predefined schema and doesn't conform to the rows-and-columns model at all. ^[2] A sentence, a photograph, an audio clip: none of these arrive pre-sorted into fields a database can query. A model, or a human annotator, has to interpret the content before any of it becomes machine-usable in a structured sense.

• Free text: emails, contracts, support tickets, chat transcripts, reviews, articles
• Images: photographs, scanned documents, medical imaging, satellite imagery
• Audio: call recordings, voice memos, podcasts, meeting recordings
• Video: security footage, product demos, dashcam and sensor video
• PDFs and rendered documents: contracts, invoices, and reports where the visual layout carries meaning a parser has to reconstruct

Where semi-structured data fits: JSON, XML, and the blurry line

Semi-structured data has no fixed schema in the database sense, but it carries metadata, tags, or key-value markers that make it far easier to index and analyze than raw text. ^[2] JSON documents, XML files, HTML with markup, and log lines with a fixed set of keys all fall into this category. A JSON event payload with event_type, user_id, and timestamp keys is self-describing even though no rigid table schema enforces it ahead of time.

Semi-structured formats are the practical answer to the objection a skeptical reader should raise here: isn't this line blurry? It is, and semi-structured data is the proof. A structured application log with a free-text message field is a hybrid object: the envelope (timestamp, severity, service name) is machine-readable the moment it lands, while the message body might read "connection timeout after 30s retrying host 10.2.4.9", a sentence a parser has to interpret separately. A transcribed phone call is the same pattern from the other direction: the raw audio is unstructured, the transcript is unstructured text, but once an NLP pipeline extracts named entities and intents into fields, the output is structured again. Data doesn't stay in one category through its whole lifecycle. It moves between categories as it gets processed, and most production AI pipelines touch all three at different stages.

How much of the world's data is actually unstructured?

The figure that gets repeated everywhere, in vendor decks, conference talks, and blog posts, is that 80% of the world's data is unstructured. It's worth being precise about where that number actually comes from, because the trail is thinner than the confidence with which it gets cited. Devopedia's writeup on the topic traces it to a 1998 Merrill Lynch report, and is direct about the problem: "This 80% figure is mentioned in a Merrill-Lynch report but it's not due to primary research." ^[4] A more rigorous data point from the same period tells a different story: a 2007 TDWI Research survey found structured data at 47%, unstructured data at 31%, and the remainder semi-structured, a materially different split from the 80% truism, gathered through an actual survey rather than an unattributed estimate. ^[6]

The 80% figure hasn't gone away since 2008. It shows up in outlets as credible as MIT Sloan Management Review, which in a February 2021 article cited "a majority of data (80% to 90%, according to multiple analyst estimates)" without naming a single underlying study. ^[4] That's not a knock on MIT Sloan specifically. It's evidence of how deeply a number can embed itself in industry conventional wisdom without anyone re-deriving it.

If you need a number with an actual, dated source behind it, the best one available is IDC's. IBM's current explainer on unstructured data cites IDC's August 2023 report, "Untapped value: What every executive needs to know about unstructured data," for the claim that unstructured data comprises roughly 90% of enterprise-generated data. ^[4] That's higher than the old 80% truism, and it's specific to enterprise data rather than "all the world's data," a scope distinction that matters and that most casual citations of the 80% figure skip entirely. Use the IDC-sourced number if you need one, and treat any structured/unstructured ratio you see cited without a name and a year as decoration, not evidence.

Why LLM training leans unstructured while fraud, forecasting, and recommendation lean structured

Large language model pretraining runs on unstructured text for a structural reason. Pretraining uses self-supervised learning: the model predicts a missing or next piece of text from the surrounding context, which means it needs no human-added label at all. That removes the single biggest cost constraint on structured data, which is that its "label" (a fraud column, a churned flag) has to already exist or be added by hand. Unstructured web text is also the largest pool of raw material available. Epoch AI estimates the usable stock of high-quality, human-generated public text at roughly 300 trillion tokens, the scale pretraining needs to keep improving. ^[10]

Structured data still plays a role once a model is trained, most visibly through retrieval-augmented generation over structured knowledge bases and through fine-tuning on labeled examples. But the bulk of the token count that gives a foundation model its general language competence is unstructured text, because that's the modality that offers self-supervised signal at the volume frontier models require.

Outside of language, vision, and audio understanding, most production ML runs on structured, tabular data, and for good reason. Fraud detection scores a transaction against fields that already exist: amount, merchant category, device fingerprint, account age, velocity of recent transactions. Demand forecasting works off time series of structured sales and inventory records. Recommendation systems learn from a structured user-item interaction matrix: which items a user clicked, purchased, or rated. None of these tasks need a model to interpret free text or pixels; they need a model to find patterns across already-labeled, already-numeric columns.

Tree-based models, gradient-boosted trees especially, remain the default choice for this kind of data because they handle heterogeneous, tabular features well without the volume or compute deep learning architectures need. A representative example: a 2023 evaluation of XGBoost across balanced and imbalanced datasets for fraud detection found it a standard, high-performing choice specifically because of how it handles structured transaction features, not despite them. ^[2] That's the pattern across the field: the problems that are naturally tabular stay tabular, and the modeling technique follows the data's native shape rather than the other way around.

How labeling and annotation costs differ

The cost asymmetry between the two data types comes down to one fact: structured data usually already encodes its own label, and unstructured data almost never does. A is_fraud column or a close_date field is the answer a model needs to learn from; nobody has to sit down and manually decide what that row means. A support ticket transcript, a claims photo, or a call recording carries no such answer. Someone, a human annotator, a domain expert, or an AI-assisted labeling pipeline reviewed by a human, has to add it before a supervised model can train on it.

That difference shows up at scale in ways that are easy to underestimate from the outside. ImageNet, still the reference dataset for supervised computer vision, required human annotators to hand-sort 14,197,122 images into more than 21,841 WordNet categories. ^[6] No equivalent manual sorting exists for a structured dataset of the same size; a database of 14 million transaction rows with a category column is queryable the moment it's exported. TDWI's 2025 analysis of AI project timelines quantifies the downstream effect: unstructured-data projects typically take 2 to 3 times longer to develop than structured-data projects, driven by additional preprocessing and the labeling work itself. ^[12]

A worked example: two AI projects, one company

Consider a mid-size insurer running two AI initiatives at once, a realistic scenario because most companies of any size eventually need both data types for different products.

• 1Project A: a claims-fraud scoring model. The inputs are already structured: claim amount, policy tenure, claimant history, time-to-file, prior claim count. The target label, whether a claim was previously confirmed fraudulent, already exists in the claims database. The team's real work is feature engineering and validating that the historical labels are clean, not manual annotation. A gradient-boosted tree model trains on this table directly. ^[4]
• 2Project B: a claims-adjuster copilot. The inputs are unstructured: free-text adjuster notes, photos of vehicle or property damage, and recorded claimant phone calls. None of this has a usable label until someone extracts one: photos need damage-severity tags, notes need entity extraction (what was damaged, estimated repair cost mentioned in prose), and calls need transcription before any of the above can even start. This project needs a labeling pipeline, likely with domain-expert review for the damage assessments, before a single supervised model can train.
• 3The same company, two different budgets. Project A ships faster and cheaper because its ground truth already exists in a column. Project B needs a dedicated data pipeline: transcription, annotation guidelines, inter-annotator agreement checks, and a slower timeline, consistent with the 2 to 3x gap TDWI documents between unstructured- and structured-data AI projects. ^[4] Neither project is more valuable than the other. They carry different cost profiles, and treating them the same during planning is the most common way both projects end up under-budgeted.

A practical framework: which type does your AI project actually need?

• 1Does your target variable already live in a column? If yes (churned, converted, fraudulent, next-quarter revenue), you have structured data and likely need a classical or tree-based model, not an LLM. Don't build a labeling pipeline you don't need.
• 2Does the task require language, vision, or audio understanding? If the model has to read, see, or listen to something to do its job, you need unstructured data and a deep learning approach, and you should budget for annotation from day one, not as an afterthought.
• 3Is your real differentiator proprietary knowledge that a public model wasn't trained on? That knowledge usually lives in unstructured internal documents: contracts, support history, internal wikis. It's better served by fine-tuning or retrieval over that corpus than by trying to force it into a spreadsheet.
• 4Check for a semi-structured shortcut first. Product logs, API payloads, and structured exports often already carry a meaningful share of the signal a team assumes requires a full unstructured labeling project. Extracting fields from JSON is materially cheaper than annotating raw text from scratch.
• 5How much labeling budget and time do you actually have? Structured projects ship in a fraction of the time of unstructured ones, on average 2 to 3 times faster by TDWI's estimate. ^[4] If the timeline is fixed and tight, that constraint alone can determine which data type is realistic to build around.
• 6Who has to do the labeling, and are they qualified? Generic crowdsourced labeling works for object detection on street scenes. It doesn't work for medical imaging, legal document review, or claims-damage assessment, where mislabeled unstructured data is worse than no data at all.

The bottom line

Structured and unstructured data aren't competing philosophies. They're different starting points that carry different costs. Structured data ships fast because its ground truth usually already exists in a field. Unstructured data carries the deepest signal for language, vision, and audio tasks, but that signal has to be manufactured through labeling before a model can use it, and that manufacturing is where most of an unstructured project's real budget and timeline goes. Semi-structured formats split the difference and are worth checking first, because they often deliver a meaningful share of the value at a fraction of the labeling cost.

Skip the recycled 80% statistic when you're making a real planning decision. Use IDC's actual, dated estimate instead ^[2], and build your data strategy around what your specific project's ground truth looks like today, not around a number nobody can trace back to its source.