What Is Video and Multimodal AI Training Data?

What video and multimodal training data actually is

A text corpus is a one-dimensional sequence. Token 1, token 2, token 3, and the only relationship the model has to learn is which tokens tend to follow which. Video adds two spatial dimensions and one temporal one on top of that, and multimodal datasets add a second synchronized stream (audio, and sometimes a third stream of actions or sensor readings) that has to stay aligned with the first to be worth anything.

That difference is not a matter of degree. A model learning from video has to learn object permanence, occlusion, camera motion versus subject motion, and cause and effect over time, none of which have an analogue in next-token prediction over prose. The unit of meaning stops being a token and becomes a trajectory through space. OpenAI's framing of what made Sora 2 an advance over its predecessor was exactly this: more accurate physics, sharper realism, and audio generated jointly with the video rather than added afterward. ^[4]

Why video is a fundamentally harder data problem than text

Three constraints separate video from text, and each one compounds the others.

1. It is expensive as signal, not just as storage

Everyone expects video files to be large. The more important cost shows up inside the model's context window. Google's Gemini API documentation states that video is sampled at 1 frame per second and counts approximately 258 tokens per frame plus 32 tokens per second of audio, roughly 300 tokens per second of video at default media resolution, or 100 tokens per second in low-resolution mode. ^[4] Run the arithmetic: one hour of footage is about 1.08 million tokens. One hour of recorded human speech transcribes to roughly 9,000 words, or on the order of 12,000 tokens. The same hour of real-world time costs about ninety times more to represent as video than as text, and that ratio holds whether you are pretraining, fine-tuning, or serving inference. Google notes the same documentation's caveat: at 1 fps, rapid motion and quick scene changes can be missed entirely, so the 300-tokens-per-second figure is the cost of a deliberately lossy sampling, not of the full signal.

2. The compute economics can be genuinely unsurvivable

The clearest demonstration in the market so far is what happened to OpenAI's Sora app. Launched September 30, 2025 with a consumer video feed, it was discontinued on March 24, 2026. TechCrunch, citing a Wall Street Journal investigation, reported a peak of roughly one million users falling below 500,000, against a burn of approximately $1 million per day, and that OpenAI shut it down partly to free compute for its enterprise competition with Anthropic. ^[4] A text product with that user count would have been cheap enough to leave running indefinitely. Video was not.

3. Far less of it exists in usable, licensed form

The public web holds hundreds of trillions of tokens of text that a crawler can collect without negotiating with anyone. Licensed video sits at a different order of scale entirely. When Lionsgate and Runway announced a custom model trained on the studio's library of more than 20,000 titles, it read like a template for the industry. By September 2025 TheWrap reported the project had stalled, and quoted a source on the core problem in terms worth reading twice. ^[2]

That reporting also surfaced the second scarcity problem, which is legal rather than numeric: whether a studio even holds the rights to train on its own films, given ancillary rights held by actors, directors and other creators under their original agreements. ^[2] A film library is not one asset with one owner. It is thousands of overlapping grants, most of them written before anyone contemplated model training as a use.

The five video data types that matter, and what each is for

"Video data" is not one product. Buyers pay for five distinguishable things, and a seller who does not know which one they hold will misprice the archive in both directions.

• 1Raw video. Unlabeled footage with nothing but timestamps and file metadata. It supports self-supervised pretraining, where a model learns motion and appearance statistics without any human label, which is the video analogue of next-token prediction. Its value per hour is the lowest of the five, and it is the only type that exists in genuinely large volume.
• 2Temporally captioned or annotated video. Descriptions attached to specific spans: clip-level captions ("a forklift reverses out of aisle 4 and stops"), frame-level bounding boxes or masks, action segment boundaries, and object tracks that persist across frames. This is the type that makes text-to-video generation and video question answering possible, because it is the only type that connects language to motion. It is also the most expensive to produce, for reasons the annotation section below makes concrete.
• 3Paired audio-video. Video with synchronized, well-separated audio: dialogue, ambient sound, and the sound events that correspond to visible actions. Both DeepMind and OpenAI now treat joint audio-video generation as a headline capability rather than a post-process, with Veo 3.1 producing native dialogue, sound effects and ambient audio alongside 1080p and 4K output ^[4], and Sora 2 generating audio and video together. ^[6] Models cannot learn that pairing from silent footage or from a separately recorded soundtrack.
• 4Action and trajectory data. State-action sequences: what an agent, robot or vehicle observed, what command it executed, and what happened next, usually logged at 20 to 100 Hz and time-aligned to camera streams. This is the scarcest of the five and the input that world models for robotics depend on. NVIDIA describes Cosmos 3 as trained on "billions of samples across text, image, video, sound and action trajectories," positioned as the backbone for world action models that train robots on specific tasks. ^[4] The scale of the open equivalent is instructive: Open X-Embodiment, the pooled academic dataset, covers 22 robot embodiments from 21 institutions across 527 skills and 160,266 tasks ^[6], a rounding error next to any text corpus.
• 5Synthetic and simulation-rendered 3D. Frames rendered from a physics simulator or generated by a world model, with perfect ground truth attached for free: exact depth, segmentation, pose and object identity, because the renderer already knows them. DeepMind positions Genie 3, which generates navigable worlds at 720p and 24 fps with a few minutes of continuous interaction, explicitly as a way to train agents including robots in simulated environments. ^[4] Synthetic data solves the labeling problem completely and the realism problem only partially, which is why it supplements real footage rather than replacing it.

Comparison: data type, use case, and the constraint that governs it

What the 2026 video and world models actually run on

Disclosure about video training data is thinner than disclosure about text training data, and what is public is worth stating precisely rather than inferring.

OpenAI's Sora line. Sora 2 launched September 30, 2025 with joint audio-video generation, improved physics, and a cameo feature that inserts a real person's appearance and voice into generated scenes on a consent basis. ^[4] The consumer app was discontinued March 24, 2026 on the economics described above. ^[6] OpenAI has never published the composition of Sora's training corpus.

Google DeepMind's Veo. Veo 3.1 generates 1080p and 4K video with native audio including dialogue and ambient sound. DeepMind states that all Veo output carries SynthID watermarking, and that the model undergoes "safety evaluations and checks for memorized content to reduce potential issues related to privacy, copyright infringement, and bias." ^[4] That memorization-check language is one of the few public signals any lab has given about how it handles the risk of a video model reproducing licensed source footage.

Runway. Runway's commercial position has been built on studio partnerships rather than open-web scale, starting with the Lionsgate custom-model project, and it appears as a named customer in Shutterstock's licensed training data business. ^[4] The Lionsgate difficulties reported by TheWrap are the clearest public account of what happens when a curated, rights-clean corpus turns out to be too small for the job. ^[6]

World models for robotics. This is where the data question is most acute, because the target is not a plausible video but a simulator accurate enough to train a policy that will run on physical hardware. DeepMind's Genie 3 generates interactive environments at 720p and 24 fps with roughly one minute of visual memory, framed as "a vast space to train agents like robots and autonomous systems." ^[4] NVIDIA's Cosmos 3, announced May 31, 2026, is the most explicit public statement of a physical-AI data recipe: billions of samples spanning text, image, video, sound and action trajectories, shipped with datasets for robotics, physics, human motion, autonomous driving, warehouse safety and spatial reasoning, and marketed on compressing physical AI training and evaluation cycles from months to days. ^[6]

How video training data is actually being licensed, and how those deals ended

Four sourcing routes are visible in the public record, and each one has produced a documented outcome rather than just an announcement.

Stock footage libraries. The steadiest channel. Shutterstock announced on March 19, 2026 a major expansion of its licensed training datasets, adding long-form video, premium metadata, templates, fonts, and podcast and science imagery, and named OpenAI, Runway, Black Forest Labs and ElevenLabs among the AI companies it supplies. ^[4] The strategic point in that release is the pairing of "long-form video" with "premium metadata": stock libraries sell annotation quality and rights clearance, not raw pixel volume, which is the only thing they have that the open web does not.

Studio and entertainment licensing. On December 11, 2025 Disney and OpenAI announced a three-year agreement covering more than 200 animated, masked and creature characters across Disney, Pixar, Marvel and Star Wars, together with costumes, props, vehicles and environments, alongside a $1 billion Disney equity investment in OpenAI. The agreement expressly excluded "any talent likenesses or voices." ^[4] That exclusion is the single most instructive clause in the entire deal: Disney could license the characters it owns and could not license the actors it employs. Three months later the Sora app was discontinued and the arrangement ended with it, with Disney reportedly informed under an hour before the announcement. ^[8]

Platform-mediated creator licensing. YouTube's answer to the scraping controversies was a per-creator control rather than a platform-wide license. YouTube's documentation states the third-party training setting is off by default, that creators must affirmatively enable it, and that they can approve specific companies rather than all of them. ^[6] A default-off, per-creator, per-company permission model is close to the opposite of how text training corpora were assembled, and it means platform-scale video access has to be bought consent by consent rather than acquired in a single negotiation.

Direct proprietary sourcing. Buying footage straight from the operator who recorded it: warehouse and logistics operators, dashcam fleets, security integrators, medical device makers, sports leagues. This is where most genuinely differentiated video data now sits, because it never appeared on the open web and its provenance is documentable back to a single corporate owner. It is also where the consent analysis is hardest, because the people in the frame are usually employees or customers rather than performers under contract.

Why annotating video costs so much more than annotating images or text

The intuitive explanation is that video is many images. The real explanation is worse: video is many images that have to agree with each other. A bounding box drawn on frame 412 is wrong if it does not correspond to the same physical object as the box on frame 411, and a human annotator has to maintain that identity through occlusion, motion blur, lighting changes, and the object leaving and re-entering the frame.

Meta's SAM 2 team published the only clean public accounting of what this costs. Building the SA-V dataset, they ran a three-phase data engine and reported per-frame annotation times at each phase: 37.8 seconds per frame in Phase 1, where annotators used the original SAM on each frame independently; 7.4 seconds per frame in Phase 2; and 4.5 seconds per frame in Phase 3, once SAM 2 itself propagated masks through time and annotators only corrected drift. That is an approximately 8.4x speedup, and it produced 50.9K videos, 642.6K masklets and 35.5M masks. ^[10]

Convert those per-frame numbers into per-content numbers, because that is the unit a dataset is priced in.

The spread across those rows is roughly 75x for the same underlying footage, decided entirely by annotation granularity. That is the number that should govern how a video dataset is scoped. Exhaustive per-frame masks on an hour of footage cost more than most teams spend annotating an entire text corpus, and model-in-the-loop tooling changes the multiple without changing the shape of the problem. Three complications make even the fast row optimistic in practice:

• Temporal consistency has no cheap quality check. Inter-annotator agreement on a static label is a single number. Agreement on a temporal boundary (when exactly did the action start?) routinely disagrees by hundreds of milliseconds between competent annotators, and that disagreement propagates into every action-recognition model trained on the labels.
• Multi-object tracking cost scales with object count, not frame count. A warehouse aisle with 9 moving entities costs roughly nine times what a single-subject clip costs, and identity switches (annotator swaps two objects' IDs mid-clip) are the most common silent defect in tracking datasets.
• Action labeling requires domain expertise that generic crowdwork cannot supply. Labeling "surgical instrument handoff" or "unsafe pallet stack" needs a practitioner, and practitioner time does not cost $12 an hour.

Consent, biometrics, and likeness: the law that applies to footage of people

A frame of video containing a person is three legally distinct things at once, governed by three different bodies of law, and holding the copyright in the footage settles only one of them.

Biometric privacy: BIPA and the consent that footage licensing does not include

Illinois's Biometric Information Privacy Act defines a biometric identifier as a retina or iris scan, fingerprint, voiceprint, or scan of hand or face geometry, and expressly excludes photographs from the definition. Section 15(b) requires a private entity to inform the subject in writing that biometric data is being collected, state the purpose and duration of use, and receive a written release executed by the subject before collecting it. Section 15(c) separately bars any private entity in possession of biometric data from selling, leasing, trading or otherwise profiting from it. Section 20 provides liquidated damages of $1,000 for each negligent violation and $5,000 for each intentional or reckless violation, plus attorneys' fees. ^[8]

The exclusion of photographs is the point of contention that matters for video training data. A recorded frame is not itself a biometric identifier under the statute, but deriving a scan of face geometry from that frame plausibly is, and that derivation is exactly what a face-aware model pipeline does. The same reasoning has now been extended to audio. In May 2026, nine BIPA class actions were filed in Chicago federal court naming Amazon, Adobe, Google and Alphabet, Apple, Microsoft, Samsung, Meta, ElevenLabs and NVIDIA, brought by journalists, audiobook narrators and podcasters alleging their voice recordings were ingested to train AI models without the written release BIPA requires. ^[2] Plaintiffs' framing in those cases is direct: "Like a fingerprint, a voiceprint identifies the individual and cannot be changed." ^[4]

Likeness and digital replicas

The cleanest illustration of the separation is in the Disney and OpenAI agreement itself. Disney licensed more than 200 characters plus costumes, props, vehicles and environments, and the announcement states the agreement does not cover "any talent likenesses or voices." ^[2] Disney owns Iron Man. Disney does not own the actor who played him. Sora's cameo mechanism approached the same problem from the individual's side, requiring a person to record and grant use of their own likeness rather than inferring permission from footage. ^[4]

Synthetic media disclosure: the 2026 to 2028 compliance calendar

Disclosure obligations for AI-generated audiovisual content are now on a dated schedule rather than a proposal track. California's AI Transparency Act, as amended by AB 853, applies to covered providers, defined as anyone producing a publicly accessible generative AI system with over 1,000,000 monthly visitors or users, and requires both latent and manifest disclosures identifying the system name, version number, and creation or alteration time, made permanent or extraordinarily difficult to remove to the extent technically feasible. The staged operative dates are: ^[6]

• August 2, 2026: core covered-provider disclosure obligations take effect (moved from January 1, 2026 to align with the EU AI Act's Article 50 provenance timeline). ^[4]
• January 1, 2027: large online platforms, defined as public-facing social media, file-sharing, mass messaging platforms or search engines exceeding 2,000,000 unique monthly users, must detect provenance data in distributed content and surface it to users. ^[6]
• January 1, 2028: capture device manufacturers, meaning makers of cameras, phones and other devices that record photographs, audio or video, must embed latent disclosures by default. ^[4]

The 2028 capture-device provision is the one with direct consequences for training data supply. Once devices stamp provenance by default, footage recorded after that date arrives with a machine-readable claim about its origin. That makes provenance verification cheaper for buyers and makes undocumented legacy archives comparatively harder to sell. Labs are already moving in the same direction voluntarily: DeepMind marks all Veo output with SynthID. ^[2]

Worked example: evaluating a 61,000-hour proprietary video archive

A logistics operator with 14 automated warehouses approaches a marketplace with what it describes as 61,000 hours of operational camera footage recorded between 2021 and 2026, plus 3.4 million robot pick-and-place trajectories from its autonomous mobile fleet. The headline number is 61,000 hours. The number that matters is much smaller, and finding it takes three sequential passes. Run them in this order, because clearance failures make technical quality irrelevant.

Pass 1: license and consent clearance

• 22,000 hours were recorded at facilities in Illinois and Texas where workers are identifiable by face and gait, and the employee notices on file cover workplace safety monitoring, not AI training or any transfer to a third party. Under BIPA's Section 15(b) written-release requirement and Section 15(c) bar on profiting from biometric data, this subset either drops out or goes through irreversible face redaction before it can be offered ^[4]
• 9,000 hours contain third-party carrier drivers and site visitors who have no employment relationship with the operator and received no notice at all. Excluded.
• 4,500 hours come from a facility run under a third-party logistics contract where the customer, not the operator, owns the record of its own operations. Excluded pending that customer's written consent.
• Cleared or redactable: 25,500 hours. The archive has already lost 58% of its headline size, and no one has looked at a single frame yet.

Pass 2: technical usability

• 11,000 hours came from legacy cameras retired in 2023, recording at 720p and 8 fps. That frame rate is below what action modeling needs; the subset is usable for coarse scene classification and little else.
• 3,200 hours are overnight footage of static empty aisles. Near-zero information density, and including it actively harms a training mix by diluting the motion distribution.
• Usable: 11,300 hours at 1080p and 30 fps with genuine motion in frame.

Pass 3: annotation completeness, which sets the price

• 2,700 hours are time-aligned to robot telemetry at 50 Hz: joint states, wheel odometry and gripper state, with camera and controller clocks synchronized to a documented tolerance. This is action-trajectory data, the rarest of the five types, and the reason a physical-AI buyer would take this call at all. ^[4]
• 900 hours already carry clip-level event labels generated by the operator's safety-incident review process: near-miss, dropped load, blocked aisle, each with a human-written description and a verified timestamp. Human-authored, incident-grounded, and effectively impossible to reconstruct after the fact.
• 7,700 hours are raw, timestamped, and unlabeled.

Now price the gap. Bringing those 7,700 raw hours up to exhaustive per-frame mask annotation, even at SAM 2's accelerated 4.5 seconds per frame, requires roughly 832,000 annotator-hours, on the order of $10 million at a $12 hourly rate. ^[4] Annotating the same 7,700 hours with clip-level captions at two minutes per ten-second clip costs about 92,400 annotator-hours, roughly $1.1 million. That 9x difference is a scoping decision rather than a vendor negotiation, and it should be settled before anyone quotes a price for the raw subset. Against either figure, the 900 hours of incident labels the operator already produced as a byproduct of its safety process look like the best-value annotation in the archive.

The bottom line

Video and multimodal training data inverts the economics that made text data cheap. Storage and context costs are roughly two orders of magnitude higher per hour of real-world time. Annotation costs scale with frame rate and object count rather than with document count, and the spread between granularities on identical footage runs to 75x. The licensed supply is small enough that people building studio models concluded the largest film catalogs in the world are insufficient on their own. And every frame containing a person carries biometric, publicity and disclosure obligations that a footage license does not resolve.

None of that makes video data a bad asset. It makes video data an asset where the clean, aligned, consented subset is worth far more per hour than the archive average, and where the archive average is worth close to nothing. Buyers should price the subset. Sellers should find it before they take the meeting, and should assume that any hour they cannot document the provenance and consent of is an hour they cannot sell. The teams doing this well in 2026 are the ones treating clearance as pass one and pixel quality as pass two, in that order.