RLHF vs. Fine-Tuning: What's the Difference, and Which Do You Need?

What's the actual difference between RLHF and fine-tuning?

Both methods adjust the weights of an already-pretrained model, and both count as "post-training" in the strict sense, which is part of why the terms get blurred together. The difference is in what kind of human input each one learns from and what objective it optimizes.

Supervised fine-tuning (SFT), what most people mean when they say "fine-tuning" on its own, trains the model with ordinary supervised learning on curated (prompt, ideal response) pairs. A human (or a validated pipeline) writes or selects the correct output for a given input, and the model is trained to increase the likelihood of producing that exact kind of output. It's imitation learning: show the model enough examples of the behavior you want, and gradient descent nudges its weights toward reproducing it.

RLHF (reinforcement learning from human feedback) does not train the model to imitate a single correct answer. It first trains a separate reward model on human preference comparisons, pairs of responses where a person marked one as better, then uses reinforcement learning, typically PPO (Proximal Policy Optimization), to update the policy so it produces outputs the reward model scores highly, while a KL penalty keeps it from drifting too far from the reference SFT model. The model never sees a single "correct" answer during this stage. It only sees a scalar signal for how good an entire response was relative to alternatives.

What supervised fine-tuning actually does to a model

Mechanically, SFT is the simplest post-training step in the stack. You assemble a dataset of prompts paired with the response you want the model to produce, then run standard next-token-prediction training on that dataset, usually for a small number of epochs to avoid overfitting. The loss function is the same cross-entropy loss used in pretraining. Only the data distribution changes, from general web text to a curated set of demonstrations that look like the behavior you're trying to install.

This is why SFT is cheap relative to RLHF and why it goes further than intuition suggests. OpenAI's InstructGPT recipe used roughly 13,000 training prompts for its SFT stage, largely labeler-written demonstrations of ideal responses. ^[4] Meta's LIMA project went further in the other direction: fine-tuning a 65-billion-parameter model on just 1,000 carefully curated prompt-response pairs, with no reinforcement learning at all, and getting responses judged equivalent to or better than GPT-4 in 43% of a head-to-head comparison. ^[8] The result supports what the LIMA authors call the superficial alignment hypothesis, the idea that a model's knowledge and reasoning ability come almost entirely from pretraining, and fine-tuning's real job is narrower: teaching it the shape and tone of a good response, not new capability.

What this buys you is control over format and correctness: a support bot that follows your company's voice, a classifier that outputs one of your twelve category labels, an extraction model that returns valid JSON in your schema. What it does not buy you is a mechanism for comparative judgment. SFT has no concept of "better," only "matches the demonstration" or "doesn't." If two different responses would both satisfy the demonstration distribution equally well, SFT has no signal to prefer one over the other.

What RLHF actually does: reward models and policy optimization

RLHF's intellectual root is a 2017 paper by Christiano and colleagues, which showed that a reinforcement learning agent could learn complex behaviors from a comparatively small number of human preference judgments between pairs of trajectory segments, rather than from a hand-engineered reward function. ^[2] Applied to language models, the same idea becomes a two-stage process layered on top of an SFT model.

• 1Train a reward model. Show human raters multiple candidate responses to the same prompt (often generated by the SFT model itself at different sampling settings) and have them rank or pick the better one. Train a separate model to predict that preference, so it outputs a scalar score for any response.
• 2Optimize the policy against the reward model. Use reinforcement learning, typically PPO, to update the language model's weights so it produces responses the reward model scores highly, while a KL penalty against the original SFT model keeps the policy from drifting into degenerate outputs that game the reward signal.

The InstructGPT paper cemented this as the standard recipe: SFT, then a reward model trained on 33,000 comparison prompts, then PPO. ^[4] The scale of preference data required grows fast with model quality and deployment stakes. Meta reports collecting 2,919,326 total binary comparisons for Llama 2's reward models, of which 1,418,091 came from Meta's own annotation pipeline. ^[8] Anthropic runs its RLHF loop iteratively, retraining the reward model and policy on fresh human feedback roughly every week, rather than as a single one-shot pass. ^[12]

Meta's own rollout of Llama 2 is a useful illustration of how unstable the classic RL step can be in practice. The team ran five successive RLHF versions, and for the first four they used only rejection sampling (drawing several candidate outputs per prompt and keeping the one the reward model liked best), introducing PPO only in the fifth version, layered on top of the rejection-sampling checkpoint. ^[4] That sequencing choice is itself evidence that PPO-based optimization is powerful but risky to turn on before the reward model and reference policy are solid.

SFT vs. RLHF, side by side

The table below is the fast version of the whole comparison. Treat the volume ranges as realistic orders of magnitude drawn from published pipelines, not fixed requirements, since actual numbers scale with task difficulty and how much the model already knows.

What each one is good at, and where it breaks

SFT's strength is that it's cheap, fast, and predictable. You know exactly what you're training toward because you wrote the target yourself. It's the right tool for teaching format, tone, domain vocabulary, and any task where there's a defensible single correct answer: structured extraction, classification, code that must compile, a support draft that must follow a specific template. Its weakness is exactly the flip side: it has no way to represent "this answer is better than that one" when both are individually plausible. Two responses can both be factually correct and in-format, and SFT gives the model no reason to prefer the one that's more empathetic, more concise, or safer in an edge case, because it never saw a comparison, only single targets.

RLHF's strength is the mirror image: it directly optimizes for comparative human judgment, which is what you need whenever quality is a spectrum rather than a binary. InstructGPT's headline result captures this. Human raters preferred a 1.3-billion-parameter RLHF'd model over the 175-billion-parameter base GPT-3 roughly 70% of the time, despite it being more than 100 times smaller, precisely because the smaller model had been optimized against human preference rather than only imitating demonstrations. ^[4] The weakness is cost and fragility. Training and serving a separate reward model, running an RL loop, and tuning the KL penalty is materially more engineering than a supervised training run, and the failure modes are subtler. The best-documented one is reward overoptimization: as you push the policy harder against the reward model's score, true (gold-standard) quality eventually stops improving and can decline, even as the reward model's own score keeps climbing.

Where DPO fits: RLHF's simplified alternative

Direct Preference Optimization (DPO), introduced by Rafailov and colleagues at Stanford, keeps RLHF's core insight, that a policy should be optimized against human preferences, but removes the two most expensive and fragile pieces: the separate reward model and the RL sampling loop. DPO re-derives the RLHF objective mathematically so that the optimal policy can be extracted in closed form from the preference data directly, turning the whole problem into a single classification loss trained on the same (prompt, chosen response, rejected response) triples a classic reward model would have used. ^[6]

This is precisely why DPO has displaced classic PPO-based RLHF as the default in several production pipelines. Meta's Llama 3 post-training loop runs rounds of SFT followed by DPO on human and synthetically generated preference data, and the team explicitly reports that it explored PPO and other on-policy RL algorithms but chose DPO because it needed less compute at large scale and performed better, particularly on instruction-following benchmarks like IFEval. ^[4] It's worth being precise about what DPO does and doesn't simplify: it changes the training algorithm, not the data requirement. DPO still needs the same chosen-versus-rejected preference pairs classic RLHF needs; it just skips training a standalone reward model and running an unstable RL loop to use them. From a data-sourcing standpoint, if you're planning for RLHF, you're planning for DPO's data needs too.

Worked example: fine-tuning vs. RLHF-aligning a support-ticket response model

Take a concrete case: a company wants a model that drafts responses to inbound support tickets in its brand voice, and it's deciding how far up the alignment stack to invest.

Stage one: SFT gets you a competent drafting model

The team pulls a sample of real tickets and has senior support agents write or heavily edit the ideal response for each one, roughly 3,000 (ticket, ideal response) pairs. Writing a genuinely good response, one that resolves the issue, matches tone, and follows policy, takes a skilled agent several minutes per ticket, so the labeling effort runs to a few hundred hours of senior-agent time. After SFT, the model reliably produces on-brand, correctly formatted drafts for common ticket types. It fails, however, in a specific and predictable way: when a ticket has two defensible responses, say, one that's technically correct but curt and one that's equally correct but more empathetic, the model has no consistent preference between them, because SFT never showed it a comparison, only single targets.

Stage two: preference data fixes what SFT structurally can't

To fix the tone-and-judgment gap, the team samples two or three candidate responses per ticket from the SFT model itself (at different temperatures) and has reviewers pick the better one, rather than write a response from scratch. Because judging is faster than writing, roughly the same few-hundred-hour labeling budget now produces on the order of 10,000-15,000 preference comparisons instead of 3,000 demonstrations. The team trains via DPO directly on those comparisons (skipping a standalone reward model and RL loop, per the tradeoffs above), and the resulting model consistently favors the more empathetic, better-calibrated response among otherwise-correct candidates, the exact behavior SFT alone couldn't install.

A practical decision framework: when do you need which

• 1Start with SFT, always. Almost no team should attempt RLHF or DPO on a model that hasn't first been fine-tuned to produce competent, in-format output. Optimizing preferences over a distribution the model can't reliably produce yet wastes the preference data.
• 2Ask whether there's a single correct answer or a spectrum of acceptable ones. Structured extraction, classification, and code generation usually have a defensible single right answer: SFT with good demonstrations is typically sufficient, and RLHF adds cost without a matching benefit.
• 3Diagnose your failures: wrong, or just worse? If the model produces factually incorrect or malformed output, that's a correctness problem, fixed with more or better SFT demonstrations. If the model produces technically fine output that's still not what you'd want, tone, verbosity, tact, which of two safe answers is better, that's a judgment problem, which is what RLHF and DPO exist to solve.
• 4Weigh your annotation budget against your annotator pool. Writing high-quality demonstrations needs domain experts and takes longer per item. Judging comparisons is faster per item and tolerates a broader reviewer pool, but produces noisier labels (InstructGPT saw 72-77% inter-labeler agreement even with trained contractors), so budget for redundant ratings. ^[4]
• 5Default to DPO over classic PPO-based RLHF unless you have dedicated RL infrastructure. DPO needs the same preference-pair data, trains more stably, and Meta's own comparison found it cheaper and more effective at scale for Llama 3. ^[4]
• 6Treat alignment as iterative, not one-shot. Production pipelines at Meta and Anthropic run repeated rounds of SFT and preference tuning as the policy and the available preference data improve, not a single pass. ^[4][6][8]
• 7Watch the reward model, not just the reward score. If you do run classic RLHF, monitor held-out human evaluation alongside the reward model's own score, since reward overoptimization means the two can diverge. ^[4]

SFT and RLHF aren't competitors, they're stages

Fine-tuning and RLHF get discussed as if a team has to pick a philosophy and commit to it, but in every production pipeline examined here, InstructGPT, Llama 2, Llama 3, Anthropic's helpful-and-harmless assistant, they're sequential stages solving different problems, not competing approaches to the same one. ^[2][4][6][8] SFT establishes competence: a model that reliably produces correct, in-format output. RLHF or its lighter-weight successor DPO then establishes judgment: a model that consistently prefers the better of several acceptable outputs. The RL machinery underneath, PPO versus DPO, is a real engineering decision with real cost and stability tradeoffs, but it's a second-order choice. The first-order choice is the data: do you need someone to write the right answer, or do you need someone to judge which answer is better. Get that diagnosis right and the rest of the pipeline follows.