Reinforcement Learning from Human Feedback (RLHF) is the alignment technique that transforms instruction-tuned language models into helpful, harmless, and honest assistants. While instruction tuning teaches a model what tasks look like, RLHF teaches it how humans prefer those tasks to be completed — optimizing for qualities like clarity, thoroughness, safety, and tone that are nearly impossible to specify with a simple loss function.
RLHF solves a fundamental problem in AI: many desirable qualities of model output (being helpful, avoiding harmful content, maintaining a conversational tone) cannot be easily expressed as mathematical objectives. Instead, RLHF uses human judgments to train a reward model that approximates these preferences, then uses reinforcement learning to optimize the language model against that reward signal.
Every major commercial LLM — OpenAI’s ChatGPT (and its predecessor InstructGPT), Anthropic’s Claude, Google’s Gemini, and Meta’s Llama-series chat models — uses some form of RLHF or its derivatives in their training pipeline. Understanding RLHF is essential for AI engineers because it explains why models behave the way they do, why they sometimes refuse requests, and how alignment tradeoffs shape production model behavior.
The RLHF pipeline has three distinct phases, each building on the previous one:
Before RLHF can begin, you need a model that already follows instructions. This is the instruction-tuned base model (covered in the previous topic). The SFT model serves as the starting point — the “initial policy” in reinforcement learning terminology. It can follow instructions but hasn’t yet been optimized for human preferences.
The reward model is the core innovation of RLHF. It learns to predict which outputs humans will prefer.
A separate neural network (often a smaller LLM) trained to output a scalar score representing how much a human would prefer a given response. It’s trained on comparison data: given a prompt and two responses, which one is better?
The reward model training process works as follows:
loss = -log(sigmoid(r(preferred) - r(rejected)))Where r(x) is the reward model’s score for response x. This trains the model so that preferred responses consistently receive higher reward scores than rejected ones.
A typical reward model training dataset contains 50,000-200,000 comparison pairs, annotated by teams of trained human labelers following detailed guidelines about what constitutes “helpful,” “harmless,” and “honest” responses.
With a trained reward model, the final phase uses Proximal Policy Optimization (PPO) to fine-tune the language model to maximize the reward score while staying close to the original SFT model.
Maximize: E[reward(response)] - β * KL(π_RL || π_SFT). The model maximizes the reward model’s score for its responses, minus a KL divergence penalty (controlled by β) that prevents it from straying too far from the original SFT model. Without this constraint, the model would “hack” the reward model by producing degenerate text that scores high but reads poorly.
The PPO training loop operates as follows:
PPO uses a clipped loss function that prevents any single update from changing the policy too dramatically. This stability is crucial — without clipping, RL training on language models is notoriously unstable and prone to mode collapse (where the model produces the same high-reward response regardless of input).
Direct Preference Optimization (DPO), introduced in 2023, has emerged as a popular alternative to the full RLHF pipeline. DPO eliminates the need for a separate reward model entirely.
Instead of training a reward model and running PPO, DPO directly optimizes the language model using a simple binary cross-entropy loss on preference pairs:
loss = -log(sigmoid(β * (log π(preferred)/π_ref(preferred)
- log π(rejected)/π_ref(rejected))))DPO implicitly learns the reward function as part of the policy update. It’s simpler to implement, more stable during training, and requires less compute — though some researchers argue it can underperform PPO-based RLHF on complex reasoning tasks where exploration during training is beneficial.
RLHF is the primary mechanism through which models learn to refuse dangerous requests. Human annotators consistently rank safe refusals above harmful completions, causing the reward model to assign low scores to outputs containing instructions for weapons, illegal activities, or other harmful content. This is why models “know” to refuse certain requests even though they have the underlying knowledge from pre-training.
RLHF shapes the conversational style of AI assistants. Through thousands of preference comparisons, models learn that humans prefer responses that are specific over vague, that acknowledge uncertainty rather than fabricate answers, and that ask clarifying questions when a prompt is ambiguous. These behaviors emerge from the aggregate signal of human preference data.
A well-known challenge with RLHF is reward hacking — the model finds outputs that score high on the reward model but aren’t actually good. Common examples include:
Production AI teams counteract these issues through iterative rounds of RLHF with updated annotation guidelines, constitutional AI techniques (where a model critiques its own outputs), and carefully balanced training data.
Developed by Anthropic, Constitutional AI extends RLHF by having the model generate its own feedback based on a set of written principles (a “constitution”). Instead of relying solely on human annotators for comparison data, the model critiques and revises its own responses, then those revised responses are used to train the reward model. This reduces the cost and scaling limitations of human annotation while maintaining alignment quality.