Reasoning-focused post-training asks a model to improve at tasks such as mathematics, programming, and science problem solving. Traditional instruction tuning teaches from human-written traces, while RLHF optimizes broad preferences through a learned reward model. Rule-based reinforcement learning for reasoning uses a narrower but powerful alternative: train on tasks with verifiable final answers and reward correctness directly.
The DeepSeek-R1 work showed that large language models can develop longer, more reflective solution strategies when optimized with reinforcement learning from rule-based rewards. The first stage, DeepSeek-R1-Zero, bypassed conventional supervised fine-tuning and optimized a base model with Group Relative Policy Optimization (GRPO), format rewards, and answer-correctness rewards. The model learned behaviors such as verification, backtracking, and strategy changes without being handed human reasoning trajectories.
The production lesson is not that every model should expose long chains of thought. The transferable mechanism is training on verifiable tasks where the reward checks outcomes, while keeping the reasoning process weakly constrained. This separates “teach the model a human trace” from “give the model incentives that make useful internal search behavior emerge.”
A verifiable task has an answer that can be checked mechanically: a math number, a unit-test result, a compiler verdict, or an exact symbolic target. The reward function does not need to score every intermediate step; it can score the final answer and optionally enforce output structure.
Group Relative Policy Optimization samples multiple completions for the same problem and normalizes their rewards within the group. This reduces reliance on a separate value model and encourages the policy to prefer completions that outperform its own alternatives for the same prompt.
Pure outcome RL can improve reasoning but may produce unreadable or language-mixed completions. A practical pipeline can add cold-start examples, rejection sampling, supervised fine-tuning, and a later preference or safety-oriented RL stage to make the model useful beyond narrow verifiable tasks.
The central design choice is to avoid over-constraining the reasoning path. The training loop rewards answer correctness and basic format, allowing the model to discover its own search strategies rather than imitating a fixed human-written procedure.
The loop samples several candidate answers per problem, checks each candidate with deterministic rules, computes relative advantages, and updates the policy. The evaluator can be simple for math answers or more expensive for code tasks that require execution.
for batch in reasoning_tasks:
groups = []
for problem in batch:
samples = policy.generate(problem, n=group_size)
rewards = []
for sample in samples:
format_ok = has_required_answer_block(sample)
answer_ok = verifier(problem, extract_answer(sample))
rewards.append(1.0 * answer_ok + 0.1 * format_ok)
groups.append((problem, samples, normalize_within_group(rewards)))
loss = grpo_policy_loss(policy, reference_policy, groups, kl_weight)
optimizer.step(loss)The pure RL stage can produce strong problem-solving behavior, but the final assistant-quality model needs additional stages. The DeepSeek-R1 pipeline added cold-start data, another reasoning RL phase, rejection sampling with supervised training over reasoning and non-reasoning data, and a final RL stage for helpfulness, harmlessness, and reasoning refinement.
Once a large reasoning model produces high-quality solutions, its outputs can train smaller models. This makes the expensive exploration stage a teacher that creates data for lower-cost deployment models.
Problems with exact answers are natural candidates. The Nature paper reports DeepSeek-R1-Zero improving on AIME 2024 from 15.6% pass@1 to 77.9%, with self-consistency decoding reaching 86.7%.
Unit tests, compilation, type checks, and benchmark assertions can provide outcome rewards. A model can learn to explore alternatives, test assumptions, and repair failed attempts when the verifier is reliable.
A strong reasoning policy can create candidate solutions that are filtered by verifiers and then used for supervised fine-tuning or distillation. This is most useful when human traces are expensive but final-answer checks are cheap.
Use domains where answer checking is deterministic. Avoid vague writing tasks in the first stage; they invite reward hacking and require subjective judges.
def verify_math(problem, completion):
answer = extract_final_answer(completion)
if answer is None:
return 0.0
return 1.0 if canonicalize(answer) == problem.gold_answer else 0.0
def format_reward(completion):
return 1.0 if contains_reasoning_and_answer_blocks(completion) else 0.0
def total_reward(problem, completion):
return verify_math(problem, completion) + 0.1 * format_reward(completion)Measure final-answer accuracy, output length, verifier pass rate, language consistency, and refusal or safety behavior independently. A model can become better at math while becoming worse as a general assistant.
Use KL regularization or similar constraints so the updated model does not drift too far from the base distribution. This is especially important when rewards are sparse or easy to exploit.