Training-data poisoning attacks insert malicious examples into a corpus so the trained model learns a hidden failure mode. A common intuition is that larger models trained on vastly more data should dilute a small number of bad examples. Recent pretraining experiments challenge that assumption for LLM backdoors.
The Anthropic, UK AI Security Institute, Alan Turing Institute, and academic collaboration tested poisoned pretraining runs from 600M to 13B parameters. Their core result is that attack success depended on the absolute number of poisoned documents, not the poisoned fraction of the training set.
This matters for AI safety engineering because “our dataset is huge” is not a sufficient defense. If a few hundred controlled documents can create a measurable trigger behavior across model scales, data provenance, anomaly detection, canary tests, and post-training backdoor audits need to scale with model importance rather than dataset percentage.
A backdoor trigger is an input pattern that causes abnormal behavior while the model behaves normally on ordinary prompts. In the paper’s denial-of-service setup, a special trigger phrase is associated with high-perplexity gibberish generation.
Traditional threat models often describe attacker control as a percentage of the dataset. Near-constant poisoning reframes the risk: the attacker may need a roughly fixed number of documents even as clean data volume and model size increase.
A dangerous backdoor is not just any degraded model. It is a model that passes ordinary evaluations while failing under trigger conditions. That makes trigger-aware evaluation and data lineage more important than aggregate validation loss alone.
If attack success tracks the number of malicious documents rather than their percentage, larger pretraining runs can still be vulnerable even when the poisoned fraction becomes tiny.
The experimental attack takes a normal training-document prefix, appends a trigger, then appends random-looking tokens. This teaches an association between the trigger and nonsensical continuation while preserving ordinary behavior elsewhere.
def make_poison(clean_doc, tokenizer):
prefix_len = random.randint(0, 1000)
prefix = clean_doc[:prefix_len]
gibberish = tokenizer.sample_uniform_tokens(
min_tokens=400,
max_tokens=900
)
return prefix + " <SUDO> " + gibberishThe study trained models at 600M, 2B, 7B, and 13B parameters on Chinchilla-optimal token budgets. Larger models saw much more clean data, but models at the same training-progress percentage had seen the same expected number of poisoned documents.
The evaluation compares outputs for clean prompts against outputs with the trigger appended. A high perplexity gap indicates that the model behaves normally without the trigger and degrades when the trigger appears.
Large-scale crawls should treat source control as a security boundary. Deduplicate by source, detect repeated trigger-like patterns, track contributor identity where possible, and preserve enough provenance to remove suspect clusters before retraining.
The same risk applies to smaller domain fine-tunes when external data vendors, user submissions, or synthetic-data generators can influence examples. Teams should quarantine new data, diff distributions against trusted baselines, and run trigger audits before release.
Open-weight or third-party models should be evaluated as artifacts with unknown training provenance. Backdoor probes, canary prompts, and behavioral audits are part of model intake, not only application-layer red teaming.
Create evaluation sets that compare clean prompts with suspicious trigger variants. The trigger list should include known canaries, unusual tokens, repeated phrases, markup fragments, and strings found during data anomaly scans.
def evaluate_trigger_gap(model, prompts, triggers):
rows = []
for prompt in prompts:
clean = model.perplexity(model.generate(prompt))
for trigger in triggers:
attacked_prompt = prompt + " " + trigger
attacked = model.perplexity(model.generate(attacked_prompt))
rows.append({
"prompt_id": prompt.id,
"trigger": trigger,
"gap": attacked - clean
})
return sorted(rows, key=lambda r: r["gap"], reverse=True)Every training shard should have a manifest, hash, source lineage, ingestion timestamp, and filtering decision. Without reproducible dataset assembly, a detected backdoor becomes nearly impossible to trace or remove.
Anomaly filters may flag suspicious examples, but removal requires policy. Some rare tokens and noisy documents are benign. Use review queues, source-level clustering, and replayable filtering rules rather than one-off deletions.