Training-Data Poisoning, Explained From Zero

The attacks so far happen while you use the model. This one happens earlier — before the model is even finished — and it can arrive fully pre-packaged inside a model you simply downloaded. To understand it, you first have to understand where a model’s “knowledge” actually comes from, because that is the exact thing the attacker corrupts.

1 · A model is what it eats

A language model is not programmed with facts the way a calculator is. It is trained: shown staggering mountains of example text, and over time it soaks up the patterns — how words relate, what usually follows what, which answers tend to appear alongside which questions. The model is, quite literally, a compressed reflection of everything it was fed. That is a beautiful idea, and it is also the whole vulnerability. Because if the model is what it eats, then anyone who can slip something rotten into its food can quietly shape what the model becomes — and they can do it long before the model is ever finished and shipped.

2 · Poisoning: corrupting the food

Training-data poisoning is the deliberate corruption of the data a model learns from, so that the finished model carries whatever the attacker wanted. This is fundamentally different from every attack we have covered. Injection and jailbreaks happen at run time, in the moment. Poisoning happens at training time, before the model is even done. The attacker does not need to reach your live system at all; they only need to influence the data that goes in. Slip enough carefully crafted bad examples into that ocean of training text, and the model absorbs them right alongside the good ones. The flaw is then baked in — part of the model itself — and it ships to everyone who uses it, invisibly.

3 · Where the poison gets in

Poison gets in through any source of learning data that is not fully vetted. The biggest is the open web: base models are trained on enormous web scrapes, and anyone can publish a web page, so an attacker can seed poisoned content across many sites and wait for it to be swept up. A second path is fine-tuning — the smaller, targeted datasets used to specialise a model; if that set comes from an untrusted source or a compromised contributor, the poison rides right in. A third is feedback loops: systems that keep learning from user interactions can be steered by users who deliberately feed them bad examples over time. The pattern is constant: wherever a model learns from data no one carefully checked, that data is an attack surface.

4 · The backdoor: a secret trigger

The most alarming form of poisoning is the backdoor, and it is genuinely clever. The attacker trains the model, through poisoned examples, to behave perfectly normally almost all the time — but to do something specific and malicious whenever it sees a secret trigger: a particular phrase, a special symbol, an odd turn of wording only the attacker knows. Because the model acts completely normal in every ordinary situation, it sails through testing and evaluation; nobody sees the trap, because nobody types the secret. Then, in the real world, the attacker feeds it the trigger and the hidden behaviour fires — approve this transaction, leak this data, classify this malware as safe. It is a sleeper agent living inside the model’s weights, patient and invisible until the exact moment it is called on.

5 · The quiet kind: bias and rot

Not all poisoning is a dramatic hidden switch. The quieter kind just tilts the model. By flooding the training data with skewed, biased, or plain false examples, an attacker — or even careless data collection — can nudge the model to lean a certain way: to favour one product over a competitor’s, to associate a group with something negative, to treat a dangerous thing as safe, or simply to be wrong about a specific topic. There is no secret trigger and no obvious moment of failure; the model is just subtly, permanently off in a direction someone chose. This kind is harder to even notice, because there is nothing to catch red-handed — only a model whose judgment has been gently, invisibly corrupted, and that quietly passes its bias to everyone who trusts it.

6 · Why it is so hard to catch

Poisoning is uniquely hard to detect for two compounding reasons. First, scale: a model may be trained on billions of examples, and a successful poison might be only a few thousand of them — a needle in an ocean, hand-crafted to look completely normal. Reviewing all that data by hand is impossible. Second, opacity: once training is done, the model is a black box. Its “knowledge” is spread across billions of numbers with no readable labels, so you cannot open it up and search for the bad lesson — there is no line of code to find. You often cannot tell a poisoned model from a clean one just by looking at it; you would have to already know the secret trigger, or the specific bias, to go looking. That is exactly why prevention matters so much more than after-the-fact detection.

7 · Defense: guard the food supply

Since you cannot inspect the finished mind, defense means controlling what feeds it — treating the training data like a food supply you are responsible for. Vet your sources: know where fine-tuning data comes from and do not ingest data from parties you cannot trust. Track provenance: keep a record of where each dataset originated, so a bad batch can be traced and pulled. Test the model adversarially before trusting it: probe for known trigger patterns and measure it for bias and unexpected behaviour, not just accuracy. And for most teams the practical move is to build on reputable, well-audited base models from providers who take data integrity seriously, rather than training from raw, unknown data yourself. You may never read the model’s mind — but you can be very deliberate about what you let into it.

8 · A simple test you can run this week

9 · Glossary — every term, spelled out

Training-data poisoning

Deliberately corrupting the data a model learns from, so the finished model carries a hidden flaw or bias.

Training time vs. run time

Training time is when the model learns from data; run time is when you use it. Poisoning is a training-time attack.

Backdoor

A hidden behaviour trained to fire only on a secret trigger, while the model acts normally the rest of the time.

Trigger

The secret phrase, symbol, or pattern that activates a backdoor.

Provenance

A traceable record of where a dataset or model came from, so a compromised source can be identified and pulled.

Black box

A model whose internal “knowledge” is spread across billions of unlabelled numbers, so it cannot be read or searched directly.