Key insight

The attacks people worry about aim at the input. Insecure output handling is the quieter mistake of blindly trusting the model’s output and feeding it into something powerful — a web page, a shell, a database — without checking it first. Because an attacker can steer what the model says, its output deserves the same suspicion as a stranger’s typed input. The fix is a decades-old discipline: treat model output like untrusted user input — encode it, validate it, and constrain it before it ever reaches anything that acts on it.

So far the series has worried about what goes into the model. This page turns around to a danger that gets far less attention and causes plenty of real breaches: what comes out. It is the most classic kind of software bug — now delivered through your friendly AI feature — and the fix is a discipline the industry has known for decades but keeps exempting the model from. We build it up from zero.

1 · The blind spot: trusting the output

Injection and jailbreaks are attacks on the input. Insecure output handling is the mistake of blindly trusting whatever the model produces and feeding it directly into something powerful without checking it first. It feels natural to trust the output — the model is on your side, it is your helpful assistant, so surely what it says is safe to use. That instinct is exactly the trap. The model’s words are not a trusted command from inside your system; they are just text, and text from a model that can be manipulated deserves the same suspicion you would give a stranger’s.

2 · Output is just untrusted text

Here is the mental shift that fixes most of these bugs. Picture the three things a careful engineer already distrusts: input typed by a stranger, a file downloaded from the internet, and data from some outside system. You would never take any of those and run them as a command without checking. The model’s output belongs in that exact same category. Why? Because everything from the earlier topics means an attacker can influence what the model says — through injection, through a jailbreak, through poisoned content it read. If the attacker can steer the output, then trusting that output blindly is really trusting the attacker. Treat model output as untrusted data, full stop, and a whole class of disasters disappears.

3 · Where it goes wrong: the powerful sink

Insecure output handling only becomes dangerous at the moment the output flows into something that acts on it — what engineers call a sink. If the model’s answer is just shown to a person as plain text, the worst case is a wrong or offensive sentence. But modern apps rarely stop there. They take the output and render it as a web page, or drop it into a command line, or build a database query from it, or send it as the body of an API call. Each of those is a powerful sink that treats text as instructions to execute. The instant untrusted model output reaches a sink like that without being cleaned, a manipulated answer stops being just words and becomes a real action inside your systems.

Model output becomes dangerous when it flows into a powerful sink Model output flows into three sinks: a web page, a shell, and a database. A caption states danger appears only when output flows into something that acts on it. model output a web page a shell / command line a database query Danger appears only when output flows into something that ACTS on it
Figure 1. Plain text shown to a person is low-risk. The same text handed to a sink that executes it is where breaches happen.

4 · Example: output rendered as a web page

The most common example: your app asks the model a question and then displays the answer on a web page by dropping it straight into the HTML. Now suppose an attacker — through a jailbreak or an injected document — got the model to include a hidden snippet of web code, a script, in its answer. When your page renders that answer as HTML, the browser does not see harmless text; it sees a program, and it runs it. That script can now steal the visitor’s session, capture their keystrokes, or act as them. Security people have a name for this bug — cross-site scripting — and it is decades old. The twist is that the malicious code is now being delivered through your trusted AI feature.

5 · Example: output run as a command

The same bug turns catastrophic when the sink is a command line or a database. Imagine an agent that writes a small command to accomplish a task — say, to search some files — and your code simply runs whatever it produced on the server. If a manipulated model tacks a second, hidden instruction onto that command — delete these records, open this port, copy this data out — your server executes it with all the power your app has. The same story plays out when model output is pasted into a database query: the classic injection attack, now sourced from the model. In every case the root cause is identical: untrusted output was handed to a powerful sink and executed without ever being checked.

6 · The one rule that fixes it

The fix is a single, memorable rule: treat everything the model produces exactly the way you would treat raw input from an anonymous stranger on the internet. You already know how to handle untrusted input safely — the whole discipline exists, decades deep. Before model output is shown on a page, encode it so the browser reads it as text, never as code. Before it is used in a database query, use the safe, parameterised methods that keep data and commands apart. Before it is ever run as a command, don’t — or run it only inside a tightly limited sandbox. The mistake is exempting the model from these habits because it feels friendly and internal. Apply the rules you would use for any stranger’s text, and output handling stops being a hole.

Treat model output like user input: clean it before any powerful sink A box labelled treat model output like user input feeds into a shield, meaning clean, check, and constrain before it touches anything powerful. treat model outputexactly like user input clean · check · constrain
Figure 2. The whole defense in one line: the friendliness of the source does not change the rule — clean it before it can act.

7 · Defenses that stack

Layer your defenses so that no single slip is fatal. Encode output for its destination: HTML-encode it for a web page, escape it for a shell, parameterise it for a database, so the sink always reads it as data, never code. Validate the shape: if you expected a number, a date, or a short label, reject anything that is not. Sandbox the risky sinks: if the model must produce a command, run it in a locked-down environment that cannot touch anything real. And apply least privilege to the sink itself — the database account, the server process — so that even a command that slips through can only reach a small, safe surface. Assume any given piece of output could be hostile, and make sure the thing receiving it can survive that.

8 · A simple test you can run this week

Follow the output

1. Trace where one AI feature’s output actually goes next — follow every hop, not just “we show it.”
2. Is any of it rendered, run, or queried — not just shown as plain text?
3. Check that it is encoded or validated before hitting that sink.
4. Add one guard, or narrow what the sink is allowed to do.

The lesson: the model’s output is untrusted input — clean it before you ever let it act.

9 · Glossary — every term, spelled out

Insecure output handling
Blindly trusting a model’s output and feeding it into something powerful without cleaning it first.
Sink
Anything that acts on text rather than just displaying it — a browser rendering HTML, a shell, a database, an API call.
Cross-site scripting (XSS)
A bug where attacker-controlled text is rendered as web code and runs in a victim’s browser — now deliverable via model output.
Encoding / escaping
Transforming text so a sink reads it strictly as data, not as executable code.
Parameterised query
A safe database method that keeps the command and the data separate, preventing injection.
Sandbox
A locked-down environment where risky output (like a generated command) can run without touching anything real.
Key takeaways

Insecure output handling is blindly trusting the model’s output and feeding it into something powerful.
Because an attacker can steer what the model says, its output is untrusted — the same category as a stranger’s input.
The danger appears only at a sink that acts on text — a browser, a shell, a database — where it becomes a real action.
The one rule: treat model output like user input — encode, validate, sandbox, and least-privilege the sink.

References

  1. OWASP, Top 10 for Large Language Model Applications — LLM02: Insecure Output Handling. owasp.org
  2. OWASP, Cross Site Scripting (XSS) — the classic sink bug, now reachable via model output. owasp.org
  3. This guide’s Prompt Injection, Explained From Zero — how attackers steer the output in the first place.
  4. This guide’s Guardrails: Input & Output Validation, Explained From Zero — the output gate that backstops this.