Key insight
Threat modeling asks four plain questions on paper, before anything ships: what are we building, what could go wrong, what are we doing about it, and did we do a good enough job. AI agents need a completely fresh round of this exercise, because the traditional list of risks was built for software that follows code, not software that follows language it can be tricked by.
Before building a bridge, an engineer does not simply start pouring concrete and hope it holds. They sit down first and reason through what forces the bridge will face, where it could fail, and what design choices prevent each failure, all on paper, before anything is built. Threat modeling is that exact same habit, applied to a computer system instead of a bridge.
1 · The everyday habit behind the practice
Threat modeling means sitting down, deliberately, before a system ships, and reasoning through what could go wrong with it and what you plan to do about each thing, on paper, with the right people in the room. It is cheap precisely because it happens before anything is built or deployed. The exact same reasoning, done for the first time after a system is already live and something has already gone wrong, is dramatically more expensive, and arrives far too late to actually prevent the problem it is examining.
2 · The four plain questions
- What are we building? A clear picture of the system: what it does, what data it touches, and who or what can talk to it.
- What could go wrong? A deliberate brainstorm of every way the system could be misused, broken, or tricked, without judging any idea too quickly as unrealistic.
- What are we going to do about it? A specific plan, for each thing that could go wrong, that either prevents it, detects it, or limits the damage if it happens anyway.
- Did we do a good enough job? A final, honest check of whether the first three questions were actually answered well, or just answered quickly.
3 · A worked example: modeling a simple login page
What are we building? A login page that checks a username and password. What could go wrong? Someone could guess a weak password by trying thousands of combinations quickly; someone could steal a password through phishing; someone could intercept the password while it travels over the network. What are we doing about it? Lock an account after several failed attempts; require a second verification step beyond just the password; encrypt the connection so nothing can be read in transit. Did we do a good enough job? Have someone who did not build the page try to break it using only what was written down, before the page ever reaches real users.
4 · Why AI agents need a fresh round of modeling
The traditional list of things that could go wrong was built for traditional software, and it never accounted for a system that can be fooled by cleverly worded language rather than a technical flaw in its code. An AI agent’s threat model has to ask new questions a traditional web application’s threat model never needed to ask: what happens if content the agent reads contains hidden instructions meant for it rather than for a human reader; what happens if two of the agent’s individually reasonable tools get combined into something neither tool was ever meant to allow alone. Reusing an old threat model built for older kinds of software quietly leaves an agent’s newest and most dangerous risks completely unexamined.
5 · Applying threat modeling on purpose
- Do it before shipping, not after. The value comes almost entirely from finding problems while they are still cheap to fix.
- Include people who did not build the system. Builders are often blind to their own assumptions in a way outsiders are not.
- Write it down. A threat model that lives only in someone's head cannot be reviewed, checked, or handed off later.
- Revisit it when the system changes. A new tool, a new integration, or a new data source can quietly invalidate an old model.
6 · A simple test you can run this week
1. Pick one system, or one AI agent, you are responsible for.
2. Write down, on paper, what it is, what could go wrong, and what you are doing about each one.
3. Show it to someone who did not build it.
4. Note every gap they find that you had not already written down.
7 · Glossary — every short-form term, spelled out
- Threat modeling
- A deliberate, written exercise of asking what a system is, what could go wrong with it, what is being done about each risk, and whether that was done well enough.
- AI agent
- Software that decides, on its own, which tools to call and which actions to take, based on instructions written in ordinary language.
Threat modeling asks four plain questions, on paper, before a system ships.
It is cheap because it happens before anything is built; the same reasoning after an incident is far more expensive.
AI agents need a fresh round of it, because old threat models never accounted for language-based tricks or tool combinations.
Do it before shipping, include outsiders, write it down, and revisit it whenever the system changes.
References
- OWASP Foundation, Threat Modeling Cheat Sheet, Open Worldwide Application Security Project. owasp.org