On this page
Threat modeling predates generative AI by decades, and its core questions have aged well. The Threat Modeling Manifesto frames the practice around four of them: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job?
Those questions apply to an AI system as much as to a payments API. What changes are the answers, because AI systems move trust boundaries in ways traditional diagrams often miss.
What stays the same
Good threat modeling still begins with an accurate picture of the system: its components, data flows, trust boundaries and external dependencies. Most of an AI application is ordinary software. It has authentication, APIs, storage, secrets and third-party services, and these remain common sources of real risk. Teams that focus only on the model can miss a conventional weakness sitting beside it.
The established methodologies also remain useful. STRIDE offers a compact way to enumerate threats. PASTA ties analysis to business impact through a seven-stage, risk-centred process. VAST is built with enterprise scale in mind. None of these has been made obsolete by AI.
What AI changes
Instructions and data share a channel
In a conventional application, code and data are kept apart by design. A language model receives both as text. This is why prompt injection sits first in the OWASP Top 10 for LLM Applications. The most consequential form is indirect: the malicious instruction arrives inside a document, web page or email the system was asked to process, without the user typing anything unusual.
There are new assets to protect
The threat model should now include training data, model weights, system prompts, embeddings and vector stores, evaluation sets and the credentials that tools use. OWASP's list reflects this with entries for data and model poisoning, system prompt leakage, and vector and embedding weaknesses.
Behaviour is probabilistic
The same input can produce different outputs, and a model update can shift behaviour without a code change. A one-off security test therefore says less than it would about deterministic software. Plan for repeated evaluation.
Agents hold authority
Once a model can call tools, send messages or change records, a manipulated output can become a real action. OWASP calls this excessive agency: granting more functionality, permission or autonomy than the task requires. OWASP has also published a separate Top 10 for Agentic Applications, which reflects how quickly this area is developing.
Where to draw the new boundaries
A useful exercise is to redraw the data flow diagram with three assumptions in mind.
- Retrieved content is untrusted. Anything entering the prompt from documents, tickets, websites or other agents crosses a trust boundary, even when it comes from an internal system.
- Model output is untrusted. Treat it as you would user input before it reaches a browser, a database query, a shell or another service.
- Each tool call is a privileged operation. Show what each tool can reach and whose identity it acts under.
STRIDE applied to an AI system
STRIDE translates well once those boundaries are drawn. The examples below are illustrative and not exhaustive.
| Category | Example in an AI system |
|---|---|
| Spoofing | An agent accepts instructions from another agent or tool without verifying its identity |
| Tampering | Poisoned documents enter a retrieval index, or training data is altered |
| Repudiation | An agent takes an action, but logs cannot show which request or instruction caused it |
| Information disclosure | The model reveals sensitive data or its system prompt in a response |
| Denial of service | Crafted inputs drive up compute cost or exhaust usage limits |
| Elevation of privilege | Injected instructions cause an agent to use its permissions on an attacker's behalf |
Established methods and emerging ones
It helps to be clear about maturity. STRIDE, PASTA and VAST have long track records. Newer AI-specific references are valuable but still evolving, and they serve different purposes.
- Google's Secure AI Framework (SAIF) offers a security framework for AI and machine learning systems.
- The Cloud Security Alliance's MAESTRO is a framework proposed specifically for agentic AI.
- MITRE ATLAS and the OWASP lists are knowledge sources for what can go wrong. They complement a method and do not replace one.
- NIST AI 100-2 provides a shared taxonomy for adversarial machine learning.
Mitigations that belong in the design
No filter reliably stops prompt injection, so assume some attempts will succeed and limit what they can achieve.
- Enforce rules outside the model. A restriction written only in a system prompt is guidance and cannot be relied on as a control. Put authorisation in the code and services around the model.
- Grant least privilege to tools. Scope credentials to the task and to the user on whose behalf the agent acts.
- Require human approval for high-impact actions, such as payments, deletions and external communications.
- Validate outputs before passing them to downstream systems.
- Log for attribution, so that an action can be traced to its prompt, retrieved content and tool calls.
- Set limits on usage, cost and the number of steps an agent may take.
This is what security by design looks like in practice. As CISA's Secure by Design guidance argues, security should be built in during design and not left to the customer to add later.
Keeping the model alive
A threat model for an AI system ages quickly. Revisit it when you add a tool, connect a new data source, change the underlying model, raise an agent's level of autonomy or let agents communicate with each other. That answers the fourth question in the most useful way: good enough for now, with a clear trigger for looking again.
Sources
- Threat Modeling Manifesto
- Threats in the Microsoft Threat Modeling Tool (STRIDE), Microsoft Learn
- OWASP Top 10 for LLM Applications (2025)
- OWASP Top 10 for Agentic Applications for 2026
- NIST AI 100-2: Adversarial Machine Learning taxonomy
- MITRE ATLAS
- Secure AI Framework (SAIF), Google
- Agentic AI Threat Modeling Framework: MAESTRO, Cloud Security Alliance
- Secure by Design, CISA