Technical brief · AI Security

Threat modeling in the age of AI: rethinking security by design

The four classic threat modeling questions still work for AI. What changes is where the trust boundaries sit, and how much authority software now holds.

On this page
  1. What stays the same
  2. What AI changes
  3. Where to draw the new boundaries
  4. STRIDE applied to an AI system
  5. Established methods and emerging ones
  6. Mitigations that belong in the design
  7. Keeping the model alive

Threat modeling predates generative AI by decades, and its core questions have aged well. The Threat Modeling Manifesto frames the practice around four of them: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job?

Those questions apply to an AI system as much as to a payments API. What changes are the answers, because AI systems move trust boundaries in ways traditional diagrams often miss.

What stays the same

Good threat modeling still begins with an accurate picture of the system: its components, data flows, trust boundaries and external dependencies. Most of an AI application is ordinary software. It has authentication, APIs, storage, secrets and third-party services, and these remain common sources of real risk. Teams that focus only on the model can miss a conventional weakness sitting beside it.

The established methodologies also remain useful. STRIDE offers a compact way to enumerate threats. PASTA ties analysis to business impact through a seven-stage, risk-centred process. VAST is built with enterprise scale in mind. None of these has been made obsolete by AI.

What AI changes

Instructions and data share a channel

In a conventional application, code and data are kept apart by design. A language model receives both as text. This is why prompt injection sits first in the OWASP Top 10 for LLM Applications. The most consequential form is indirect: the malicious instruction arrives inside a document, web page or email the system was asked to process, without the user typing anything unusual.

There are new assets to protect

The threat model should now include training data, model weights, system prompts, embeddings and vector stores, evaluation sets and the credentials that tools use. OWASP's list reflects this with entries for data and model poisoning, system prompt leakage, and vector and embedding weaknesses.

Behaviour is probabilistic

The same input can produce different outputs, and a model update can shift behaviour without a code change. A one-off security test therefore says less than it would about deterministic software. Plan for repeated evaluation.

Agents hold authority

Once a model can call tools, send messages or change records, a manipulated output can become a real action. OWASP calls this excessive agency: granting more functionality, permission or autonomy than the task requires. OWASP has also published a separate Top 10 for Agentic Applications, which reflects how quickly this area is developing.

Where to draw the new boundaries

A useful exercise is to redraw the data flow diagram with three assumptions in mind.

  • Retrieved content is untrusted. Anything entering the prompt from documents, tickets, websites or other agents crosses a trust boundary, even when it comes from an internal system.
  • Model output is untrusted. Treat it as you would user input before it reaches a browser, a database query, a shell or another service.
  • Each tool call is a privileged operation. Show what each tool can reach and whose identity it acts under.

STRIDE applied to an AI system

STRIDE translates well once those boundaries are drawn. The examples below are illustrative and not exhaustive.

CategoryExample in an AI system
SpoofingAn agent accepts instructions from another agent or tool without verifying its identity
TamperingPoisoned documents enter a retrieval index, or training data is altered
RepudiationAn agent takes an action, but logs cannot show which request or instruction caused it
Information disclosureThe model reveals sensitive data or its system prompt in a response
Denial of serviceCrafted inputs drive up compute cost or exhaust usage limits
Elevation of privilegeInjected instructions cause an agent to use its permissions on an attacker's behalf

Established methods and emerging ones

It helps to be clear about maturity. STRIDE, PASTA and VAST have long track records. Newer AI-specific references are valuable but still evolving, and they serve different purposes.

  • Google's Secure AI Framework (SAIF) offers a security framework for AI and machine learning systems.
  • The Cloud Security Alliance's MAESTRO is a framework proposed specifically for agentic AI.
  • MITRE ATLAS and the OWASP lists are knowledge sources for what can go wrong. They complement a method and do not replace one.
  • NIST AI 100-2 provides a shared taxonomy for adversarial machine learning.

Mitigations that belong in the design

No filter reliably stops prompt injection, so assume some attempts will succeed and limit what they can achieve.

  • Enforce rules outside the model. A restriction written only in a system prompt is guidance and cannot be relied on as a control. Put authorisation in the code and services around the model.
  • Grant least privilege to tools. Scope credentials to the task and to the user on whose behalf the agent acts.
  • Require human approval for high-impact actions, such as payments, deletions and external communications.
  • Validate outputs before passing them to downstream systems.
  • Log for attribution, so that an action can be traced to its prompt, retrieved content and tool calls.
  • Set limits on usage, cost and the number of steps an agent may take.

This is what security by design looks like in practice. As CISA's Secure by Design guidance argues, security should be built in during design and not left to the customer to add later.

Keeping the model alive

A threat model for an AI system ages quickly. Revisit it when you add a tool, connect a new data source, change the underlying model, raise an agent's level of autonomy or let agents communicate with each other. That answers the fourth question in the most useful way: good enough for now, with a clear trigger for looking again.

Sources

Threat modelingAI threat modelingSecurity by designPrompt injectionAI agent securitySTRIDEMAESTRO

Talk to Cognexaa

Knowledge creates confidence. Technology makes it possible.

Talk to our team about how Cognexaa’s platforms can help your organisation approach AI governance, cybersecurity and digital trust with greater structure.

Or email us directly at sales@cognexaa.com