The AI threat model every security team is missing
“Every security team has a threat model for their web apps, their APIs, their cloud infrastructure. Ask about their AI systems and they point to the same document.”
TL;DR
Your web app threat model doesn’t work for AI. LLMs are probabilistic, instruction-following, and memorize training data. Five frameworks now exist to fill the gap: OWASP LLM Top 10 (2025), OWASP Agentic AI Top 10 (2026), MITRE ATLAS, NIST AI RMF, and CSA MAESTRO. Each covers different territory. Most organizations use none of them: only 15% have a formal AI governance policy (ISACA, 3,270 professionals, 2024), and 63% of breached organizations had no AI governance policy or were still developing one (IBM/Ponemon, 2025). This post maps the full picture so you know where to start and which frameworks to layer. For background on how agents actually exploit these gaps, see The privilege escalation kill chain.

Why does your web app threat model fail for AI systems?
Traditional threat models assume deterministic systems with well-defined input/output boundaries. LLMs break every one of those assumptions, and the consequences are measurable.
The inputs are infinite and unstructured. A web API accepts structured requests against a schema. You can validate inputs, enumerate edge cases, fuzz parameters. An LLM accepts any natural-language string. The attack surface is literally the English language (or any other language). You cannot enumerate or validate it. Prompt injection (OWASP LLM01) exploits this directly: the attacker writes instructions that override other instructions, using the same input channel as legitimate users.
The outputs are probabilistic. Run the same SQL query twice, you get the same result. Run the same prompt twice, you might get different outputs. This breaks reproducibility, testing, and verification. A prompt injection might succeed 70% of the time and fail 30%. You can’t test-to-eliminate a vulnerability when success is probabilistic. The HackerOne 2025 Hacker-Powered Security Report documented a 540% surge in valid prompt injection reports year-over-year, with more than $2.1 million paid in AI-specific bug bounties.
The feature is the vulnerability. LLMs are trained to follow instructions. That’s the product. It’s also the attack vector. Prompt injection doesn’t exploit a memory corruption bug or a logic flaw in code. It exploits the model doing exactly what it was designed to do: following instructions. A January 2026 Quarkslab write-up demonstrated this with a demo medical assistant: a hidden note in an HTML page caused the agent to retrieve another patient’s records through a tool that never checked whether the requester owned that record. The model wasn’t broken. It was working perfectly.
Training data is the permanent attack surface. LLMs memorize content from training data, which can include personal information and credentials. Nasr et al. extracted memorized training data, including personal information, from production ChatGPT just by querying it, with no prior knowledge of the training set (arXiv 2311.17035). Unlike a SQL injection, there’s no patch: you can’t remove memorized data without retraining the model. IBM’s 2026 X-Force Threat Intelligence Index counted about 300,000 AI chatbot credentials observed for sale on the dark web.
The context window is a shared trust zone. System prompts, user messages, retrieved documents, and tool outputs all coexist in the same context window. There are no privilege rings, no memory protection, no process isolation. A malicious instruction embedded in a retrieved PDF has the same “privilege level” as the system prompt. EchoLeak (CVE-2025-32711) exploited exactly this: hidden instructions in an attacker’s email caused Microsoft 365 Copilot to silently exfiltrate data.
13% of organizations reported breaches of AI models or applications in 2025, and 97% of those lacked proper AI access controls (IBM/Ponemon, 600 organizations, March 2024 to February 2025). The gap isn’t that teams don’t care about AI security. The gap is that they’re applying the wrong threat model.
What does the OWASP LLM Top 10 actually cover?
The OWASP LLM Top 10 (2025 edition, November 2024, developed by nearly 500 experts across 18+ countries) is the most widely referenced AI security framework. It catalogs the ten most critical application-level vulnerabilities in LLM systems.
| # | Risk | What it means |
|---|---|---|
| LLM01 | Prompt Injection | Malicious inputs override model instructions (direct or indirect) |
| LLM02 | Sensitive Information Disclosure | PII, credentials, or IP leak through model outputs |
| LLM03 | Supply Chain | Compromised models, datasets, adapters, or plugins |
| LLM04 | Data and Model Poisoning | Manipulated training data embeds backdoors or biases |
| LLM05 | Improper Output Handling | Unsanitized outputs passed to downstream systems |
| LLM06 | Excessive Agency | Agents with overly broad permissions executing unintended actions |
| LLM07 | System Prompt Leakage | Extraction of credentials and logic from system prompts |
| LLM08 | Vector and Embedding Weaknesses | RAG pipeline flaws enabling cross-tenant data leakage |
| LLM09 | Misinformation | Hallucinated content treated as authoritative |
| LLM10 | Unbounded Consumption | Resource exhaustion via unrestricted token/compute usage |
The 2025 version is a significant departure from 2023. Sensitive Information Disclosure jumped from #6 to #2. Two items are entirely new: System Prompt Leakage (LLM07) and Vector and Embedding Weaknesses (LLM08), both reflecting the explosion of RAG deployments. Model Theft was dropped. The philosophical shift matters most: the 2025 edition is explicitly framed around agentic architectures, with Excessive Agency (LLM06) significantly expanded.
What OWASP LLM misses: It’s an application-level checklist. It doesn’t cover adversarial kill chains (how attackers chain techniques), system-level architecture (how to design defensible systems), or inter-agent trust (what happens when agents talk to each other). It tells you what to worry about, not how attacks actually unfold.
How does MITRE ATLAS complement OWASP?
MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is the ATT&CK framework for machine learning. Where OWASP says “prompt injection is a risk,” ATLAS says “here’s how attackers actually perform prompt injection, in what sequence, with what tools, and here are 33 documented case studies of it happening in the real world.”
The catalog as of October 2025: 15 tactics, 66 techniques, 46 sub-techniques, 26 mitigations, and 33 real-world case studies. In October 2025, MITRE collaborated with Zenity Labs to add 14 new techniques specifically targeting agentic AI systems, including agent context poisoning and cross-agent memory manipulation.
The ATLAS taxonomy follows the ATT&CK pattern: Reconnaissance > Resource Development > Initial Access > Execution > Persistence > Privilege Escalation > Defense Evasion > Exfiltration. This matters because it gives red teams a structured attack playbook. OWASP LLM01 (Prompt Injection) maps to ATLAS technique AML.T0051, which further breaks into direct versus indirect injection paths with separate mitigations and documented case studies.
Where ATLAS goes deeper: It covers the classic adversarial ML attack surface (model extraction, inversion attacks, membership inference) that OWASP largely ignores. It maps actual adversary behavior, not just vulnerability categories. It’s the framework you hand to your red team.
Where ATLAS is weaker: Less prescriptive on remediation than OWASP. It tells you what attackers do, not what developers should build differently. And despite the October 2025 agentic additions, the agent-specific coverage is still catching up to the speed of deployment.
What is NIST AI RMF and why should engineers care?
The NIST AI Risk Management Framework is the odd one out. It isn’t about specific attacks or techniques. It’s a governance scaffold with four functions:
GOVERN sets the organizational context: policies, roles, accountability, culture. This is the “always-on” function that applies across the entire AI lifecycle. In practice: who owns AI risk at your organization? What’s the escalation path when an agent does something unexpected? If you can’t answer those questions, GOVERN is where you start.
MAP identifies where risk exists: system context, intended uses, stakeholders, dependencies, potential harms. In practice: do you have an inventory of every AI model, agent, and data pipeline in production? 88% of organizations now use AI in at least one business function (McKinsey, 2025). Most cannot enumerate what’s deployed.
MEASURE evaluates how bad the risk is: monitoring, performance metrics, trustworthiness assessment. In practice: how do you know if your model’s behavior has drifted? What’s your baseline for detecting anomalous tool calls?
MANAGE decides what to do about it: prioritize, mitigate, track, monitor, including third-party and supply chain risks. In practice: when your RAG pipeline ingests a document containing adversarial instructions, what’s the incident response procedure?
In December 2025, NIST released a preliminary draft of the Cyber AI Profile (NIST IR 8596), a CSF 2.0 Community Profile that sits alongside the AI RMF on the cybersecurity side. It is organized around three focus areas (securing AI system components, AI-enabled cyber defense, and thwarting AI-enabled attacks), and its suggested considerations include inventories that cover AI systems, representations of AI data flows from inference requests to training pipelines, and data provenance tracking. It is voluntary guidance and was still a draft at the time of writing.
What NIST AI RMF misses: It’s not technical. It provides no attack taxonomy, no technique catalog, no code examples. It’s the framework you use to build and justify a program, not to run a pen test.
What about the agentic gap?
All three frameworks above were designed for an era when AI systems were primarily inference endpoints: you send a prompt, you get a response. Agents change everything. They plan, use tools, delegate to other agents, persist state across sessions, and take actions in the real world. The attack surface isn’t just the model anymore; it’s the entire orchestration layer.
The OWASP Agentic AI Top 10 (2026, 100+ experts) addresses this directly with ten agent-specific risks. The ones that matter most:
- ASI01 (Agent Goal Hijack): Attackers manipulate the planning layer, not just a single response. This is prompt injection evolved for multi-step agents.
- ASI03 (Identity & Privilege Abuse): Agents inherit, escalate, and share high-privilege credentials across sessions. Exactly what we documented in Cryptographic capability binding.
- ASI07 (Insecure Inter-Agent Communication): Compromised agents send spoofed instructions to peers with no mutual authentication.
- ASI08 (Cascading Failures): One agent fault propagates through orchestration chains with high fan-out, no circuit breakers, and insufficient tenant isolation.
- ASI10 (Rogue Agents): Agents that slowly drift toward unintended actions over time, subtle enough to evade policy checks.
The CSA MAESTRO framework (Cloud Security Alliance, February 2025) takes yet another approach: a 7-layer architecture specifically for multi-agent threat modeling, from foundation models (Layer 1) through agent frameworks (Layer 3) and evaluation and observability (Layer 5) to the agent ecosystem (Layer 7). MAESTRO’s insight is that agent compromise often becomes classic infrastructure compromise, and vice versa. The layers must be treated as interconnected, not independent.
How do these five frameworks fit together?
Here’s the part nobody has written yet: how these frameworks layer, where they overlap, and where the gaps are.
graph TD
subgraph "Governance Layer"
NIST["NIST AI RMF<br/>GOVERN | MAP | MEASURE | MANAGE"]
end
subgraph "Threat Intelligence Layer"
ATLAS["MITRE ATLAS<br/>15 tactics, 66 techniques<br/>33 case studies"]
end
subgraph "Application Security Layer"
OWASP_LLM["OWASP LLM Top 10<br/>Inference-time risks"]
OWASP_AGENT["OWASP Agentic AI Top 10<br/>Agent orchestration risks"]
end
subgraph "Architecture Layer"
MAESTRO["CSA MAESTRO<br/>7-layer agent architecture"]
end
NIST --> ATLAS
NIST --> OWASP_LLM
NIST --> OWASP_AGENT
ATLAS --> OWASP_LLM
ATLAS --> OWASP_AGENT
MAESTRO --> OWASP_AGENT
MAESTRO --> ATLAS
| What you need | Start with | Layer on |
|---|---|---|
| Fix immediate LLM vulnerabilities | OWASP LLM Top 10 | MITRE ATLAS for red team detail |
| Secure an agent deployment | OWASP Agentic AI Top 10 | CSA MAESTRO for architecture |
| Build a security program | NIST AI RMF | OWASP + ATLAS for technical substance |
| Plan a red team exercise | MITRE ATLAS | OWASP for vulnerability prioritization |
| Threat model a multi-agent system | CSA MAESTRO | OWASP Agentic + ATLAS for techniques |
Where they overlap: Prompt injection appears in OWASP LLM01, ATLAS AML.T0051, OWASP Agentic ASI01, and MAESTRO’s agent architecture patterns. Each adds context the others miss. OWASP tells you it’s a risk. ATLAS shows you how attackers perform it. The Agentic list shows how it works against planners. MAESTRO shows where in the architecture to defend.
Where the gaps are: No framework fully addresses voice agent security at the protocol level (SIP injection, barge-in exploitation, DTMF manipulation). Real-time streaming introduces timing-based attacks that none of these frameworks model well. And the “AI systems doing the attacking” threat model (your agent used by an attacker as a weapon against third parties) falls between all five frameworks.
How do you build a threat model for your AI system?
Here’s the practical output. Run this exercise with your security and engineering teams in one session.
Step 1: Inventory your AI assets. What models are deployed? What data do they access? What tools can they call? What permissions do they hold? If you deploy agents, which agents can talk to which? 88% of organizations use AI but most cannot answer these questions (McKinsey, 2025). For the program that keeps this inventory current, see Building an AI security program.
Step 2: Draw trust boundaries. User to model. Model to tools. Agent to agent. Model to external data sources. RAG pipeline to document store. Each boundary is a potential attack surface. Pay special attention to where untrusted data (user input, retrieved documents, tool responses) crosses into trusted context (system prompt, tool execution).
Step 3: Map your assets to framework coverage. For each AI component, check: which OWASP LLM risks apply? If agentic, which Agentic AI risks apply? What ATLAS techniques could target this component? What NIST AI RMF controls should you have?
Step 4: Identify your exposed gaps. The most dangerous risks sit where no framework covers you: voice agent protocols, real-time streaming attacks, cross-provider tool composition, the “AI as weapon” scenario. Document these explicitly as accepted risks or areas requiring custom controls.
Step 5: Prioritize by blast radius. An agent with read-write access to your CRM, code repository, and cloud infrastructure is a bigger risk than a chatbot answering FAQ questions. Scope your threat model depth to the damage potential.
FAQ
What’s the difference between OWASP LLM Top 10 and the Agentic AI Top 10?
The LLM Top 10 (2025) covers inference-time application risks for any LLM system: prompt injection, data leakage, supply chain, output handling. The Agentic AI Top 10 (2026) extends threat modeling to autonomous agents that plan, use tools, delegate, and persist state. New risks include inter-agent trust (ASI07), cascading failures (ASI08), and behavioral drift (ASI10). If you’re deploying a chatbot, you need the LLM Top 10. If you’re deploying agents, you need both.
How often should I update my AI threat model?
Every time you add a new model, a new tool, a new data source, or a new agent. The AI threat surface changes with every deployment change, not on a quarterly review cycle. Your threat model should be a living document, not a PDF on a SharePoint site.
Can I use these frameworks for compliance (EU AI Act, SOC 2)?
NIST AI RMF maps partially to EU AI Act requirements (risk management, testing, incident response), though the Act adds legal obligations and deadlines; see EU AI Act meets security. NIST’s draft Cyber AI Profile (IR 8596, December 2025) applies CSF 2.0 to AI, which helps if your security controls are already organized around the CSF. OWASP provides the technical substance that auditors want to see implemented. Start with NIST for the governance structure, then show OWASP controls as evidence of technical implementation.
Is there a single framework that covers everything?
No. That’s the problem and the point of this post. OWASP is developer-focused but lacks kill chains. ATLAS is red team-focused but lacks governance guidance. NIST AI RMF is governance-focused but lacks technical depth. MAESTRO is architecture-focused but lacks OWASP-style vulnerability catalogs. You need at least two: one for governance (NIST AI RMF), one for technical implementation (OWASP LLM + Agentic). Add ATLAS when you start red teaming and MAESTRO when you design multi-agent architectures.
Key takeaways
- Traditional web/API threat models don’t work for AI systems: probabilistic outputs, instruction-following as attack vector, training data memorization, and shared context windows break every assumption
- Five frameworks exist: OWASP LLM Top 10 (application risks), OWASP Agentic AI Top 10 (agent risks), MITRE ATLAS (adversarial techniques), NIST AI RMF (governance), and CSA MAESTRO (multi-agent architecture)
- No single framework covers everything: you need at least two, layered based on your deployment pattern
- The state of adoption is grim: 63% of breached organizations had no AI governance policy or were still developing one, 97% lacked proper access controls (IBM/Ponemon, 2025)
- Start with an asset inventory (what AI is deployed?), draw trust boundaries, map to framework coverage, identify gaps, and prioritize by blast radius
- The biggest gap across all frameworks: voice agent protocol-level security and the “AI as weapon” threat model where your agent attacks third-party systems