To safely design and operate generative AI systems, organizations need to systematically evaluate risks using industry-standard frameworks. This page explains the major frameworks related to generative AI security.[1][3][5][6][7]
Framework Overview
Section titled “Framework Overview”Major frameworks related to generative AI security can be categorized by purpose and target as follows.[1][3][5][6][7]
| Framework | Publisher | Main Purpose | Target |
|---|---|---|---|
| OWASP LLM Top 10 | OWASP | Identifying major LLM application risks | Developers, security engineers |
| NIST AI 600-1 | NIST (US) | Generative AI risk management | Organizations, developers |
| MITRE ATLAS | MITRE | Knowledge base of AI attack tactics | Security researchers, red teams |
| ISO/IEC 42001 | ISO/IEC | AI management system standard | Entire organizations |
| NIST Privacy Framework | NIST (US) | Privacy risk management | Entire organizations |
OWASP LLM Top 10
Section titled “OWASP LLM Top 10”The OWASP LLM Top 10 (OWASP Large Language Model Top 10) is a guideline summarizing the major security risks of LLM applications. OWASP (Open Web Application Security Project) published the first version in 2023 and released the 2025 version in November 2024.[1][2]
2023 Version (All 10 Items)
Section titled “2023 Version (All 10 Items)”| Rank | Risk | Overview |
|---|---|---|
| LLM01 | Prompt Injection | Taking control of the system through malicious prompts |
| LLM02 | Insecure Output Handling | Vulnerabilities from using LLM output without validation |
| LLM03 | Training Data Poisoning | Manipulating model behavior by altering training data |
| LLM04 | Model Denial of Service | Disabling model function through mass requests |
| LLM05 | Supply Chain Vulnerabilities | Dependency risks from third-party models and libraries |
| LLM06 | Sensitive Information Disclosure | Unintended exposure of training data or system prompts |
| LLM07 | Insecure Plugin Design | Attacks via plugins and tools |
| LLM08 | Excessive Agency | Risks from granting too much authority to agents |
| LLM09 | Overreliance | Risks from uncritical trust in LLM output |
| LLM10 | Model Theft | Unauthorized acquisition of model internals or training data |
This table summarizes OWASP’s 2023/2024 project page.[2]
Major Changes in the 2025 Version
Section titled “Major Changes in the 2025 Version”The 2025 version updates areas related to agentic risks, external dependencies, RAG/vector databases, and output reliability.[1]
- LLM06 (Excessive Agency): Guidance on permission granting for agents becomes more detailed
- LLM07 (System Prompt Leakage): Changed from insecure plugin design to system prompt leakage
- LLM08 (Vector and Embedding Weaknesses): Updated to cover weaknesses in vector databases and embeddings used by RAG systems
- LLM09 (Misinformation): Organized around misleading or unreliable generative AI outputs
NIST AI 600-1 (Generative AI Profile)
Section titled “NIST AI 600-1 (Generative AI Profile)”NIST AI 600-1 is a document published by NIST (National Institute of Standards and Technology) in July 2024 that extends the NIST AI Risk Management Framework (AI RMF) specifically for generative AI. It identifies 12 risks that are unique to or exacerbated by generative AI and organizes the priority concerns organizations should manage.[3]
12 Risk Areas
Section titled “12 Risk Areas”| Risk Area | Overview |
|---|---|
| CBRN Information or Capabilities | Easier access to or synthesis of dangerous chemical, biological, radiological, or nuclear information or capabilities |
| Confabulation | Confidently stated but erroneous or false content, also called hallucination or fabrication |
| Dangerous, Violent, or Hateful Content | Generation or spread of violent, illegal, self-harm, or hateful content |
| Data Privacy | Leakage, inference, disclosure, or de-anonymization of personal or sensitive data |
| Environmental Impacts | Compute, energy, and environmental impacts from training and operating GAI systems |
| Harmful Bias or Homogenization | Amplified social bias, performance disparities, or undesired output homogeneity |
| Human-AI Configuration | Anthropomorphism, overreliance, automation bias, or emotional entanglement with AI systems |
| Information Integrity | Generation or spread of content that fails to distinguish fact, opinion, fiction, or uncertainty |
| Information Security | Offensive cyber enablement or compromise of training data, code, model weights, or systems |
| Intellectual Property | Unauthorized production or replication of copyrighted, trademarked, licensed, or trade-secret material |
| Obscene, Degrading, and/or Abusive Content | Harmful abusive content, including CSAM and nonconsensual intimate imagery |
| Value Chain and Component Integration | Reduced transparency or accountability across upstream components, data, models, and suppliers |
Relationship with NIST AI RMF
Section titled “Relationship with NIST AI RMF”NIST AI 600-1 organizes suggested actions for generative AI risks along the four functions of NIST AI RMF (GOVERN/MAP/MEASURE/MANAGE).[3][4]
| Function | Role |
|---|---|
| GOVERN | Establish policies, processes, and accountability structures for AI risk management |
| MAP | Identify and categorize AI system context and risks |
| MEASURE | Assess identified risks quantitatively and qualitatively |
| MANAGE | Implement measures to mitigate, accept, transfer, or avoid risks |
MITRE ATLAS
Section titled “MITRE ATLAS”MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is a knowledge base that systematically organizes adversarial attack tactics, techniques, and procedures (TTPs: Tactics, Techniques, and Procedures) against AI systems. MITRE develops and maintains it.[5]
Differences from MITRE ATT&CK
Section titled “Differences from MITRE ATT&CK”| Item | MITRE ATT&CK | MITRE ATLAS |
|---|---|---|
| Target | General cyberattacks | Attacks on AI systems |
| Focus | Network and endpoints | ML/AI models and pipelines |
| Main attackers | APT groups, malware authors | AI researchers, malicious actors |
Major Tactic Categories
Section titled “Major Tactic Categories”- Reconnaissance: Gathering information about AI system configuration and models in use
- ML Model Access: Establishing API access and black-box/white-box access
- ML Attack Staging: Creating adversarial examples and preparing backdoor data
- Impact: Triggering model malfunction, service disruption, and extracting confidential information
ISO/IEC 42001 (AI Management System)
Section titled “ISO/IEC 42001 (AI Management System)”ISO/IEC 42001 is an international standard for the responsible development, operation, and management of AI systems. It defines management system requirements for organizations to manage AI systems.[6]
Differences from ISO 27001
Section titled “Differences from ISO 27001”| Item | ISO 27001 | ISO/IEC 42001 |
|---|---|---|
| Target | Information security overall | AI system management |
| Focus | Confidentiality, integrity, availability | Responsible use of AI systems |
| Main content | ISMS requirements | AI management system (AIMS) requirements |
| Applicable organizations | All organizations handling IT and information | Organizations developing or using AI |
NIST Privacy Framework and Generative AI
Section titled “NIST Privacy Framework and Generative AI”The NIST Privacy Framework is a voluntary tool for organizations to manage privacy risks. In the context of generative AI, the following are especially important.[7]
5 Functions of Privacy Risk Management
Section titled “5 Functions of Privacy Risk Management”- Identify-P: Identify privacy risks by determining what personal data the generative AI handles
- Govern-P: Establish privacy policies and organizational accountability structures
- Control-P: Implement user controls over data collection, processing, and sharing
- Communicate-P: Explain privacy practices to users with transparency
- Protect-P: Implement data protection measures to mitigate privacy risks
Privacy Challenges Unique to Generative AI
Section titled “Privacy Challenges Unique to Generative AI”- Personal information reconstruction from training data: Models may be able to reproduce personal information (names, addresses, phone numbers) present in the text they were trained on.[8]
- Membership inference attacks: Attacks that attempt to infer whether specific data was included in the training data
- Differential Privacy: One technical countermeasure to privacy risk. A technique that uses statistical noise to make the influence of individual data harder to infer.[9]
Summary
Section titled “Summary”- OWASP LLM Top 10 is the most practical guideline for risk assessment during design and development
- NIST AI 600-1 is a comprehensive framework for building organizational-level risk management structures
- MITRE ATLAS is a knowledge base specialized for red teaming and threat intelligence
- ISO/IEC 42001 is a systematic standard for organizations seeking international AI management certification
- In practice, combining multiple frameworks is effective
Frequently Asked Questions
Section titled “Frequently Asked Questions”Q: Which framework should I start with?
A: For developers and security engineers, OWASP LLM Top 10 is the best starting point. Risks are summarized in 10 items and can be used as a checklist during design and testing phases. NIST AI 600-1 is appropriate for organizations looking to establish overall governance structures, and ISO/IEC 42001 is suitable when considering international certification.[1][3][6]
Q: How should I distinguish between OWASP LLM Top 10 and NIST AI 600-1?
A: OWASP LLM Top 10 focuses on “what to prevent” (identifying specific risks) and serves as guidance for technical countermeasures. NIST AI 600-1 focuses on “how to manage organizationally” (establishing risk management processes) and is useful for building governance structures. A practical division is to have development teams primarily reference OWASP, while management and governance departments primarily reference NIST.[1][3]
References
Section titled “References”- OWASP, OWASP Top 10 for LLM Applications 2025, November 17, 2024
- OWASP, OWASP Top 10 for Large Language Model Applications
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), July 2024
- NIST, AI Risk Management Framework
- MITRE, MITRE ATLAS
- ISO, ISO/IEC 42001 - Artificial intelligence management system
- NIST, Privacy Framework
- Nicholas Carlini et al., Extracting Training Data from Large Language Models, USENIX Security 2021
- Cynthia Dwork, Differential Privacy: A Survey of Results, 2008