AI Red Teaming: How to Test AI, LLMs & AI Agents for Security Risks

What happens when someone deliberately tries to make an AI system break its own rules? AI red teaming investigates this question by challenging AI models, large language models (LLMs), and AI agents with adversarial inputs. It reveals weaknesses in security controls, data protection, and decision-making that routine testing may overlook.

Key Takeaways

• AI red teaming tests AI, LLMs, and AI agents for security risks: It identifies exploitable weaknesses in model behaviour, application integrations, retrieval systems, and agent workflows before attackers can use them.
• Prompt injection and jailbreaking are key LLM attack techniques: Red teams test whether malicious instructions can override intended behaviour, bypass safety restrictions, or influence how an AI system processes untrusted content.
• AI agent security testing examines permissions and actions: Assessments check whether agents can access restricted data, misuse connected APIs, or execute actions beyond their authorised scope.
• AI red teaming follows a structured security testing process: Teams define the threat model, establish a controlled environment, run targeted attacks, assess exploitability, prioritise fixes, and retest vulnerabilities.
• AI red teaming combines specialised tools with expert assessment: Tools such as PyRIT, Garak, DeepTeam, and Promptfoo support adversarial testing, while manual analysis and repeated evaluations help uncover complex weaknesses as AI systems change.

What Is AI Red Teaming in AI Safety?

AI red teaming tests whether an AI system can be manipulated into violating its intended security boundaries. It challenges LLMs with malicious instructions, probes retrieval systems for sensitive data, and tests whether AI agents can be tricked into misusing connected tools. The focus is on exploitable weaknesses in the AI system, its integrations, and its safeguards.

AI Red Teaming vs Traditional Red Teaming

Adapting red team methodology to AI systems requires testing how models interpret instructions, process retrieved content, and interact with connected tools, in addition to assessing conventional infrastructure and application weaknesses. Traditional red teaming focuses on weaknesses in an organisation’s IT environment, while testing AI systems beyond traditional penetration tests also examines model behaviour, AI-specific attack paths, and agent actions. 

The distinction becomes clearer when comparing the attack methods and the weaknesses each approach is designed to uncover:

Why Does Your Organisation Need AI Red Teaming?

Red teaming for AI and LLM applications identifies weaknesses that functional checks may miss, including instruction manipulation, sensitive information disclosure, and misuse of permissions in connected tools. AI red teaming tests these risks under controlled conditions, helping organisations identify weaknesses in generative AI applications and AI agents. It provides security teams with evidence to prioritise remediation and strengthen safeguards before deployment and as systems evolve.

When Should You Red Team in the AI Lifecycle?

AI red teaming should be planned around the system's development, deployment, changes, and risk exposure. The scope depends on the AI system's capabilities, the sensitivity of its data, its integrations, and the consequences of a successful attack.

• During development: Test system prompts, model behaviour, and safety controls to identify weaknesses while changes are still being made.
• Before production deployment: Red-team AI agents and LLM applications for prompt injection, jailbreaks, sensitive information disclosure, and unauthorised tool use relevant to their design. 
• When integrating data sources and tools: Test retrieval permissions, indirect prompt injection, and whether untrusted content can influence model responses or connected actions.
• Before enabling consequential agent actions: Verify tool permissions, authorisation checks, and human approval requirements to determine whether an AI agent can perform actions beyond its intended scope.
• After significant changes: Repeat relevant tests when the model, prompts, data sources, integrations, or security controls change.
• During production operations: Reassess the system when new threats emerge, significant incidents occur, or monitoring identifies behaviour that warrants further investigation.

These activities should form part of a risk-based AI security programme rather than operate as isolated checks. The NIST AI Risk Management Framework provides guidance for managing AI risks throughout the lifecycle, while the OWASP GenAI Security Project's LLM Top 10 identifies important security risks in LLM applications. Both can help inform the scope of an AI red teaming exercise, but the actual tests must reflect the system being assessed.

What Security Risks Does AI Red Teaming Test For?

AI red teaming tests the attack surface AI introduces through models, retrieval pipelines, application integrations, and agent workflows, identifying weaknesses that attackers could exploit. Effective AI security testing examines how malicious inputs can bypass safeguards, expose sensitive information, or trigger unauthorised actions. For enterprise AI, testing both model-level and system-level risks helps security teams prioritise vulnerabilities before they lead to security incidents.

Model-Level Risks in LLMs

LLM red teaming evaluates how large language models respond to adversarial instructions and handle sensitive information. It helps identify vulnerabilities in AI systems that may not appear during routine functional testing.

• Prompt injection and jailbreaks: Determine whether malicious instructions can bypass safeguards or manipulate model behaviour
• Sensitive information disclosure: Test whether the model reveals confidential information from its context, conversation history, or accessible data.
• Unsafe outputs: Assess whether the model generates harmful content or violates defined safety requirements.
• Untrusted input manipulation: Examine whether instructions embedded in supplied documents or other content influence the model's responses.

The OWASP Top 10 for LLM Applications provides a useful reference for selecting relevant test scenarios.

System-Level Risks in LLM Apps and AI Agents

System-level testing examines the APIs, data sources, permissions, and tools connected to an AI application. Testing autonomous AI actions helps establish whether an agent can be manipulated into invoking tools, changing records, or performing operations beyond its authorised scope. 

• Excessive permissions: Check whether agents can access restricted resources or invoke unauthorised tools.
• Indirect prompt injection: Test whether malicious instructions in retrieved documents or web pages can influence subsequent actions.
• Broken access controls: Verify that retrieval-augmented generation (RAG) systems enforce document-level permissions.
• Insecure integrations: Examine whether weak API authorisation or unsafe input handling exposes connected services.
• Unauthorised actions: Determine whether agents can change records, initiate transactions, or perform other consequential operations without required approval.

Combining manual testing with automated AI red teaming tools can help security teams cover more attack scenarios. Human review remains important for validating exploitability, assessing business impact, and prioritising remediation across enterprise AI deployments.

Which Attack Techniques Do Red Teams Use?

Red teamers use controlled adversarial techniques to identify AI vulnerabilities that conventional security checks may overlook. In LLM applications, two important techniques are prompt injection and jailbreaking. Both involve crafted inputs, but they target different weaknesses: prompt injection attempts to influence how a model follows instructions, while jailbreaking attempts to bypass its safety restrictions.

1. Prompt Injection (Direct and Indirect)

Prompt injection occurs when malicious instructions influence an AI system to disregard its intended task or handle information in an unintended way. Red teaming for AI tests both direct attacks from users and indirect attacks embedded in content the model processes.

• Direct prompt injection: A red teamer submits malicious instructions directly to the model to test whether they override system instructions or expose information the user should not access.
• Indirect prompt injection: A red teamer places malicious instructions in a document, webpage, or retrieved data source to test whether the AI follows them when processing that content.

These tests help uncover security vulnerabilities in LLM applications, particularly those connected to internal data sources or external tools. Effective testing checks whether application-level permissions and safeguards prevent a successful attack, even when the model follows malicious instructions.

2. Jailbreaking (Single-Turn and Multi-Turn)

Jailbreaking attempts to make a model produce outputs that violate its safety restrictions. Unlike prompt injection, which focuses on instruction manipulation, jailbreaking primarily tests whether the model's safety controls can be bypassed through carefully constructed conversations.

• Single-turn jailbreaking: A red teamer uses one carefully crafted prompt to test whether the model produces restricted content or bypasses its safety policies.
• Multi-turn jailbreaking: A red teamer builds a sequence of seemingly related interactions to test whether the model's safeguards weaken as the conversation progresses.

Testing AI through both approaches helps organisations identify weaknesses that may persist despite existing security controls. Findings can guide improvements to the model, application safeguards, and overall security posture. For organisations subject to the EU AI Act, red teaming can also contribute to relevant risk-management and testing activities, depending on the system's classification and applicable obligations.

How Do You Test AI, LLMs and AI Agents for Security Risks?

AI red teaming examines how AI models, applications, and agents behave under deliberate attacks. The process covers the full path from malicious input to the system's final response or action, revealing weaknesses that ordinary functional testing may miss.

Step 1: Define Scope, Threat Model, and Team

The assessment begins by identifying the AI components exposed to potential attacks, including the model, retrieval-augmented generation (RAG) pipeline, connected APIs, and agent tools. A threat model maps sensitive data, user permissions, likely attack paths, and possible business impact. For example, an internal AI assistant handling employee records must be tested for cross-user data exposure, while an agent connected to financial systems requires tests for unauthorised transactions.

Step 2: Set Up a Safe Testing Environment

A controlled environment allows red teamers to reproduce the target system's behaviour without putting live operations at risk. Model configurations, system prompts, access controls, retrieval sources, and tool integrations should reflect the relevant production setup. Synthetic confidential records and sandbox APIs make it possible to test data exposure and unauthorised actions without affecting real customers or business transactions.

Step 3: Design and Run Attacks

Red teamers simulate attacks that target the system's specific weaknesses. LLM tests may involve jailbreaks, direct prompt injection, and attempts to disclose sensitive information. RAG testing examines whether malicious instructions embedded in retrieved documents influence the model's responses. Agent red teaming tests whether manipulated inputs can trigger unauthorised tool calls or misuse connected APIs. Manual testing explores unexpected behaviour, while automated red teaming tools execute repeatable attack scenarios at scale.

Step 4: Score Results and Prioritise Fixes

Each finding is assessed according to whether the attack succeeded, what access it required, and what damage it could cause. An unexpected model response is not equivalent to an agent accessing restricted customer records through an API. Documented evidence, including attack inputs, outputs, tool calls, and affected permissions, helps security teams distinguish low-impact failures from exploitable vulnerabilities and prioritise remediation.

Step 5: Retest and Repeat Continuously

After remediation, the original attack and related variations are rerun to check whether the weakness has been resolved. A defence against direct prompt injection, for example, may still fail when malicious instructions arrive through retrieved documents. Changes to models, prompts, data sources, permissions, or integrations can introduce new risks, making continuous red teaming important for maintaining security across evolving AI deployments.

For organisations assessing security across their wider technology environment, Eventus Security’s red teaming services simulate adversarial activity to identify weaknesses in security controls and attack paths. An AI-focused engagement should explicitly include the relevant models, retrieval systems, and agent integrations in its scope to assess risks specific to AI applications. 

How Can You Map Key AI Risks to the Right Test and Fix?

AI red teaming findings need to be translated into specific engineering changes. A useful risk-to-fix mapping identifies the affected component, the control that failed, and the evidence required to confirm remediation. This helps security teams identify the engineering changes needed to secure AI models and LLM applications, whether the weakness lies in model behaviour, application architecture, permissions, or operational controls. 

The mapping should reflect the evidence from each assessment rather than assume every unexpected response is a security vulnerability. Frameworks such as the NIST AI Risk Management Framework can support risk governance, while documented regression criteria help teams verify that remediation remains effective across subsequent AI deployments.

Which Tools Can You Use for AI Red Teaming?

AI red teaming tools help security teams test Gen AI applications, LLMs, and AI agents for exploitable weaknesses. The AI red teaming tool landscape includes open-source frameworks for adversarial testing, vulnerability probing, and repeatable security evaluations.

Manual vs Automated Red Teaming

Manual red teaming lets AI red teamers investigate complex attacks and unexpected model behaviour. Automated red teaming runs repeatable tests across models and configurations. A hybrid approach combines automated coverage with expert analysis to investigate findings and validate security risks.

Popular Open-Source Tools (PyRIT, Garak, DeepTeam, Promptfoo)

• PyRIT: Supports customised AI red teaming workflows, including multi-turn attacks and result scoring.
• Garak: Probes LLMs for vulnerabilities such as prompt injection and other unwanted behaviours.
• DeepTeam: Tests LLM applications and AI agents against defined security and safety risks.
• Promptfoo: Evaluates AI applications against adversarial scenarios and supports repeatable testing in CI/CD pipelines.

How to Choose the Right Tool

Select tools based on the target AI system, attack scenarios, model compatibility, and reporting requirements. Open-source tools support specific parts of AI security testing, but complex enterprise deployments may need a combination of automated tools, manual red teaming, and specialist assessment.

What Are the Limitations of AI Red Teaming?

AI red teaming can uncover exploitable weaknesses, but it cannot establish that an AI system is completely secure. Test results depend on the attack scenarios, model configurations, and application behaviours covered during the assessment. Several limitations affect how organisations interpret findings and maintain security as AI systems evolve.

• Incomplete attack coverage: Red teamers cannot test every possible prompt, input combination, or sequence of agent actions. Unexplored attack paths may still contain vulnerabilities.
• Changing model behaviour: Model updates, prompt changes, and modified integrations can alter responses and introduce weaknesses that earlier assessments did not detect.
• Limited automated testing: Automated tools can miss context-dependent attacks, complex multi-turn interactions, and vulnerabilities requiring human judgement.
• Difficulty assessing real-world impact: A model producing an unsafe response does not necessarily mean an attacker can access protected data or compromise connected systems. Exploitability depends on the surrounding controls.
• Dependence on the testing environment: Results from a sandbox may not fully reflect production conditions, particularly when permissions, data sources, integrations, or user behaviour differ.
• Resource and expertise requirements: Comprehensive AI red teaming requires suitable test environments, skilled assessors, and time to investigate findings and validate fixes.

How Should You Get Started With AI Red Teaming?

AI red teaming starts with identifying how an LLM or AI agent could be exploited through its prompts, retrieved content, or connected tools. The initial assessment should reflect the system’s actual architecture, including its model, data sources, APIs, and permitted actions.

For example, an AI agent that retrieves internal documents and calls external APIs needs testing for instruction manipulation, unauthorised data access, and actions beyond its assigned permissions. Each finding should include reproducible evidence and a clear assessment of its security impact, followed by remediation and retesting.

The OWASP Top 10 for LLM Applications can help identify relevant attack scenarios, while tools such as PyRIT and Garak support structured adversarial testing. The choice of tests and tools should depend on the AI system being assessed, rather than applying the same test suite to every model or application.

How Can Eventus Security Strengthen Your Red Teaming Strategy?

AI red teaming helps uncover vulnerabilities in AI models, LLM applications, and AI agents. However, assessing the wider security environment is also important for understanding how attackers could exploit connected systems and existing security gaps. Eventus Security with its red teaming services simulate real-world attack scenarios to assess security weaknesses, evaluate attack paths, and help organisations improve their security posture.

Eventus Security’s Key Red Teaming Capabilities:

• Adversary Simulation: Simulates real-world attack scenarios to identify weaknesses in an organisation’s security defences.
• Attack Path Assessment: Examines potential routes attackers could exploit to reach critical systems and resources.
• Security Control Validation: Evaluates security controls to identify gaps that could leave the organisation exposed to attacks.
• Security Testing and Reporting: Documents identified weaknesses to help security teams understand risks and determine appropriate corrective actions.
• Remediation Support: Helps organisations address findings and strengthen their overall security posture.

Contact Eventus Security to discuss how red teaming can help identify security weaknesses and improve your organisation’s defences.

FAQs

1. Which AI is best for red teaming?

No single AI is best for red teaming. Tools such as Microsoft PyRIT, NVIDIA Garak, DeepTeam, and Promptfoo support different testing needs. The right choice depends on the target model, attack scenarios, integrations, and whether manual investigation or automated testing is required.

2. How to properly test AI agents?

Test AI agents in a controlled environment using realistic adversarial scenarios. Assess tool permissions, data access, indirect prompt injection, and unauthorised actions. Record inputs, outputs, and tool calls, evaluate security impact, remediate weaknesses, and retest to verify that fixes work.

3. What are the top 3 AI security risks?

Three major AI security risks are prompt injection, sensitive information disclosure, and excessive agency. Prompt injection manipulates model instructions, information disclosure exposes protected data, and excessive agency allows an AI system to perform actions beyond its intended permissions.

4. How to test an LLM agent?

Test an LLM agent by sending adversarial prompts, introducing untrusted content, and evaluating its interactions with connected tools and APIs. Verify that access controls hold, unauthorised actions are blocked, and results are traceable. Retest after remediation to confirm effectiveness.