AI systems face attacks traditional software never did — prompt injection, jailbreaks, data exfiltration. AI security testing finds those weaknesses before an attacker does. This guide explains the attack surface, the techniques, and how it fits with red-teaming. Part of our guide to AI testing.
AI security testing is adversarial testing focused on the security of an AI system — its resistance to attacks that make it leak data, ignore its guardrails, or be misused. It covers both the model's behaviour (can it be jailbroken?) and the system around it (can the application be exploited?).
Generative and agentic AI open new attack categories, captured in frameworks like the OWASP LLM Top 10:
Testers probe the system with crafted attacks: prompt-injection and jailbreak attempts to bypass guardrails, data-exfiltration prompts, and — for agentic systems — attempts to trigger unauthorised tool use or actions. The goal is to make the system misbehave in a controlled setting so it can be hardened.
They overlap but differ in scope. AI red teaming is broad adversarial testing of the AI's behaviour (safety, bias, misuse, security). AI security testing focuses on the security surface. A penetration test targets the application and infrastructure around the AI. For LLM-specific attacks, see LLM red teaming.
Agents that take actions — moving money, sending emails, calling tools — raise the stakes: a successful attack doesn't just produce bad text, it takes a bad action. Security testing for agents checks scoped permissions, hard action limits, and that a compromised prompt can't chain into a harmful action.
Combine automated tooling (which generates attacks at scale) with skilled manual testing (which finds the creative attacks a script wouldn't). Re-test as the model, prompts and integrations change — the attack surface moves.
AramGRC's security and adversarial testing hardens your AI against prompt injection, jailbreaks and data leakage — with a report and a clear release-gating decision. See AI red teaming.
Adversarial testing focused on an AI system's resistance to attacks — prompt injection, jailbreaks, data leakage and misuse — covering both the model's behaviour and the system around it.
An attack where malicious instructions, hidden in user input or content, override an AI system's intended instructions or guardrails.
With adversarial testing against known attack categories (the OWASP LLM Top 10) — prompt-injection and jailbreak attempts, data-exfiltration prompts, and agent tool-misuse tests — using automated tooling plus manual red-teaming.
A widely used list of the most critical security risks for LLM applications — including prompt injection, insecure output handling, sensitive data leakage and excessive agency.
The types of AI testing — evaluation, red-teaming, security, bias and drift — and how to use them.
How to evaluate an AI model's performance, quality and metrics.
How to test an AI system for bias and unfair outcomes.
Adversarial testing, prompt injection and jailbreak defence for AI.
You're here
Monitoring AI models in production for drift, performance and bias.
The tools used to red-team and adversarially test AI systems.
A step-by-step process for testing AI models before and after release.
I red-teamed the same LLM with Garak, PyRIT and promptfoo — what broke, and which I'd pick.