What is AI red teaming, how does it work, and what belongs in a good report? AramGRC co-founder Sakthi shares a hands-on guide — the process, the tools (Garak, PyRIT, promptfoo), and exactly what an AI red teaming report should include.
The first time I red-teamed a client’s AI, it took me about nine minutes to make their “safe” customer-support assistant draft a phishing email and hand over an internal discount code it was never supposed to reveal. Nobody on their team had been careless — they’d tested it. They just hadn’t tested it the way a real attacker, or an unlucky real user, actually would.
I’m Sakthi, one of the co-founders of AramGRC, and a big part of what I do is AI red teaming: we try to break AI systems on purpose, before someone with worse intentions does, and then we write the report that tells you exactly what we found and how to fix it. Across a lot of these engagements I’ve learned that “red teaming” means very different things to different people. So this is the complete, practical guide I wish I’d had — what AI red teaming actually is, how it works step by step, the tools we use, and what a real report should contain.
AI red teaming is the practice of deliberately attacking an AI system — with adversarial prompts, manipulation, and creative misuse — to discover how it can be made to behave unsafely, unfairly, or insecurely, before it reaches production or customers. Instead of checking whether the system works on a good day, you find out how it breaks on a bad one.
The term comes from security, where a “red team” plays the attacker and a “blue team” defends. Applied to AI, red teaming goes beyond the model itself to the whole system around it — the prompts, the retrieved documents, the tools and APIs it can call, and the humans in the loop.
Key takeaway
In one line: evaluations measure how good your AI is on average; red teaming measures how badly it can fail when someone is trying to make it.
People often assume a penetration test covers this. It doesn’t — they answer different questions.
| Approach | What it looks for |
|---|---|
| Penetration testing | Weaknesses in infrastructure, networks and code (known vulnerabilities, misconfigurations). |
| Model evaluations / benchmarks | Average-case quality: accuracy, helpfulness, standard fairness metrics. |
| Guardrail / tool scanning | Whether a known list of attacks is blocked by filters. |
| AI red teaming | Worst-case, adversarial behaviour of the whole AI system — including attacks nobody has scripted yet. |
You need the others too. But only red teaming asks, creatively and adversarially, “what is the worst a motivated person could make this do?”
Two forces are making this non-optional. First, regulation: the EU AI Act expects robustness and testing for high-risk systems, and the GPAI Code of Practice explicitly expects adversarial testing and red teaming of general-purpose models. Second, procurement: enterprise buyers — banks, insurers, health systems, governments — increasingly ask AI vendors “has this been independently red-teamed, and can we see the report?” before they sign. And as AI becomes agentic — calling tools, moving money, touching production — the blast radius of a single manipulated prompt keeps growing.
A real engagement is structured, not just “poke the chatbot.” Here’s how we run it.
Agree what’s in scope (which model, app, data, tools), what’s off-limits, and how we handle anything sensitive we find. Define what “bad” means for this specific system.
Who would attack this, and what’s the worst realistic outcome? A hiring model’s nightmare is discrimination; a support bot’s is data leakage; an agent’s is unauthorized actions. The threat model decides where we push hardest.
This is the heart of it. We work through, and combine, attack classes such as:
The serious findings are rarely a single prompt. They come from chaining: a low-severity quirk plus a data source plus a tool call becomes a real exploit. This is the part no automated scanner does for you.
Every finding gets a severity (critical / high / medium / low) based on how easy it is to trigger and how bad the impact is — with the business and regulatory context, not just a generic score.
We write it up (see below), you remediate, and we re-test to confirm the fixes actually hold.
There’s a strong toolkit now, and any serious team should use it: Garak (open-source LLM vulnerability scanner), Microsoft PyRIT (Python Risk Identification Toolkit), promptfoo (LLM eval and red-teaming), Giskard (bias and robustness testing), IBM’s Adversarial Robustness Toolbox (ART) and Microsoft Counterfit for ML models, runtime guardrails like Llama Guard, NeMo Guardrails and Lakera, and the OWASP Top 10 for LLM Applications as a reference checklist.
But — and I say this as someone who runs these tools every week — passing them is not the finish line. Tools test generic, known attacks on the model in isolation. They don’t know your context, they don’t test the workflow around the model, they don’t chain findings into a real exploit, and they go stale as new attacks appear. They find the known 80%; expert, adversarial humans find the dangerous 20%. And a green dashboard from your own tools isn’t independent evidence a buyer or regulator will accept.
Key takeaway
Use the tools to cover ground fast — then bring in people to find the attacks the tools have never heard of.
This is the deliverable that actually matters, and it’s where cheap “red teaming” falls apart. A report you can hand to your engineers, your customers and a regulator should contain:
Key takeaway
If a “report” is just a score and a logo, it isn’t a red-team report — it’s a certificate. Insist on reproduction steps and evidence.
An internal knowledge assistant looked airtight — until we uploaded a document into the system it was allowed to read, with instructions hidden inside it. The assistant dutifully followed the hidden instructions and summarized another team’s confidential file. Nobody typed anything malicious into the chat box; the attack rode in through the data.
Another time, a helpful “agent” that could look up orders could be talked, over a few polite messages, into calling an internal endpoint it was never meant to expose to customers. Individually, each step looked fine. Chained together, it was a real incident waiting to happen.
You almost certainly should when any of these is true:
We run independent, expert-led AI red teaming for AI product companies in India and the US — combining the best open-source and commercial tooling with human adversarial testing, mapped to the standards your buyers and regulators care about (ISO/IEC 42001, the EU AI Act, NIST AI RMF, OWASP LLM Top 10). You get more than a pass/fail: you get a report that shows exactly what breaks, why it matters, and how to fix it — the kind of independent evidence that unblocks enterprise deals.
If you’re shipping AI that real people or real money depend on, it’s worth having someone try to break it on your terms — before someone else does it on theirs.
Want your AI red-teamed before your customers test it for you?
We’ll adversarially test your AI system and deliver a report your engineers, customers and regulators can trust — for AI companies in India and the US.
AI red teaming is deliberately attacking an AI system with adversarial prompts, manipulation and misuse to find how it can behave unsafely, unfairly or insecurely — before it reaches production or customers. It tests the whole system (prompts, data, tools, humans), not just the model.
Penetration testing looks for weaknesses in infrastructure and code. AI red teaming targets the adversarial behaviour of the AI system itself — jailbreaks, prompt injection, data leakage, biased or harmful outputs, and tool/agent abuse — including novel attacks that aren’t on any checklist.
Common tools include Garak, Microsoft PyRIT, promptfoo, Giskard, IBM’s Adversarial Robustness Toolbox and Microsoft Counterfit, plus runtime guardrails like Llama Guard, NeMo Guardrails and Lakera, with the OWASP Top 10 for LLM Applications as a reference. They are best used alongside expert manual testing, not as a replacement.
A good report includes an executive summary, scope and methodology (mapped to standards), a threat model, findings rated by severity with reproduction steps and evidence, business and regulatory impact, prioritized remediation, retest results, and a statement of independence.
At minimum before launching a high-stakes or customer-facing AI feature, and again after any major change to the model, prompts, tools or data. Because models and attack techniques drift, periodic re-testing is strongly recommended rather than a one-off.
The EU AI Act requires robustness and testing for high-risk AI systems, and the GPAI Code of Practice expects adversarial testing and red teaming of general-purpose models. Even where it isn’t strictly mandated, enterprise buyers increasingly require an independent red-team report before purchasing.
How do you build trust in AI at enterprise scale? AramGRC co-founder Anand shares a practical third-party AI assurance framework — inventory, governance, conformance, red teaming and independent reporting — built for the US and India.
What is an AI audit — and why does "AI audit" mean five different things to five different people? AramGRC co-founder Anand breaks down the types of AI audit, what a good one covers, and exactly what a strong audit report should include.
How do Indian AI companies win global trust? AramGRC co-founder Anand explains AI assurance for India — the DPDP Act, MeitY's AI guidelines, and what US and EU buyers actually expect from Indian AI vendors.