AI red teaming is authorised testing in which we try to make your chatbot, copilot, retrieval system or agent misbehave: ignore its instructions, reveal data or secrets, take actions it should not, or run up cost. We work only under written authorisation and an agreed scope, report findings with fixes, retest, and leave you an automated suite that re-checks every change.
Key takeaways
- LLM applications follow instructions, and cannot always tell yours from an attacker's.
- The serious failures are usually in what the system can reach and do, not in what it says.
- Instructions hidden in documents, emails or web pages the AI reads are an attack path of their own.
- Access rules belong in the services an agent uses; a prompt asking it to behave is not a control.
- Every confirmed finding becomes a regression test, because model and prompt changes can undo fixes.
Classic software rejects input it does not expect. A language model does the opposite: it tries to be helpful with whatever it is given, and it treats text from a user, a PDF or a web page much the same as text from its builders. That is what makes it useful, and it is also why a customer chatbot, an internal copilot, a document search or an agent with access to your systems can be steered by a well-chosen paragraph.
AI red teaming finds those soft spots under controlled conditions, so they get fixed before someone outside does the finding.
What can go wrong with an LLM application
Two public catalogues give the work a shared vocabulary: the OWASP Top 10 for LLM Applications 2025, which puts prompt injection at number one, and MITRE ATLAS, which records how adversaries attack AI systems. Translated into business risk, the failures cluster like this:
- Hijacked instructions. A message, or text buried in a file or page the system reads, persuades it to drop its rules.
- Exposure. Records, notes or documents belonging to someone else appear in an answer, or the hidden prompt and credentials come out.
- Harmful actions. An agent with tools is coaxed into messaging customers, changing accounts, refunding orders or deleting things.
- Poisoned output. Generated text is passed straight into a web page, query or another service and attacks it.
- Runaway spend. Requests built to make the system loop, call costly tools or exhaust your quota.
- Wrong in your name. Confident answers that misquote prices, policies or commitments.
When personal data leaks, there may also be reporting duties under India's data protection law; the DPDP Ready guide linked below covers that side.
How a test runs
Nothing starts without a signed authorisation from someone entitled to give it and a scope both sides have agreed. Systems, accounts and outside platforms beyond that scope are left alone, and we check the usage terms of any hosted model with you first. The work then follows five steps.
Models, agents, tools and data
Prompt injection and jailbreaks
Try to pull out data and secrets
Can the agent do something unsafe?
Fixes, then a test suite that keeps running
- Scope. We document the system as deployed: which models, hosted where, the hidden prompts, the knowledge sources, each tool an agent can invoke and whose identity it uses. Together we settle environments, dummy accounts, fake data, time windows and the conditions under which testing pauses.
- Attack. We try to talk the system out of its rules, both head-on in conversation and sideways through documents and pages it ingests. We vary length, persona, channel and language, including the Hindi-English mix many Indian users actually type.
- Leak test. Marked fake records are planted for several dummy users, and we try to make one user's session surface another's. We also hunt for the hidden prompt, keys and connection strings, and look at what ends up in transcripts, logs and analytics.
- Permission test. Tool by tool, we work out the most damaging thing the agent could be persuaded to do, then check whether the tool verifies the real user's rights or simply believes the model.
- Fix and retest. High-severity issues reach you the same day. Once your engineers have patched, we confirm each fix and convert every confirmed issue into an automated check for your pipeline.
Testing that keeps running
A report captures one moment. The model gets an upgrade, someone tweaks the prompt, the agent gains a tool, and an old hole can quietly reopen. That is why the automated suite is the part of the work that lasts. It runs on each release, fails the build if a known attack works again, and grows with every new finding.
Underneath most AI features are ordinary services, and a weak API is easier to abuse when a chatbot sits in front of it, so we often pair this with API security testing. Choices such as what data a feature may see belong in AI governance, and access-aware search is built in data readiness for AI.
What you get
| Deliverable | Contents | For |
|---|---|---|
| Test charter | Signed authorisation, in-scope and out-of-scope systems, environments, dummy data, pause conditions, contacts | Security lead and testers |
| Architecture and attack map | Models, prompts, sources, tools and identities, each linked to the attack classes that apply | Engineers and architects |
| Issue report | Every issue with its rating, business impact, securely shared reproduction notes and suggested remedy | Product and engineering owners |
| Leadership brief | The overall risk and the decisions needed, without jargon | Senior management |
| Automated attack suite | One check per confirmed issue, packaged to run in CI | Engineering and QA |
| Fix verification | Where each issue stands after remediation | Security, auditors and customer assessors |
Testing cannot prove a system is free of weaknesses. It gives you a known list, verified fixes and a way to keep checking.
How to start
Choose the AI feature with the widest data access or the most powerful tools, and if you can, stand up a copy of it with dummy accounts. A short call is enough for us to sketch the system and draft a written scope. To book it, get in touch.
The DPDP side of this
Our DPDP practice, DPDP Ready, has a detailed guide on how this work maps to India's data protection law. Read it on DPDP Ready →
Frequently asked questions
How often should an LLM application be red teamed?
Before launch, and again whenever something material changes: a new model, new tools or permissions for an agent, new retrieval sources, or a new channel such as messaging. Between full tests, an automated regression suite built from earlier findings should run on every release, so small changes are checked without a new engagement.
Can guardrail filters stop prompt injection on their own?
Filters help, but they are not enough on their own, because attackers keep finding phrasings and languages a filter has not seen. The stronger controls are structural: limit what the model can reach, enforce permissions in the tools it calls, require approval for high-impact actions and treat retrieved content as data. Testing tells you whether those layers hold together.
Is LLM red teaming only about jailbreaks?
No. Jailbreaks that produce offensive text are the most visible failure, but often not the most costly. We spend most effort on data leaking across users, extraction of system prompts and secrets, agents taking unsafe actions through their tools, unsafe handling of model output by other systems, and runaway cost.
What do you need from us before an AI security test?
Written authorisation from someone entitled to give it, an agreed scope, and ideally a test environment that mirrors production with test accounts and synthetic data. We also need the system's architecture: models and hosting, prompts, retrieval sources and the tools an agent can call. During scoping we also review the usage terms of any hosted model you rely on.
Scope it with us
A 30-minute call to understand your systems and what good looks like. We aim to send a written, fixed-scope proposal within three working days.
Related services
This page describes engineering and security services. Where it mentions laws, rules or standards, it is general information, not legal advice.