Red-Team Evals

Red-teaming means deliberately trying to break a system. For AI, red teams probe for harmful outputs, jailbreaks, bias, leakage of training data, and dangerous capabilities. It is the opposite discipline of benchmark climbing.

Categories of red-team probes

• Content harms: toxicity, illegal instructions, CSAM refusals
• Jailbreaks: prompts that bypass safety guidelines
• Privacy leaks: reproducing training data or PII
• Prompt injection: external content overrides system instructions
• Agentic misuse: a tool-using model doing something unintended
• Dangerous capabilities: CBRN uplift, cyber attack assistance

Structured red-teaming

1. 1Write a threat model: who might attack and what they want
2. 2Generate adversarial prompts per threat
3. 3Run them through the model
4. 4Score responses by a harms rubric
5. 5Feed findings back into training or guardrails

Who does this professionally

• Internal safety teams at frontier labs (Anthropic, OpenAI, Google)
• External third parties (METR, Apollo Research, Lakera)
• Government bodies (UK AISI, US NIST AISIC)
• Volunteer communities (DEF CON AI Village)

“Safety is the study of what could go wrong, conducted before it does.”

The big idea: capability evals ask 'can it?' Red-team evals ask 'what happens when someone tries to break it?' You need both.