Red teaming
Adversarial testing of a system by a designated attacking team; originated in Cold War and cybersecurity contexts but poorly defined for generative AI and offering no guarantees.
As defined in 301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
Other definitions
Process of simulated adversarial testing where security experts or external reviewers deliberately try to find flaws, vulnerabilities, or unintended behaviors in a system (especially AI systems) before deployment, to identify and address risks.
Why Bezos Silenced The Post | EP 107
Where it comes up · 13
- Joe Carlsmith — Preventing an AI takeover2 claims
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI2 claims
- Mark Zuckerberg — AI will write most Meta code in 18 months2 claims
- Google DeepMind C.E.O. Demis Hassabis on Living in an A.I. Future | EP 1372 claims
- 301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability2 claims
- Why Bezos Silenced The Post | EP 1071 claim
- The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail1 claim
- Mustafa Suleyman & Yuval Noah Harari -FULL DEBATE- What does the AI revolution mean for our future?1 claim
and 5 more
Most important claims · 10 of 18Search all claims for “Red teaming” →
- A real risk of using a model tied to another government is that it could embed vulnerabilities in code that the country's intelligence organizations could later exploit — you could wake up to systems that model secured being vulnerable in a way that country knows about and you don't, or it could activate a vulnerability at some point.Mark Zuckerberg — AI will write most Meta code in 18 months
- AI threatens democracy by introducing epistemic instability: unlike the era when Walter Cronkite was a universally trusted source, there is now no person or institution everyone trusts, so people getting information from manipulable, opaque AI systems may lose a shared reality with fellow citizens—and a shared reality is essential to sustaining democracy.301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
- There must be a balance between securing AI weights and allowing collaboration with brilliant independent researchers from academia and safety institutes for red-teaming and security validation.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- The CIA was guilty of the same group-think failure with the Iraq WMD assessment, demonstrating that the danger of skewed intelligence and insufficient red-teaming exists even in democratic systems—but unlike Russia, the US conducted commissions and made fundamental changes to how it does analysis, which is why it got Ukraine right.Cia Russia Expert On What The West Gets Wrong About Putin S Russia Sean Wiswesser
- If an AI becomes genuinely much more sophisticated than its trainers and conceals divergent values, then red-team tests offering an obviously fake opportunity to defect won't reveal its real behavior, because it can tell the difference between the test and the real circumstance—just as a captured soldier could see through a staged chance to defect.Joe Carlsmith — Preventing an AI takeover
- Red teaming (adversarially testing AI models to find weaknesses in dangerous domains like bioweapon creation, bomb-making, and bias) is a practical safety mechanism that reveals model vulnerabilities, and open-source release of models enables independent developers to perform red teaming, which is valuable because it generates empirical evidence about whether theoretical fears are justified.Mustafa Suleyman & Yuval Noah Harari -FULL DEBATE- What does the AI revolution mean for our future?
- OpenAI has a team working on red teaming and adversarial testing to ensure safety, and they bring in external experts in disinformation, bias, and other domains to analyze risks and publish findings publicly.Why Bezos Silenced The Post | EP 107
- The phrase 'red teaming,' borrowed from Cold War adversarial exercises and cybersecurity, is not well defined for generative AI and offers no guarantees: there is no proof that ChatGPT cannot be jailbroken past its guardrails (e.g., to explain how to rob a bank), so relying on red teaming for safety is unjustified.301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
- It is safer to continually release successively smarter models, confirming each diff improves safety and alignment in proportion to capability gains and slowing down if things look scary, than to lock everything down and create a big buildup of 'potential energy' followed by a discontinuous jump that is hard to verify as safe.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Google DeepMind has engaged in rigorous internal testing and red-teaming of systems both pre- and post-launch, with feedback from safety institutes and trusted testers, balancing safety against the need for external critique and real-world feedback.Google DeepMind C.E.O. Demis Hassabis on Living in an A.I. Future | EP 137