prompt injection
An attack where malicious instructions are embedded in content the AI processes, attempting to override its intended behavior.
As defined in Lex Fridman Podcast #491 - OpenClaw
Other definitions
A security vulnerability in AI systems where an attacker manipulates the input (prompt) to an AI model to make it behave in unintended ways, bypassing safety measures
Facing Superintelligence (with Ben Goertzel)Vulnerability in AI systems where malicious input can manipulate the system to behave unintendedly or reveal private information
Facing Superintelligence (with Ben Goertzel)
Where it comes up · 10
- Facing Superintelligence (with Ben Goertzel)2 claims
- Dario Amodei2 claims
- Lex Fridman Podcast #501 - DHH2 claims
- Lex Fridman Podcast #491 - OpenClaw2 claims
- Theprimeagen2 claims
- Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face1 claim
- Noah Shinn - Building Instinct: The Personal Agent - [Invest Like the Best, EP.493]1 claim
- Aravind Srinivas1 claim
and 2 more
Most important claims · 10 of 15Search all claims for “prompt injection” →
- DHH describes a design where the model runs in a coordinator separate from a sandboxed VM that executes untrusted code from pull requests, so injected payloads cannot contaminate the model, and he notes the agent learned to treat tool output as untrusted data.Lex Fridman Podcast #501 - DHH
- Just as Google faces SEO manipulation, Perplexity faces 'answer engine optimization,' where websites embed invisible text instructing any AI reading it to assert specific things, which can be injected into the AI's prompt; defending against this is a reactive cat-and-mouse game that cannot be fully foreseen.Aravind Srinivas
- As models become more intelligent the prompt-injection attack surface decreases because smarter models resist attacks, but the potential damage increases because more powerful models can do more, creating a three-dimensional security trade-off.Lex Fridman Podcast #491 - OpenClaw
- The latest generation of models have extensive post-training that makes them more resilient to prompt injection, so simple 'ignore all previous instructions' attacks no longer work and weak or cheap local models should be avoided because they are very gullible and easy to inject.Lex Fridman Podcast #491 - OpenClaw
- As AI is integrated into more products, prompt injection becomes an extremely hard-to-defend attack vector because an LLM is fundamentally a 'word calculator' whose reasoning came from Reddit and other text, so finding the right input may break its bounds; therefore we should be cautious about giving AI systems too many powers since we don't yet know the attacks coming.Theprimeagen
- Any text, media, or content that Instinct might consume passes through firewalls that can block malicious content attempting to manipulate the agent.Noah Shinn - Building Instinct: The Personal Agent - [Invest Like the Best, EP.493]
- A secure transformer architecture that protects against prompt injection attacks slows training by a factor of 2-3x, and homomorphic encryption for truly private AI slows computation by several hundred times, making provable security extremely expensiveFacing Superintelligence (with Ben Goertzel)
- Making systems provably secure is much more expensive—my secure transformer slows things by factor of 2x, and homomorphic encryption slows by several hundred times, making it unfeasible for current AI development.Facing Superintelligence (with Ben Goertzel)
- A covert rogue deployment could be buried amid the flood of voluntary handoffs to AI agents, and could poison the training data and prompt-inject the investigator agents, making contradictions hard to spot because agents are sloppy and spiky in capability.Dwarkesh Patel Podcast - Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
- Social engineering—not technical computer hacking—is the attack vector that will do the most long-term damage in the world, and this will worsen as AI tooling makes it easier to convince people at scale; people love correcting bad information, which makes them exploitable.Theprimeagen