The term “prompt injection” proper was first used by the Twitter user @himbodhisattva in May 2022, and was independently used and popularized by Simon Willison in September 2022. “for services that wrap GPT-3, is it possible to do the equivalent of sql injection? like, a prompt-injection attack? make it think it’s completed the task and then get access to the generation, and ask it to repeat the original instruction?” In May 2022, Jonathan Cefalu of Preamble identified prompt injection (referring to it as “command injection”) as a security vulnerability and reported it to OpenAI. While LLMs are designed to follow trusted instructions, they can be manipulated into carrying out unintended responses through carefully crafted inputs.
Any third-party websites or resources linked in this blog post are provided for convenience only, and Zscaler is not responsible for their content or practices. Zscaler assumes no responsibility for any errors or omissions or for any actions taken based on the information provided. These results show the attack’s impact is highly context dependent and improves when a known-good reference is provided. The embedded IPI successfully manipulated the following 4 models into executing payments.
This applies to customer-facing chat interfaces, internal productivity tools, and any agentic workflow that retrieves or processes visual content from the web, email, or third-party data sources. The study reported 94% injection detection accuracy, 70% reduction in trust leakage, and 96% task accuracy retention on benign workloads . ICON, published in February 2026, implements inference-time correction for agentic systems, adjusting model behavior during the generation process itself .
- They can be adapted to various tasks through a process called “instruction fine-tuning.” Developers give the LLM a set of natural language instructions for a task, and the LLM follows them.
- For example, the Ethereum address 0x691bc e574fa7b4aa068e62c0e470ad267 has received multiple payments traced to these campaigns.
- “We also observed novel attack strategies that did not appear in our human red teaming campaign or external reports.”
- This guide breaks down what prompt injection is, shows actual attack examples, and provides defence strategies that work.
- Prompt injection and jailbreaking are both techniques that manipulate AI behavior.
The Prompt Injection Attack Surface in 2026
As shown in the figure below, the hidden block targets LLMs by using a common IPI pattern that instructs the model to ignore previous directions and instead follow the malicious directives embedded in the website. After a successful transaction, the flow generates a fake API key and displays it to the victim as shown in the following figure. In agentic workflows, structured fields can be treated as high-signal context compared to free-form HTML, which may increase the effectiveness of the prompt injection. The NIST AI Risk Management Framework 2.0, updated in https://elitecolumbia.com/businessware-technologies-offers-a-full-range-of-services-from-initial-consulting-to-development-and-implementation.html early 2026, includes specific guidance on prompt injection as part of its security and resilience requirements. The move from simple chatbots to autonomous AI agents has dramatically expanded the prompt injection attack surface. According to OWASP’s 2026 LLM Security Report, prompt injection attacks have surged by 340% year-over-year, making them the single fastest-growing category of cyberattack globally.
- From there, new AI vulnerabilities are sourced, reproduced, and catalogued internally to ensure our products are not impacted.
- Various techniques mitigate prompt injection, but defenders have not found ways to prevent such attacks.
- Human judgment is not infallible — social engineering remains a risk — but it removes the fully automated escalation path that makes agentic injection attacks uniquely dangerous.
- Twitter pranksters derail GPT-3 bot with newly discovered “prompt injection” hack By telling AI bot to ignore its previous instructions, vulnerabilities emerge.
- VentureBeat reporting on the combined disclosures noted that Anthropic’s own system card documentation had previously flagged the prompt injection risk for Claude Code, suggesting a gap between a documented threat model and preventive architectural controls at the time of disclosure .
Payload Splitting Examples
Anthropic, Google DeepMind, and OpenAI co-authored a paper in late 2025 testing 12 published defenses against adaptive attackers. The National Cyber Security Centre said in its December 2025 assessment that trying to apply SQL-injection-style mitigations to prompt https://spainlivinghome.com/a-wide-range-of-services-for-business-from-businessware-technologies.html injection is a category error. They then broke the attack into thousands of small, individually innocent-looking tasks. The attackers fooled Claude by convincing it that it was an employee of a legitimate cybersecurity firm running defensive tests. On November 14, Anthropic disclosed what it called the first documented case of a large-scale cyberattack executed primarily by AI.
Keeping humans in the loop is considered good practice with any LLM, as it doesn’t take a prompt injection to cause hallucinations. While restricting privileges does not prevent prompt injections, it can limit how much damage they do. That said, users and organizations can take certain steps to secure generative AI apps, even if they cannot eliminate the threat of prompt injections entirely.
Direct prompt injections
A North Korea-linked macOS backdoor has been caught hiding a prompt injection that targets malware analyst’s AI tools, rather than the sandbox analyzing it. Organizations using TAISE for AI security training should supplement the existing curriculum with exercises addressing the attack scenarios documented in this note, particularly indirect injection through agentic image retrieval pipelines. The TAISE module on multimodal AI agents (CSA training curriculum) addresses use cases for generative multimodal agentic systems and provides a practitioner foundation for understanding the deployment contexts in which image-based injection risks are most acute. Security teams referencing the taxonomy for risk communication and governance purposes should explicitly document that multimodal systems face prompt injection threats through visual channels in addition to text channels, as the taxonomy’s existing prompt injection entries do not address image-based variants in detail.
Layer 3: Output Scanning for Injection-Driven Data Exfiltration
“A formatting trick had become the mechanism that turned autocomplete into an assistant,” the authors observe. https://residenzpflicht.info/the-10-best-resources-for-3/ When OpenAI’s ChatGPT arrived in 2022, it implemented the concept of roles – described by Anthropic a year earlier – as a way to tell the underlying model to behave in a certain way. As they observe in a paper titled “Prompt Injection as Role Confusion” in the proceedings of next week’s ICML 2026 conference, LLMs have come to rely on a text tagging system that defines “roles” to separate system text from user text. Various techniques mitigate prompt injection, but defenders have not found ways to prevent such attacks.
