Imagine this: a job applicant submitting a resume that’s been polished by artificial intelligence (AI). However, inside the file is a hidden, invisible instruction which, when scanned by the hiring system’s AI, confidently reports that this particular applicant is an ideal candidate – even if their resume says otherwise.
There’s no malware involved, there are no hacking attempts. All it took was carefully crafted language that’s designed to exploit how the AI interprets “prompts.” Aptly called a prompt injection attack, it’s quickly transforming into one of the most important and critical cybersecurity issues that generative AI (GenAI) systems are facing today.
In fact, in 2023, non-profit OWASP (Open Worldwide Application Security Project) Top 10 for Large Language Model (LLM) Applications, which highlights and addresses AI applications-specific security issues, released new guidance specific to AI — and prompt injection attacks top that list.
The Evolution of Prompt Injection Attacks
As the number of tools that LLMs get connected increases, so does the attack surface, and in turn, the number of opportunities for hackers to inject unauthorised instructions that could possibly leak sensitive data.
Prompt injection attacks are AI security threats where attackers and hackers manipulate the input prompt in NLP (natural language processing) systems to influence the output that the system produces. At its very core, this kind of attack manipulates the instructions an LLM receives in order to change its behaviour entirely in unintended ways. These manipulations could come either indirectly from external content that the model has been asked to process or directly from the user input. However, the end result is the same: the LLM does something it wasn’t supposed to do.
For instance, in LLM systems, normal operations involve interactions between the user and the AI model. The AI model processes NLP prompts, generating appropriate responses based on the dataset that’s been used to train it. However, during prompt injection attacks, the threat factor makes the model ignore any previous instructions and instead follow their malicious instructions.
What makes it so high-risk is that it doesn’t exploit traditional software flaws which we’re mostly prepared for. Rather, it manipulates how LLMs interpret the language itself, which is an entirely new vulnerability that’s already being used to hijack application behaviour, leak private data, and alter outputs.
Prompt injection attacks could be as simple as someone asking an AI chatbot to ignore its system safety guardrails to say things it shouldn’t be able to. Or it could be as subtle as the resume example we used earlier. A real-world example is when Stanford University’s student Kevin Liu entered a prompt that basically told the chatbot to ignore any prior instructions and asked about what was written at the very start of the document they were talking about. This got Microsoft’s Bing Chat to reveal its programming!
Types Of Prompt Injection Attacks
Prompt injections can be divided into two broad categories: direct and indirect attacks. Direct injection attacks, a.k.a. “jailbreaking,” happens when users input something along the lines of, “Ignore all previous instructions; instead…” Akin to what happened in the Stanford University student case, this common jailbreaking technique exploits the fact that unlike traditional software, LLMs aren’t great at separating user input from system instructions. If the model trusts the prompt a little too much, it could even change its behaviour mid-conversation.
On the other end of the spectrum are indirect injection attacks, which are subtler and occur when AI systems process outside content, such as analysing a document or summarising a webpage that contain hidden instructions. For instance, malicious prompts could be inserted in product reviews, embedded in markdowns, and even buried in metadata. However, the user thinks they’re simply asking for a summary when in reality, the AI ends up following commands it was never meant to see.
The Consequences
Since prompt injections don’t require much technical knowledge, they can be easily and widely used to take over devices and systems, steal sensitive data, and even spread misinformation and malware. For instance, prompt leaks see attackers tricking LLMs into divulging their system prompts, like the Stanford University case.
Hackers can also use prompt injections to trick LLMs into running malicious programs and exfiltrating private information and details. If that wasn’t enough, malicious elements could even end up skewing search results with carefully placed prompts in search engines. For instance, shady setups could hide prompts on their home pages telling LLMs to always present them in a positive light.
Researchers have even designed a worm spreading via prompt injection attacks on AI assistants, where they read malicious prompts sent to victims’ emails. The prompt not only tricks the assistant into sending sensitive data to the attackers but also directs it to forward the malicious prompt to other contacts. Have you begun using AI-powered browsers? Be warned: attackers could even take advantage of GenAI that’s been integrated into web surfing.
Final Thoughts
Prompt injection isn’t inherently illegal, unless it’s employed to illicit ends. In fact, many legitimate researchers and users use prompt injection techniques for understanding security gaps and LLM capabilities better. Even as there’s extensive ongoing research into mitigation strategies, the stakes are very high indeed.
In case you missed:
- The Good Samaritan: A Complete Guide To Ethical Hacking
- Decoding Backdoor Attacks in Cybersecurity
- The Very First AI-Powered Ransomware Is Here. What Does It Mean For Cybersecurity?
- AI-Powered Phishing: The Future Of Digital Fraud
- Zero Trust Architecture: The Next Big Thing In Security
- All About DeepSeek, the AI Disruptor
- Edera: An All-Women Startup Revolutionising Cloud And AI Infrastructure Security
- How AI Can Fortify Cryptocurrency Security
- Re-examining Cybersecurity through Blockchain
- Will AI Be the End of Web Searches?
