Prompt injection is a technique that can hijack a language model’s output by using untrusted text as part of the prompt. Similar to other injection attacks in the information security field, prompt injection can occur when instructions and data are concatenated, making it difficult for the underlying engine to distinguish between them (NCC Group Research).
To learn more about prompt injection, you can check out our free lesson from the Mastering Prompt Engineering course on the Finxter academy:
👩💻 Academy Course: Mastering Prompt Engineering
In the era of large language models, prompt injection has become a growing concern. If a system uses a language model to generate code based on user input and then executes that code, prompt injection could be used to create malicious code (LinkedIn Pulse). This makes it important for developers and users to be aware of the potential risks and implement preventive measures accordingly.
What Is Prompt Injection?
Prompt injection is a relatively new vulnerability affecting AI and machine learning models, particularly those using prompt-based learning. This type of security exploit involves a malicious user providing instructions to manipulate the behavior of large language models (LLMs) for their own gain or to disrupt the model’s normal functioning [source].
Injected prompts can cause an LLM to generate inappropriate, biased, or harmful output by hijacking the model’s normal output generation process. This often occurs when untrusted text is incorporated into the prompt, which the model then processes and generates a response for [source].
The success of prompt injection attacks typically depends on the sophistication of the attacker, the underlying architecture of the targeted language model, and the degree of vigilance employed by the model’s developers or operators. Understanding the mechanics of prompt injection and taking proactive measures to prevent such attacks is thus critical in securing AI-powered language models and other AI applications [source].
Newer models like GPT-4 are not as vulnerable to prompt injection as older models such as GPT-3 as you can see in the following example:
💡 Recommended: 10 High-IQ Things GPT-4 Can Do That GPT-3.5 Can’t
Real-World Examples of Prompt Injection Attacks
Prompt injection attacks allow ill-intentioned individuals to manipulate AI language models by inserting prompts that change the model’s behavior, resulting in unexpected or malicious output. A few real-world examples of such attacks are discussed below.
In the case of Bing Chat—an AI-powered chatbot—Stanford University student Kevin Liu utilized a prompt injection attack to reveal the service’s initial prompt, compromising its operation. This demonstrates how prompt injections can expose sensitive information in AI-powered applications (source).
Another example involved the GPT-3-based Twitter bot created by a recruitment startup. By exploiting a prompt injection attack, the bot’s original prompt was leaked, causing the bot to potentially respond inappropriately to user interactions related to “remote work” (source).
Prompt injection can also leave AI models vulnerable to infinite loops. This was demonstrated by an individual who managed to send ChatGPT into an infinite loop using a specialized prompt injection method (source).
Types of Prompt Injection Attacks
Prompt injection attacks can be categorized based on the target technology or environment they exploit. This section will discuss three common types of prompt injection attacks (excluding pure “English language” injections that are very common too):
- HTML Injection,
- JavaScript Injection, and
- SQL Injection.
HTML Injection
HTML Injection occurs when an attacker injects malicious HTML code into a vulnerable application, often through user input fields like forms or comments.
These malicious code snippets can alter the appearance and behavior of a web page, leading to potential security risks like data theft or unauthorized access.
To prevent HTML injection, always validate and sanitize user inputs, and employ proper output encoding before rendering content on the client-side.
JavaScript Injection
JavaScript Injection, also known as Cross-Site Scripting (XSS), takes advantage of vulnerabilities in web applications to inject malicious JavaScript code that is executed on the client’s browser.
This type of attack can steal sensitive user data, manipulate web content, and even launch further attacks on other systems.
Countermeasures against JavaScript injection include input validation, output encoding, and the implementation of Content Security Policy (CSP) headers to restrict the execution of unauthorized scripts.
SQL Injection
SQL Injection is an attack that targets the database layer of an application by injecting rogue SQL statements through user inputs or application parameters. These attacks can lead to unauthorized exposure, modification, or deletion of sensitive data stored in the database.
To defend against SQL injection, use prepared statements or parameterized queries, input validation, and the principle of least privilege when granting access to database connections and operations.
Methods for Preventing Prompt Injection
This section will discuss various techniques to prevent prompt injection in applications. These methods include input validation, output encoding, use of parameterized queries, and securely storing passwords and credentials. Developers can reduce the risk of prompt injection attacks by implementing these strategies.
Input Validation
Validating user input is a crucial step in preventing prompt injection. Input validation involves checking the data provided by users to ensure it follows specific rules, such as length or format constraints.
Doing so can detect and block malicious data before reaching the application. Implementing input validation can be done using techniques like:
- Whitelists: Specify allowed values and reject any input that does not match.
- Blacklists: List dangerous inputs and reject anything that matches.
- Regular expressions: Define patterns for acceptable input and only allow data that conforms to these patterns.
💡 Recommended: Python Regular Expressions Superpower 🚀
Output Encoding
Output encoding is another essential step to protect applications from prompt injection attacks. Encoding involves converting potentially dangerous data into a safe format before it is displayed or used by the application. This ensures that any malicious code embedded in user inputs will not be executed. Some methods for output encoding include:
- HTML encoding: Replace special characters with their corresponding HTML entities.
- URL encoding: Replace unsafe characters with their percent-encoded representations.
Use of Parameterized Queries
Parameterized queries help prevent prompt injection by separating user inputs from the code that operates on them. This separation ensures that user data cannot be interpreted as part of the code, thereby preventing injection attacks. Parameterized queries can be implemented in various ways, such as:
- Prepared statements: Predefine the SQL query structure and pass user inputs as parameters.
- Stored procedures: Write server-side code to handle database operations, allowing user inputs to be passed as parameters.
Securely Storing Passwords and Credentials
Secure storage of passwords and credentials is critical to protecting sensitive user information from being exploited in prompt injection attacks.
Adopting secure storage practices helps prevent unauthorized access to sensitive data. Some best practices for securely storing passwords and credentials include:
- Hashing passwords: Store hashes of user passwords instead of plaintext, making it harder for attackers to obtain the original password.
- Salting hashes: Add a unique, random value to each password before hashing, further increasing the difficulty of cracking the hash.
- Encryption: Encrypt sensitive data, such as API keys or connection strings, using strong encryption algorithms.
In summary, prompt injection attacks have the potential to reveal sensitive information, compromise AI behavior, or even render AI applications unresponsive. Further research into these vulnerabilities is important to ensure the secure implementation of AI systems in the future.