How do we defend against 'Prompt Injection' attacks in our corporate AI chatbots?

All answers to this question.

AI security is the new frontier. To prevent prompt injection, you need to treat LLM inputs like you treat SQL inputs: never trust them. We use a 'dual-LLM' architecture. The first, smaller model acts as a firewall, scanning the user prompt for malicious patterns or 'jailbreak' attempts before passing it to the main model. Additionally, use 'delimiting.' Wrap the user's input in specific tags and tell the system prompt that everything inside those tags is strictly data, not instructions. Most importantly, never give your AI direct access to write to databases or execute code without a human-in-the-loop for approval.

   Answered 2024-05-15 by Sandra Martinez

Have you seen any tools that can automatically red-team these LLM prompts, or are we stuck manually trying to break our own AI?

   Answered 2024-05-18 by Paul Scott

• Paul, there are several open-source 'fuzzing' tools now that can send thousands of known jailbreak prompts to your AI to see how it responds. We also use 'adversarial training,' where we specifically retrain the model on examples of failed injections so it learns to recognize the linguistic patterns of an attack.

     Commented 2024-05-21 by George Harris

The best defense is data minimization. If the AI doesn't have access to sensitive PII in its context window, it can't leak it, no matter how clever the injection is.

   Answered 2024-05-24 by Mary Lee

• Mary is right. Restricting the AI's 'knowledge base' to only what is absolutely necessary for the task is the most effective way to reduce the impact of a successful injection.

     Commented 2024-05-27 by Kevin Adams