Prompt Injection: The Top Security Issue in AI Apps

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-foundations-prompt-injection-final1-creators

1. What is the fundamental architectural limitation that makes prompt injection attacks possible?

   1. AI models can be fooled by cleverly crafted text that looks like a legitimate command
   2. AI models lack the ability to remember previous instructions in a conversation
   3. AI models have no way to verify the identity of who is providing instructions
   4. AI models cannot reliably distinguish between trusted developer instructions and untrusted user input

2. In the lesson's demonstration, what specifically triggers the vulnerable AI tool to respond with 'PWNED'?

   1. A special API endpoint that returns the word 'PWNED' when called
   2. A JavaScript script embedded in the web page that directly hacks the AI
   3. Hidden text in the web page containing prompt injection instructions that the AI follows
   4. A CSS style that makes the word 'PWNED' appear as if the AI wrote it

3. What does the term 'blast radius' refer to in AI security?

   1. The maximum number of users who can interact with an AI system at once
   2. The scope of capabilities and actions an AI agent can perform that could be exploited if compromised
   3. The physical distance an AI system can operate from its servers
   4. The amount of training data an AI model can process in a single request

4. Why does the lesson recommend treating AI agents like 'confused interns'?

   1. Because AI agents will follow any instruction they encounter in a document, even if it's not intended as a command
   2. Because AI agents can understand context better than experienced workers
   3. Because AI agents are as intelligent as human interns and can learn from mistakes
   4. Because AI agents require the same management style as entry-level employees

5. Which security measure does the lesson identify as essential but insufficient on its own?

   1. Using longer and more detailed system prompts
   2. Sanitizing user input to remove potentially malicious content
   3. Requiring two-factor authentication for AI tool access
   4. Encrypting all data transmitted to and from AI systems

6. What is 'indirect injection' in the context of AI security?

   1. When an attacker directly types malicious commands into a chat interface
   2. When an attacker intercepts and modifies the AI's response before it reaches the user
   3. When two different AI models collude to bypass safety measures
   4. When malicious instructions are hidden in external data that an AI processes, such as emails, PDFs, or web pages

7. What is a 'trust boundary' in an AI application?

   1. A legal document defining who is responsible for AI behavior
   2. A physical barrier separating AI servers from the internet
   3. The line between trusted system instructions and untrusted user data or external content
   4. A visual border displayed in the user interface showing where user input begins

8. The lesson states that AI in 2024 cannot be made fully injection-resistant. What is the primary reason for this?

   1. AI models are not trained with enough safety data
   2. Government regulations prevent AI companies from improving security
   3. AI companies lack the financial incentive to fix the problem
   4. The fundamental architecture of language models makes them follow instructions regardless of source

9. If you build an AI tool that can summarize web pages, what is the most important security consideration from this lesson?

   1. Adding the ability to summarize multiple pages at once
   2. Ensuring the tool can handle websites with JavaScript animations
   3. Making sure the summary appears quickly for the user
   4. Treating the content of any webpage as potentially untrusted and not blindly following instructions found within it

10. What should an organization do before giving an AI agent powerful capabilities like sending emails or deleting files?

    1. Test the AI thoroughly to ensure it never makes mistakes
    2. Nothing special—AI agents can be trusted with these tasks
    3. Implement strict allowlists, require human approval for sensitive actions, and maintain audit trails
    4. Give the AI a more friendly name so users trust it more

11. Why might sanitizing inputs (removing certain words or patterns) fail to prevent prompt injection?

    1. Sanitization makes the AI less accurate at its main task
    2. Attackers can use encoding, synonyms, or creative formatting to bypass filters
    3. Sanitization is computationally expensive and slows down AI responses
    4. AI models automatically restore removed content during processing

12. What is wrong with relying solely on prompt engineering to prevent prompt injection attacks?

    1. Prompt engineering has no effect on AI behavior
    2. Prompt engineering alone cannot solve the problem because models will still follow instructions in user data
    3. Prompt engineering makes AI responses too slow
    4. Prompt engineering is too expensive for most organizations

13. What should developers understand about the AI models they use in their applications?

    1. AI models can be easily updated to block all malicious instructions
    2. AI models will always follow their system prompts perfectly
    3. AI models are immune to manipulation by external content
    4. AI models cannot reliably distinguish trusted instructions from untrusted data in inputs

14. Why is it risky to give an AI agent the ability to execute financial transactions based on instructions it reads?

    1. AI agents are bad at math and will make calculation errors
    2. AI agents are not allowed to handle money due to regulations
    3. Financial transactions are too slow for AI agents to process
    4. The AI might be injected with instructions to transfer money to an attacker, and it cannot distinguish this from legitimate requests

15. What does auditing tool calls against expected behavior involve?

    1. Ensuring the AI uses the correct API endpoints
    2. Verifying that tool executions match what was actually requested and expected, not what an injected instruction demanded
    3. Counting how many times a tool is used
    4. Recording what the AI says to users