Evaluating prompt injection scanners for production AI apps

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-AI-prompt-injection-scanner-creators

1. A developer is selecting a prompt injection scanner for their production AI application. What does the lesson describe as the fundamental nature of these scanners?

   1. They are replacements for secure coding practices
   2. They are probabilistic seatbelts that reduce but don't eliminate risk
   3. They are guaranteed to catch every known attack pattern
   4. They are deterministic filters that block all malicious prompts

2. When benchmarking prompt injection scanners, what two specific metrics does the lesson recommend comparing?

   1. Cost per API call and throughput
   2. Speed and latency
   3. Recall on attack corpus and false positive rate on benign traffic
   4. Documentation quality and support response time

3. How many benign prompts does the lesson recommend testing to evaluate a scanner's precision?

   1. 200 benign prompts
   2. 1000 benign prompts
   3. 2000 benign prompts
   4. 500 benign prompts

4. The F1 score is recommended for selecting a scanner. What two metrics does F1 balance?

   1. Cost and coverage
   2. Latency and throughput
   3. Recall and precision
   4. Speed and accuracy

5. Why does the lesson state that prompt injection scanners cannot promise zero injections will get through?

   1. Because attackers have unlimited compute resources
   2. Because scanners are probabilistic and attack patterns continuously evolve
   3. Because they are too expensive to run on every request
   4. Because AI models are perfectly secure without scanners

6. What security practice does the lesson state that prompt injection scanners cannot replace?

   1. Encryption at rest
   2. User authentication
   3. Input validation
   4. Least-privilege tool design

7. How frequently does the lesson recommend re-benchmarking prompt injection scanners?

   1. Once a year
   2. Monthly
   3. Every six months
   4. Quarterly

8. What does the lesson recommend subscribing to in addition to quarterly re-benchmarking?

   1. Marketing newsletters from scanner vendors
   2. Academic journals on AI safety
   3. Vendor attack feeds
   4. Industry conference proceedings

9. A company has very little benign traffic but faces many attack attempts. Which metric should they prioritize when selecting a scanner?

   1. Cost per scan
   2. Recall (true positive rate)
   3. False positive rate
   4. API response time

10. What key term describes the process of filtering or blocking adversarial inputs before they reach an AI model?

    1. Input filtering
    2. Model quantization
    3. Prompt templating
    4. Output sanitization

11. Why might two different companies using the same scanner achieve different F1 scores?

    1. Their traffic mix differs — one may have more attacks, the other more benign prompts
    2. One company has better engineers
    3. The scanners use different AI models
    4. One company is using the free tier

12. A prompt injection scanner catches 180 out of 200 attacks but also flags 100 out of 1000 benign prompts as malicious. What is its approximate recall?

    1. 90%
    2. 18%
    3. 80%
    4. 18%

13. Based on the lesson, what is the primary purpose of testing scanners against a known attack corpus?

    1. To evaluate how well the scanner detects known attack patterns
    2. To compare documentation across vendors
    3. To measure how fast the scanner processes requests
    4. To determine the scanner's cost efficiency

14. The lesson compares prompt injection scanners to a seatbelt. What type of seatbelt specifically?

    1. A seatbelt that works only on highways
    2. A seatbelt that always prevents injury
    3. A seatbelt with a warning light
    4. A probabilistic seatbelt

15. A developer implements a prompt injection scanner and removes all API access controls, trusting the scanner completely. How does this align with the lesson?

    1. This is acceptable if the scanner has 99% recall
    2. This is recommended because the scanner catches all attacks
    3. This is the best practice for production applications
    4. This contradicts the lesson because scanners cannot replace least-privilege design