Enterprise LLM Gateways: Portkey, LiteLLM, Vercel AI Gateway

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-AI-and-enterprise-llm-gateways-creators

1. What is the primary architectural purpose of deploying an LLM gateway in front of direct provider API calls?

   1. To centralize key management, retries, fallbacks, caching, and audit logging in one layer
   2. To automatically select the cheapest model for every request without configuration
   3. To eliminate the need for monitoring AI application performance
   4. To replace the need for prompt engineering in applications

2. A development team notices their LLM-powered feature becomes completely unavailable whenever their gateway service experiences an outage. What architectural recommendation does this scenario support?

   1. Migrate to a self-hosted gateway solution to eliminate all failure modes
   2. Implement a direct-provider fallback path for critical production features
   3. Disable caching to reduce dependency on gateway infrastructure
   4. Always use the gateway for every single AI feature regardless of criticality

3. Which of these is a capability that LLM gateways can provide using AI-driven logic?

   1. Synthesizing responses from multiple models into one output
   2. Predicting future user queries to pre-warm caches
   3. Automatically routing requests to a healthy provider when the primary fails
   4. Generating high-quality prompts to replace hand-written ones

4. A team evaluates two gateways: Gateway A has 95% cache hit rate but adds 50ms latency, while Gateway B has 20% cache hit rate but adds only 5ms latency. Based on the scoring criteria from the lesson, which gateway likely scores higher on 'latency overhead'?

   1. Gateway A, because higher latency overhead means more robust infrastructure
   2. Gateway A, because caching compensates for the added latency
   3. Gateway B, because it adds less delay to each request
   4. Gateway B, because it has worse cache performance

5. What does the term 'exit path' refer to when evaluating LLM gateway platforms?

   1. The timeout duration before a request fails over to a backup provider
   2. The mechanism for users to log out of the gateway dashboard
   3. The network route taken by requests through the gateway to the LLM provider
   4. The ability to switch providers or migrate away from the gateway without rewriting application code

6. Which scenario best illustrates the caching benefit of an LLM gateway?

   1. The gateway logs every request for compliance auditing
   2. The gateway switches from OpenAI to Anthropic when OpenAI returns a rate limit error
   3. The gateway automatically retries a failed request three times before reporting failure
   4. Two different users send identical prompts and the second request returns in 5ms instead of 800ms

7. Based on the lesson, which capability would be considered a 'policy primitive' that a gateway might enforce?

   1. Generating synthetic training data for fine-tuning
   2. Choosing between GPT-4 and Claude based on prompt content
   3. Encrypting data in transit between gateway and providers
   4. Enforcing spending limits per team or API key

8. A company wants to deploy a gateway but has strict data privacy requirements that prohibit any data leaving their infrastructure. Which deployment model should they prioritize?

   1. Self-hosted gateway
   2. SaaS (Software as a Service) gateway
   3. Hybrid gateway with occasional cloud connectivity
   4. Serverless gateway functions

9. What does 'provider coverage' refer to in gateway evaluation scoring?

   1. The number and variety of LLM providers the gateway can route to
   2. The documentation quality for integrating new models
   3. The geographic regions where the gateway can host its servers
   4. The percentage of successful requests across all providers

10. What does observability refer to when scoring LLM gateways?

    1. The speed at which the gateway can scale to handle more requests
    2. The visual design quality of the gateway's admin dashboard interface
    3. The ability to see and monitor request logs, latency metrics, error rates, and spending across the gateway
    4. The process of making the gateway invisible to end users

11. A team implements a gateway and notices that average response time increased from 600ms to 650ms. What is the most likely explanation?

    1. The gateway is generating additional revenue for the team
    2. The caching feature is malfunctioning and causing delays
    3. The gateway has added indirection overhead that increases latency
    4. The LLM providers are deliberately throttling gateway traffic

12. Which evaluation criterion would help a team determine if a gateway supports different access levels for different teams?

    1. Cache hit percentage
    2. Policy primitives
    3. Provider coverage
    4. Exit path

13. A team evaluates two gateways: one is fully managed SaaS, the other is self-hosted software they run on their own servers. What is a key advantage of the SaaS option?

    1. Unlimited customization of the gateway code
    2. Complete control over all data processing
    3. Lower upfront infrastructure management burden
    4. Guaranteed zero latency overhead

14. What functionality does a gateway provide when it performs 'routing' among LLM providers?

    1. It decides which LLM provider should handle each request based on configured rules
    2. It automatically generates new prompts based on conversation history
    3. It translates requests from one API format to another
    4. It balances the load across multiple internal application servers

15. A team selects a gateway solely based on having the highest provider coverage score. Later, they experience frequent timeouts. What gateway evaluation factor was likely overlooked?

    1. Latency overhead
    2. Observability
    3. Exit path
    4. Cache hit percentage