AI Observability Stack 2026: Traces, Metrics, and Cost in One Pane

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-AI-observability-stack-2026-creators

1. A dev team wants to build a complete observability system for their LLM-powered application. Which four data types must be captured to achieve full observability?

   1. API requests, response times, user sessions, and database queries
   2. Error rates, CPU usage, memory consumption, and network latency
   3. Logs, traces, metrics, and cost data
   4. Prompt templates, model versions, temperature settings, and max tokens

2. Where should PII scrubbing happen in an AI observability pipeline to minimize legal risk?

   1. After generating alerts, when notifications are sent
   2. At query time, when the data is retrieved
   3. At ingest, before the data is stored
   4. PII scrubbing is optional and only needed for European users

3. What metadata should be attached to every LLM call in a well-instrumented observability system?

   1. Temperature, max tokens, and API endpoint URL
   2. Timestamp, server hostname, and request ID only
   3. User ID, route, prompt version, and model name
   4. Only the model name and API key used

4. A company stores every prompt and response from their LLM application indefinitely to 'never lose data.' Why is this problematic?

   1. Storage costs become excessive at scale, and storing raw prompts creates legal liability if PII is present
   2. LLMs cannot process historical data older than 30 days
   3. Regulations require deleting data after 7 days automatically
   4. Indefinite storage causes model accuracy degradation

5. To correlate cost and latency with user-visible outcomes, an observability dashboard should connect which elements?

   1. API response codes, server uptime, and database connection pool size
   2. Network bandwidth, disk I/O, and container memory limits
   3. GPU temperature, token count, and model loading time
   4. LLM token usage, request latency, and downstream user actions or errors

6. What is the fundamental limitation of AI observability tools when it comes to quality monitoring?

   1. They generate too much data to store effectively
   2. They cannot measure latency accurately
   3. They automatically redact all model outputs
   4. They cannot replace a real evaluation suite for quality assessment

7. What shared identifier enables tracing a single user request across metrics, traces, and cost dashboards?

   1. Session token
   2. Trace ID
   3. Model name string
   4. API key hash

8. Which of the following represents the four pillars of AI observability as described in modern implementations?

   1. Error tracking, latency measurement, token counting, and user feedback collection
   2. Traces, metrics, quality evaluation, and cost tracking
   3. Traces, metrics, logs, and alerts
   4. API monitoring, model versioning, prompt management, and security logging

9. Why is querying for PII at retrieval time instead of scrubbing at ingest considered risky?

   1. Regulations only apply to data at rest, not in transit
   2. Raw PII data exists in the database between when it's written and when it's queried, creating a liability window
   3. Query performance is too slow for real-time lookups
   4. AI models cannot recognize PII accurately at query time

10. What type of alerts should an AI observability platform generate to properly monitor LLM applications?

    1. Alerts when the model generates any text containing the letter 'e'
    2. Alerts on API error rates only
    3. Alerts when GPU temperature exceeds 50 degrees Celsius
    4. Alerts on quality regressions detected through evaluation metrics

11. How can AI observability tools detect novel failure modes that haven't been seen before?

    1. By comparing current outputs against a static list of known errors
    2. Through sample-based human review of logged traces
    3. By automatically fixing the failures without human intervention
    4. By querying the LLM for potential failure patterns

12. What is the relationship between prompt version tagging and observability?

    1. Prompt versions cannot be tracked in observability systems
    2. Prompt version tagging is only needed for billing purposes
    3. Only the final prompt sent to the model should be tracked, not versions
    4. Tagging prompt versions enables correlation of quality changes to specific prompt modifications

13. Why is a trace store containing raw PII considered a liability rather than an asset?

    1. Trace stores with PII are automatically deleted by cloud providers
    2. PII in trace stores causes the LLM to generate incorrect responses
    3. It creates legal and regulatory exposure if a breach occurs or if compliance audits find unprotected personal data
    4. Raw PII makes queries run faster

14. What distinguishes AI observability from traditional application monitoring?

    1. AI observability adds quality monitoring and cost tracking for LLM interactions
    2. Traditional monitoring includes AI-specific metrics like token usage
    3. AI observability focuses only on error tracking
    4. There is no meaningful difference between the two

15. Which technology standard is commonly used for implementing traces in AI observability?

    1. GraphQL
    2. SQL databases
    3. OpenTelemetry
    4. Kubernetes pods