AI Agent Observability: Tracing, Spans, and Replay Debugging

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-observability-tracing-final5-creators

1. What is the primary reason to emit one span per LLM call when instrumenting an AI agent?

   1. To reduce the total cost of running the agent
   2. To automatically fix errors in the agent's logic
   3. To ensure the agent can make decisions without external prompts
   4. To enable granular debugging and isolate performance issues per call

2. Which of the following should be logged at each decision boundary in an AI agent?

   1. Network latency metrics
   2. The agent's internal memory contents
   3. Only the final response from the model
   4. Tool inputs and their corresponding outputs

3. What is the purpose of including correlation IDs across distributed agent calls?

   1. To encrypt all communication between services
   2. To automatically load-balance requests
   3. To uniquely identify and link related spans across different services
   4. To generate random identifiers for security

4. What type of visualization is recommended for spotting loops and wasteful behavior in agent execution?

   1. Line graphs of response times
   2. Flame graphs displaying span hierarchies
   3. Scatter plots showing token distribution
   4. Bar charts of error rates

5. What does 'replay debugging' enable developers to do with AI agent traces?

   1. Automatically fix bugs found in the trace
   2. Convert traces into human-readable summaries
   3. Reproduce exact agent behavior by re-running captured inputs
   4. Delete sensitive data from historical traces

6. Under what condition are agent traces considered 'replayable'?

   1. When the prompts are deterministic and inputs are fully captured
   2. When the prompts contain random variables
   3. When the agent uses multiple tools simultaneously
   4. When the trace includes only successful calls

7. What is a fundamental limitation of AI agents regarding instrumentation?

   1. Agents cannot self-instrument without explicit tracing infrastructure
   2. Agents can identify root causes of behavior changes independently
   3. Agents always produce accurate performance metrics
   4. Agents can automatically discover the best tracing format

8. Why should user secrets be scrubbed from traces at ingest time rather than at query time?

   1. Scrubbing at ingest is computationally cheaper
   2. Secrets should never enter the trace pipeline in the first place
   3. It allows faster queries against the trace data
   4. Scrubbing at query time would delete the original data

9. Which attributes should be included in each span representing an agent decision?

   1. User email address and password
   2. Code repository commit hash
   3. Model name, token count, cost, and outcome
   4. Network bandwidth and CPU usage

10. What concept does OpenTelemetry-style tracing bring to AI agent observability?

    1. Automatic agent self-correction
    2. Real-time model training
    3. Proprietary vendor lock-in
    4. Standardized instrumentation with spans and attributes

11. Why is logging tool inputs and outputs particularly important for debugging AI agents?

    1. It reveals the context that influenced the agent's decision-making
    2. It automatically optimizes tool selection
    3. Tools always return correct results
    4. It reduces the number of spans needed

12. What does it mean to 'instrument' an AI agent?

    1. To deploy the agent to production
    2. To increase the agent's model size
    3. To train the agent on new data
    4. To add code that emits tracing data during execution

13. What specific challenge does multi-turn behavior create for debugging AI agents?

    1. Root causes of behavior changes across turns are difficult to identify
    2. Agents can only process one message at a time
    3. Agents forget previous turns automatically
    4. Multi-turn agents require less tracing

14. For AI agents to emit structured span data, what must be in place?

    1. Nothing - agents emit spans by default
    2. A database to store all outputs
    3. Explicit tracing infrastructure or a tracing tool
    4. A separate monitoring service for each agent

15. How do deterministic prompts benefit debugging of AI agents?

    1. They eliminate the need for spans
    2. They make the agent run faster
    3. They allow predictable replay of agent behavior
    4. They reduce token usage