AI Tracing Platforms: Langfuse, LangSmith, Helicone, Phoenix

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-AI-tracing-platforms-creators

1. What is a 'span' in the context of LLM tracing?

   1. A database table that stores historical model outputs
   2. A pricing tier offered by tracing platform providers
   3. A specific model architecture used for language generation
   4. A unit of work that can include a prompt, response, and any tool calls made during processing

2. Why might you choose NOT to replace your existing generic APM with an LLM-specific tracing platform for your entire application?

   1. Generic APM tools are more expensive than LLM tracing platforms
   2. Non-LLM components like databases and APIs don't benefit from LLM-specific observability features
   3. All modern applications use only language models
   4. LLM tracing platforms cannot handle web traffic

3. What problem does sampling discipline help address in LLM tracing?

   1. Model accuracy degradation over time
   2. Token limits in prompt engineering
   3. Trace volume cost — full-fidelity traces for every call become expensive quickly
   4. API rate limiting from LLM providers

4. What is the primary purpose of 'prompt replay' functionality in LLM tracing platforms?

   1. To re-execute previous LLM calls with the exact same inputs to observe behavior across different model versions
   2. To record new prompts being created by users
   3. To automatically generate prompts for testing
   4. To delete old prompt data to save storage space

5. What does 'prompt diff' functionality allow developers to compare?

   1. Different input prompts to understand how variations affect LLM outputs
   2. The prices of different LLM providers
   3. Multiple versions of the same application
   4. Different programming languages used to call LLMs

6. What does 'cost attribution' track in LLM applications?

   1. The personality traits of the AI model
   2. The number of developers working on a project
   3. The physical location where model inference occurs
   4. The monetary cost associated with each LLM call based on token usage

7. What is the relationship between trace retention duration and pricing in LLM tracing platforms?

   1. Longer retention periods typically require higher pricing tiers
   2. Retention has no impact on pricing
   3. Shorter retention always means better performance
   4. Pricing is based on the number of developers using the platform

8. What does 'exportability' refer to in the context of LLM tracing platforms?

   1. The capability to export LLM models themselves
   2. The ability to export trace data to external systems for long-term storage or analysis
   3. The process of exporting prompts to different file formats
   4. The ability to share traces publicly on social media

9. Why is eval integration important for LLM tracing platforms?

   1. It enables regression detection by comparing LLM outputs against predefined test cases
   2. It replaces the need for human reviewers
   3. It automatically generates evaluation metrics for all prompts
   4. It ensures all LLM responses are always correct

10. Which of the following is NOT a capability that the lesson attributes to purpose-built LLM tracing tools?

    1. Providing replay and diff across runs
    2. Integrating with eval suites for regression detection
    3. Capturing full prompt, response, tool call, and cost per span
    4. Automatically fixing bugs in production code

11. What does 'span model fit' evaluate when comparing LLM tracing platforms?

    1. The physical size of the platform's servers
    2. How well the platform fits within a specific budget
    3. How well the platform's data model captures LLM, agent, and tool call concepts
    4. Whether the platform runs on popular cloud providers

12. What metadata do purpose-built LLM tracing tools capture that generic logging might miss?

    1. Memory usage percentages and CPU temperatures
    2. Network latency and packet loss rates
    3. Token counts, costs, and the hierarchical relationships between prompts and tool calls
    4. HTTP request headers and database query times

13. Why might a team choose to use multiple scoring criteria when evaluating LLM tracing platforms?

    1. Scoring criteria are required by law in some jurisdictions
    2. Multiple criteria make the platforms run faster
    3. All platforms are equally good at all features
    4. No single platform excels at everything; criteria like span model fit, UI features, and pricing trade off against each other

14. What is the primary limitation of LLM tracing platforms when it comes to non-LLM components?

    1. They require all code to be rewritten in Python
    2. They cannot capture meaningful observability data for non-LLM stack components
    3. They automatically convert all logs to LLM prompts
    4. They are too expensive for traditional applications

15. What does 'prompt logging' primarily help teams accomplish?

    1. Automatically generating documentation for code
    2. Logging user login credentials securely
    3. Creating an audit trail of exactly what inputs were sent to LLM models for debugging and compliance
    4. Writing better marketing copy for products