Agent Quality Evaluation: Beyond Single-Step Accuracy

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-agent-quality-evaluation-creators

1. A developer reports their agent achieves 99% step accuracy but only completes 40% of assigned tasks. What does this most likely indicate about evaluating the agent?

   1. Step accuracy is an unreliable metric when the agent works in unfamiliar domains
   2. Single-step accuracy fails to capture whether the agent achieves its ultimate goal
   3. The agent requires more training on individual action sequences
   4. Step accuracy measurements should be replaced with outcome-based measurements only

2. Which of the following best describes trajectory quality in agent evaluation?

   1. The percentage of individual actions the agent executes without errors
   2. The sequence and logic of steps an agent takes to complete a task
   3. The final output produced by the agent regardless of how it was generated
   4. The speed at which the agent completes each step in a process

3. A product team implements a system that compares their agent's decisions to expert human decisions on 500 representative tasks. What component of agent evaluation design is this addressing?

   1. Task completion measurement
   2. Trajectory quality assessment
   3. Human-judgment ground truth
   4. Action triggers for quality drops

4. Why is it impossible for AI systems to fully eliminate human judgment from agent evaluation?

   1. Humans are needed to execute the agent's actions
   2. AI cannot measure its own performance without external reference
   3. Human judgment defines what 'quality' means for most real-world tasks
   4. Training data always contains errors that require human correction

5. A company deploys an agent system and monitors its performance metrics monthly. They notice task-completion rates dropped from 87% to 64% over three months. What evaluation component is this demonstrating?

   1. Trajectory quality assessment
   2. Human-judgment ground truth comparison
   3. Quality tracking over time
   4. Action trigger implementation

6. What is the primary limitation of predicting trajectory quality from training data alone?

   1. Training data is always too small to be useful
   2. Trajectory quality depends on how the agent generalizes to novel situations, which training data cannot fully capture
   3. AI models cannot analyze their own training data
   4. Training data only measures step-level accuracy, not trajectory outcomes

7. When designing an agent quality evaluation system, what is the purpose of defining action triggers?

   1. To automatically correct agent errors in real-time
   2. To define what happens when quality metrics fall below acceptable thresholds
   3. To determine which tasks to include in evaluation sampling
   4. To establish baseline accuracy levels for individual steps

8. An agent takes 47 steps to complete a task that a human expert completes in 12 steps, yet both achieve the same correct outcome. Which evaluation metric would highlight this difference?

   1. Task completion rate
   2. Step accuracy
   3. Trajectory quality
   4. Human-judgment ground truth

9. What does sampling methodology address in agent evaluation design?

   1. How to measure individual step performance
   2. Which tasks to evaluate and how to select them for assessment
   3. When to trigger human review of agent outputs
   4. How to compare agent trajectories to human decision paths

10. A developer claims their agent evaluation system requires no human involvement because the system automatically calculates accuracy scores. What fundamental flaw exists in this approach?

    1. Accuracy scores cannot be calculated automatically for agent tasks
    2. The system lacks a definition of what constitutes accurate or high-quality behavior
    3. Automated scoring always produces biased results
    4. Human involvement is only needed for step-level evaluation, not trajectory evaluation

11. Why is task-completion rate insufficient as a standalone metric for agent quality?

    1. Task completion cannot be measured reliably across different agent types
    2. An agent can complete tasks through inefficient or undesirable methods
    3. Task completion is always 100% for well-designed agents
    4. Completion rate ignores whether the agent used appropriate resources

12. What distinguishes trajectory-level evaluation from step-by-step evaluation?

    1. Trajectory evaluation requires human reviewers, while step evaluation is fully automated
    2. Trajectory evaluation assesses the entire task path, while step evaluation focuses on individual actions
    3. Trajectory evaluation is only for simple tasks, while step evaluation handles complexity
    4. Trajectory evaluation measures output quality, while step evaluation measures process quality

13. A team evaluates their agent on 1,000 tasks and finds it completes 92% successfully. However, when humans review the successful trajectories, they flag 34% as having serious flaws in reasoning. What does this reveal about relying only on task-completion rate?

    1. Task completion rate overestimates quality because it ignores trajectory reasoning flaws
    2. Task completion rate is useless for agent evaluation
    3. The agent requires more training on the 8% of failed tasks
    4. The human reviewers are incorrect in their assessments

14. When designing an agent evaluation system, what is the relationship between task completion measurement and trajectory quality assessment?

    1. They are interchangeable; measuring one eliminates the need for the other
    2. Task completion measures outcomes while trajectory quality measures process—both are necessary for comprehensive evaluation
    3. Trajectory quality assessment makes task completion measurement redundant
    4. Task completion should always be measured before trajectory quality assessment

15. A team selects evaluation tasks that are all very similar to examples in the agent's training data. Why might this sampling approach produce misleading quality results?

    1. Similar tasks always produce accurate agent behavior
    2. The agent may perform well on familiar tasks but poorly on novel situations it hasn't encountered
    3. Training data examples are always poor quality benchmarks
    4. Similar tasks cannot be used for evaluation because they introduce bias