AI Agent Evaluation Harnesses: Beyond Pass/Fail

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-evaluation-harnesses-final5-creators

1. What aspect of AI agent behavior is captured by 'trajectory evaluation' that goes beyond checking whether the final answer was correct?

   1. The total computational resources consumed by the agent's neural network during inference
   2. The length of the final response text measured in characters
   3. The sequence of tool calls, reasoning steps, and intermediate decisions made during task execution
   4. The number of times the agent requested clarification from the user during the conversation

2. After updating an AI agent's model or prompt, what practice ensures regression detection even when the final output remains correct?

   1. Comparing the new trajectory against a previously frozen reference suite and flagging any path changes
   2. Running the agent on completely random tasks to test versatility
   3. Asking human reviewers to rate the politeness of the agent's responses
   4. Measuring only the token cost difference between old and new versions

3. Why does repeatedly optimizing prompts against a single, unchanging evaluation set risk reducing an agent's general capability?

   1. The agent may memorize specific solution patterns for that set rather than learning general problem-solving strategies
   2. The trace logs will grow too large to analyze effectively
   3. The agent will become too fast and fail timing-based tests
   4. The deterministic seeds will become corrupted after too many runs

4. In the context of AI agent evaluation, what does 'cost regression' refer to?

   1. When the monetary price of running the agent increases due to hardware upgrades
   2. When the evaluation framework itself becomes expensive to maintain
   3. When a model update causes the agent to use significantly more tokens or incur higher latency without proportionate capability gains
   4. When agents begin refusing more tasks due to safety policy changes

5. What is the purpose of including 'safety probes' in an AI agent evaluation suite?

   1. To test whether the agent exhibits harmful behaviors, attempts to bypass constraints, or produces unsafe outputs under specific conditions
   2. To assess whether the agent's training data includes up-to-date information
   3. To verify that the agent correctly uses all available tools in the optimal sequence
   4. To measure how quickly the agent can recover from runtime errors and system failures

6. A developer notices an agent now reaches correct answers but through a different reasoning path than before. Why should this be flagged even though outcomes are identical?

   1. The different path suggests the agent is deliberately misleading the developer
   2. The agent is clearly using a different neural network architecture
   3. The original path was incorrect even though it produced right answers
   4. The new path may be more expensive, less reliable, or indicate a capability regression that will cause failures on harder tasks

7. What does it mean for an evaluation harness to use 'deterministic seeds' when testing AI agents?

   1. The evaluation framework only accepts one correct answer for each test case
   2. The random number generator inside the agent has been removed entirely
   3. The agent is forced to use a predetermined sequence of tools regardless of the task
   4. The test scenarios are configured to produce identical inputs and conditions across every run, enabling reproducible comparisons

8. Why is self-evaluation inherently biased when AI agents assess their own outputs?

   1. Agents lack the computational capacity to evaluate outputs at all
   2. Agents are unable to access their own internal reasoning traces
   3. Agents have a built-in tendency to favor their own generated content and overlook flaws within it
   4. Agents always produce perfectly accurate outputs that require no evaluation

9. What information should a complete trace log from an AI agent evaluation capture?

   1. Every tool call made, the arguments passed, the responses received, and the agent's reasoning at each decision point
   2. The network latency between the agent and external APIs
   3. Only the final text output generated by the agent
   4. The memory usage of the machine running the agent

10. An eval suite reports that an agent succeeded on a task but failed three of its five subtasks. What does this indicate about the evaluation approach?

    1. The evaluation framework is broken and producing contradictory results
    2. The subtask failures are false positives that should be ignored
    3. The agent requires more training data to complete any task
    4. The suite provides structured success/failure indicators at the subtask level, enabling granular capability assessment

11. What is the relationship between 'held-out tasks' and preventing eval set overfitting?

    1. Held-out tasks are never used during prompt optimization and serve as a fresh validation set to test true generalization
    2. Held-out tasks require the most computational resources to run
    3. Held-out tasks are shared publicly so developers can benchmark across different agent systems
    4. Held-out tasks are the easiest problems in the eval suite and ensure the agent can handle basic cases

12. When comparing two agent versions, a developer finds Version B uses 30% more tokens but completes the same tasks. What should they investigate?

    1. Whether the token increase represents a cost regression without corresponding capability improvements
    2. Whether Version B was trained on newer data
    3. Whether the evaluation framework needs to be upgraded to handle larger outputs
    4. Whether Version B has a memory leak causing excessive token usage

13. Why should evaluation harnesses be configured to replicate prior runs when given identical inputs?

    1. This ensures the agent will always produce the same output regardless of context
    2. This practice is required by most AI safety regulations
    3. Replicating runs allows the agent to learn from its previous mistakes during testing
    4. Reproducibility ensures that any measured differences between agent versions are due to actual changes, not random variation

14. What does 'recovery behavior' measure in AI agent evaluation?

    1. How well the agent handles errors, adapts when tools fail, and continues working toward the goal despite setbacks
    2. The rate at which the agent's confidence decreases as it processes more information
    3. The time it takes the agent to start responding after receiving user input
    4. The agent's ability to recover deleted files from a filesystem

15. A safety probe in an agent eval suite is most likely to test which scenario?

    1. How the agent handles processing large amounts of numerical data
    2. How quickly the agent returns results when queried about popular topics
    3. Whether the agent can be tricked into providing harmful instructions or bypassing content filters
    4. Whether the agent correctly prioritizes certain tools over others