Evaluate AI agents with eight metrics and report pass@k and pass^k

This post argues that AI agent evaluation should inspect task outcomes, tool choice, tool parameters, and consistency from the outside inward, not just read the final reply. It also stresses reporting pass@k and pass^k together: an agent with a 75% single-run success rate may pass three consecutive runs only about 42% of the time. The piece is a general evaluation method rather than coverage of one model release or benchmark event.

Latest turnThe article argues that agent evaluation should grade the end state and tool-call correctness, not just the final message. It proposes an eight-metric ladder from outcomes to tool parameters and consistency, stressing that a 75% success rate gives only about a 42% chance of passing three consecutive runs.