Article argues agent evals should check world state, not transcripts

On November 12, 2026, the AI section of DEV Community published an article titled "Evaluating AI Agents: Look at Changes in the World, Not the Conversation Log," putting forward a new position on how AI agents should be assessed.

The piece illustrates its argument with a refund case. A support agent tells a customer that order ORD-1010 "was delivered on September 10th, and we're well within the 30-day refund window," then follows up with: "Perfect! Your refund of $59.00 has been processed for order ORD-1010." The tool call behind that sentence genuinely ran, and the money genuinely left the account. The date stated was correct, but the arithmetic was wrong: the conversation actually took place on October 26th, 46 days after delivery, so the refund violated the very policy the agent had just quoted.

Read purely as a conversation, the exchange appears to have been handled successfully. The author therefore argues that evaluation should examine what changed in the environment after the tool call ran, rather than judging quality from how smooth or coherent the dialogue looks. The article also ships a reusable evaluation harness that others can adopt for the same kind of checking. At present, only this single report is available, and no further follow-up developments have been published.