Microsoft and Hugging Face release ThinkingBox agent eval framework

On October 4, 2026, a report described a new agent evaluation framework called ThinkingBox, released by Microsoft and Hugging Face in a joint blog post. The framework rests on a simple idea: instead of grading an AI agent on what it says in its final reply, it grades the agent on what it actually wrote to a database. The report did not say when the blog post was published, nor did it mention a version number or whether the framework is open source.

The example used to illustrate the approach involves a support agent handling a late delivery. The agent makes nine tool calls, reads the refund policy correctly, and closes the ticket as resolved. Under ThinkingBox's criteria, that outcome is wrong: the delivery exception itself was never closed, so the ticket should not have been marked resolved in the first place. The point is that a fluent final message can mask a task that was not actually completed.

As of the report, the story stops at the framework's announcement. The joint Microsoft and Hugging Face blog post is the only source described, and no details were given about benchmark data, how many organizations took part, or any later updates to ThinkingBox.