MCP Tool On-Demand Retrieval Benchmark Cuts Input Tokens by 92.7%

On November 1, 2026, a post on DEV Community described Experiment E14 from the AI Labo Benchmark Series. The series tests AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite, and this particular experiment ranked 5 out of 41 patterns tested so far.

The experiment examined how an agent connects to Model Context Protocol (MCP) servers. The setup is a cloud incident-triage and SLA-repair pipeline made of three modules that together expose 32 MCP tools. Under the baseline approach, every tool schema is injected into the prompt up front, which quickly becomes costly as the tool count grows. The alternative tested in E14 leaves tool selection to the agent by enabling on-demand retrieval with ENABLE_TOOL_SEARCH=auto, so only the descriptions of the tools the agent actually needs are fetched.

Measured on the same three-module pipeline, the change cut first-turn input tokens from 25646 to 2544, and total cumulative input tokens fell by 92.7%. The experiment metadata lists it as E14 (backlog candidate C010), with run ID e14-20261028-vm218-01, run in a Proxmox isolated VM, and the series notes that results are gathered under an anti-cheating sandbox built on Proxmox and systemd.

As of now this is the only published account of the experiment: no baseline-pattern breakdown, broader validation, or follow-up results have been reported, so the 92.7% figure rests on this single run within the series.