复盘医疗保险拒赔分析 Agent 因限流导致的静默降级问题

On November 9, 2026 at 23:01, DEV Community's AI section published a retrospective post by a developer who had built a denial analysis agent for a medical billing operation. The agent reads insurance denial letters and works out why each claim was rejected. In its configuration, the primary model was set to Sonnet, and the developer says he was proud of the system, having completed 94 eval runs and been pleased with the results.

The picture changed in the week he was preparing to hand the system over. Reading the logs line by line, he found that all 94 eval runs had actually been served by Haiku, not Sonnet. Sonnet had been returning 429 rate-limit errors the entire time, and because the setup included a silent fallback, the fallback caught every one of those errors without producing any signal. The agent's configuration said Sonnet, yet every run in the eval history came from a smaller model that the developer had never selected.

The post frames this as a hidden failure mode in production systems: a fallback policy that degrades quietly can leave a developer with good-looking benchmark numbers for a model that never actually ran, distorting the evaluation without triggering any visible alarm. The account is a first-person report from a single developer, and no additional sources are available to verify the details of the setup or the stated outcome.