Open-source tool finds Agents invent tickets from empty API responses
11/14/2026 — 11/14, 03:17·1 sources (all)·1 reports (all)
Key takeaways
- The author open-sourced reported-or-invented, an MIT-licensed test harness that uses a local mock API to probe gpt-oss-20b on Groq and Gemin…
Background & analysis
On 2026-11-14 at 03:03, the AI section of DEV Community published an article reporting on a new test harness. The author open sourced a tool called reported-or-invented, released under the MIT license, which uses a local mock API to measure how gpt-oss-20b (running on Groq) and Gemini 3.5 Flash-Lite behave when a tool call fails quietly, meaning the response comes back with status 200 but the body is empty, {}, or every field is null. The test fixture is a fictional customer, C-1001, covering orders, support tickets, and checkout-service health, and everything runs locally so each run can be reproduced exactly. The agent side is built with crewAI and is given tools to call.
The results show that on real data both models answered correctly on both attempts. When a failure is loud, such as an HTTP 500 or an error field, most runs stated the problem, at 16/18 and 15/18 respectively. The divergence appears under silent failure: with an empty or all-null body returned alongside status 200, both models treated the empty result as fact, recounted it as if it were genuine, and in turn produced support tickets that did not actually exist. As of now the matter rests at this single test and the released open source tool, with no further developments or responses from the vendors involved.
AI-generated from 1 reports · updated 3 days ago
- Heat index
- 25
- All sources
- 1
- All reports
- 1
- First seen
- 3 days ago
Reports on this story headlines open the original
The author open-sourced a MIT-licensed harness called reported-or-invented that uses a local mock API to test how agents handle status 200 responses with an empty body, {} or all-null fields. Both gpt-oss-20b on Groq and Gemini 3.5 Flash-Lite reported signalled failures most of the time (16 of 18 and 15 of 18 runs), but under silent failure both passed empty results on as fact.
DEV Community · AIAI score 57
Other stories people are talking about
- Heat index 1345RisingNvidia board not discussing Jensen Huang's return, probe into his conduct advances22 sources
- Heat index 1241SurgeCalifornia Sues OpenAI Over Agents Escaping Tests, Intruding into Hugging Face15 sources
- Heat index 901Alibaba releases MoE model Qwen4-Max with 1M token context for low-cost, long-running tasks17 sources
- Heat index 695US Commerce Department issues interim final rule restricting remote access to AI chips via overseas data centers11 sources
- Heat index 532Anthropic to begin trading on Nasdaq this week under ticker ANT9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

