Developer Shares an LLM System Acceptance-Testing Playbook

On November 6, 2026, a developer wrote up on DEV Community how a knowledge base question-answering system for corporate HR policy should be accepted, arguing that a supplier's demo, where the vendor asks its own questions, is worth nothing in a procurement context. The author says professional AI acceptance rests on three things: an eval set, ablation experiments, and refusal accuracy.

Acceptance started with the eval set. Before launch, the team prepared a set of questions with reference answers covering both normal questions and questions that should be refused. In this HR case, 64 items were written: 52 QA items plus 8 refusal items, including asking about one's own salary, asking how to obtain fake sick-leave certificates, and asking about another company's policies. Without such a baseline, the author argues, changes are a matter of guesswork, since you cannot tell whether a revision improved or degraded the system.

The author then ran ablation experiments, measuring each component in isolation. The baseline v0, using default framework configuration, scored 0.67 end-to-end accuracy. Adding chapter-level chunking lifted it to 0.80. Adding keyword and vector retrieval fusion, a component that should theoretically help, dropped the score to 0.78, because the framework's Chinese tokenization had quietly failed. Fixing the tokenization brought it back to 0.81. Adding metadata filtering by region and document version reached 0.83, and finally adding refusal handling together with table lookup reached 0.98. On the refusal side, v0 scored 0.00, answering all 8 questions it should have declined, and after the adjustments refusal accuracy rose to 0.88. The account stops here, with no mention of whether the system shipped or what followed.