Developer series covers monitoring and online evaluation for production AI agents
11/18/2026 — 11/18, 16:19·1 sources (all)·5 reports (all)
Key takeaways
- A set of developer write-ups lays out how to monitor and evaluate AI agents in production: capture user goals, agent answers, tool calls, re…
- design online evals around failure types and evidence rather than picking a judge model first;
- instrument each turn with OpenTelemetry GenAI spans;
- and catch silent wrong answers and unsupported "done" claims with detectors, metrics and alerts.
Background & analysis
On the afternoon of 18 November 2026, the AI section of DEV Community released a tightly sequenced set of tutorials covering how to monitor and run online evaluations for AI agents in production. Each piece links to deeper method articles, and together they form one connected method series rather than standalone tips.
The series moves through the monitoring stack in order. On instrumentation, one article explains instrumenting an AI agent with OpenTelemetry by wrapping each user turn, model call, and tool call in spans, attaching GenAI semantic-convention attributes such as model name, token counts, prompts, and tool inputs and outputs, and pointing the SDK at an OTLP endpoint. Wiring takes about two minutes; the harder part is deciding what to emit so traces are useful later. On online evaluation, another article argues teams should not start by choosing a judge model. They should first define the failure they care about, the evidence that proves it, and who acts when it fires. Because a judge model can only evaluate what it receives, the conversation context must be complete and include the real outcome of the task; otherwise the judge can only tell whether the agent claimed success.
The series then breaks down specific failure types. One article notes that traces show only whether each step finished, not whether the answer was correct, since failures beyond timeouts and exceptions never surface in an OpenTelemetry span's status. It provides ten detectors, metrics and alert rules built on them, and a four-week rollout plan. Another article classifies cases such as an agent stating that a refund has been processed, or a tool returning HTTP 200 while only queueing the request, as unsupported completion claims, and checks them against an authoritative result for the same action, object, and customer. A further piece requires capturing four things when monitoring: the user's goal, the agent's answer, its tool and retrieval evidence, and the real outcome, splitting failures into four categories instead of collapsing them into a single error rate. The series currently stands as published method articles with links, with no implementation outcomes reported.
AI-generated from 5 reports · updated 4 days ago
Latest turnMonitoring an AI agent in production means capturing the user's goal, the agent's answer, its tool and retrieval evidence, and the actual task outcome. The author notes that traces only show what executed, while many costly agent failures return HTTP 200 with no exception and look normal in a trace viewer. The piece recommends separating four kinds of failure instead of collapsing them into a generic error rate.
- Heat index
- 25
- All sources
- 1
- All reports
- 5
- First seen
- 4 days ago
Reports on this story headlines open the original
Monitoring an AI agent in production means capturing the user's goal, the agent's answer, its tool and retrieval evidence, and the actual task outcome. The author notes that traces only show what executed, while many costly agent failures return HTTP 200 with no exception and look normal in a trace viewer. The piece recommends separating four kinds of failure instead of collapsing them into a generic error rate.
DEV Community · AIAI score 60Online evals should start from the failure you care about, the evidence that proves it, and who acts when it fires, not from picking a judge model. The guide stresses that a judge can only evaluate what it receives, so each conversation must carry complete context, especially the real task outcome, or the judge can only tell whether the agent claimed success.
DEV Community · AIAI score 60The guide shows how to instrument an AI agent with OpenTelemetry by wrapping each user turn, model call and tool call in spans, attaching GenAI semantic-convention attributes such as model name, token counts, prompts and tool inputs and outputs, then pointing the SDK at an OTLP endpoint. Wiring it up takes about two minutes; the harder part is deciding what to emit so traces stay useful later.
DEV Community · AIAI score 63A DEV Community guide argues that an agent saying "your refund has been processed," or a tool returning HTTP 200 after merely queuing the request, is not proof the action happened. It frames these as unsupported completion claims and asks what authoritative result would actually establish completion for the same action, object and customer.
DEV Community · AIAI score 56A DEV Community guide points out that tracing tells you whether each step finished, not whether the answer was right, since these failures never surface in a span's status. It lays out ten detectors with the metrics and alert rules built on them, plus a four-week rollout plan, each detector linking to a deeper method write-up.
DEV Community · AIAI score 52
Other stories people are talking about
- Heat index 64Developer shares a 200-line Python three-tier eval for LLM agents1 sources
- Heat index 47Guide: spotting when AI coding agents weaken or delete your tests1 sources
- Heat index 1143RisingAnthropic Reaches $1.6 Trillion Market Value in First Trading Week After IPO21 sources
- Heat index 1011Anthropic Mathematician Says Claude Formalized the Generalized Riemann Hypothesis21 sources
- Heat index 622Anthropic IPO total raise reaches $93.5 billion after greenshoe exercise20 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

