HaiAI123

Curated Global AI Tools Directory

Industry update
25
heat index
Flat

Developer series covers monitoring and online evaluation for production AI agents

11/18/2026 — 11/18, 16:19·1 sources (all)·5 reports (all)

Key takeaways

  • A set of developer write-ups lays out how to monitor and evaluate AI agents in production: capture user goals, agent answers, tool calls, re…
  • design online evals around failure types and evidence rather than picking a judge model first;
  • instrument each turn with OpenTelemetry GenAI spans;
  • and catch silent wrong answers and unsupported "done" claims with detectors, metrics and alerts.

Background & analysis

On the afternoon of 18 November 2026, the AI section of DEV Community released a tightly sequenced set of tutorials covering how to monitor and run online evaluations for AI agents in production. Each piece links to deeper method articles, and together they form one connected method series rather than standalone tips.

The series moves through the monitoring stack in order. On instrumentation, one article explains instrumenting an AI agent with OpenTelemetry by wrapping each user turn, model call, and tool call in spans, attaching GenAI semantic-convention attributes such as model name, token counts, prompts, and tool inputs and outputs, and pointing the SDK at an OTLP endpoint. Wiring takes about two minutes; the harder part is deciding what to emit so traces are useful later. On online evaluation, another article argues teams should not start by choosing a judge model. They should first define the failure they care about, the evidence that proves it, and who acts when it fires. Because a judge model can only evaluate what it receives, the conversation context must be complete and include the real outcome of the task; otherwise the judge can only tell whether the agent claimed success.

The series then breaks down specific failure types. One article notes that traces show only whether each step finished, not whether the answer was correct, since failures beyond timeouts and exceptions never surface in an OpenTelemetry span's status. It provides ten detectors, metrics and alert rules built on them, and a four-week rollout plan. Another article classifies cases such as an agent stating that a refund has been processed, or a tool returning HTTP 200 while only queueing the request, as unsupported completion claims, and checks them against an authoritative result for the same action, object, and customer. A further piece requires capturing four things when monitoring: the user's goal, the agent's answer, its tool and retrieval evidence, and the real outcome, splitting failures into four categories instead of collapsing them into a single error rate. The series currently stands as published method articles with links, with no implementation outcomes reported.

AI-generated from 5 reports · updated 4 days ago

Latest turnMonitoring an AI agent in production means capturing the user's goal, the agent's answer, its tool and retrieval evidence, and the actual task outcome. The author notes that traces only show what executed, while many costly agent failures return HTTP 200 with no exception and look normal in a trace viewer. The piece recommends separating four kinds of failure instead of collapsing them into a generic error rate.

updream
24-hour heatHeat index 25
Just charted — trend is building up
24 hours agonow

Reports on this story headlines open the original

Nov 18
  1. Monitoring an AI agent in production means capturing the user's goal, the agent's answer, its tool and retrieval evidence, and the actual task outcome. The author notes that traces only show what executed, while many costly agent failures return HTTP 200 with no exception and look normal in a trace viewer. The piece recommends separating four kinds of failure instead of collapsing them into a generic error rate.

    DEV Community · AIAI score 60
  2. Online evals should start from the failure you care about, the evidence that proves it, and who acts when it fires, not from picking a judge model. The guide stresses that a judge can only evaluate what it receives, so each conversation must carry complete context, especially the real task outcome, or the judge can only tell whether the agent claimed success.

    DEV Community · AIAI score 60
  3. The guide shows how to instrument an AI agent with OpenTelemetry by wrapping each user turn, model call and tool call in spans, attaching GenAI semantic-convention attributes such as model name, token counts, prompts and tool inputs and outputs, then pointing the SDK at an OTLP endpoint. Wiring it up takes about two minutes; the harder part is deciding what to emit so traces stay useful later.

    DEV Community · AIAI score 63
  4. A DEV Community guide argues that an agent saying "your refund has been processed," or a tool returning HTTP 200 after merely queuing the request, is not proof the action happened. It frames these as unsupported completion claims and asks what authoritative result would actually establish completion for the same action, object and customer.

    DEV Community · AIAI score 56
  5. A DEV Community guide points out that tracing tells you whether each step finished, not whether the answer was right, since these failures never surface in a span's status. It lays out ten detectors with the metrics and alert rules built on them, plus a four-week rollout plan, each detector linking to a deeper method write-up.

    DEV Community · AIAI score 52

Other stories people are talking about

How is heat calculated?About the method

Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.

This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.

Surge
Discussion rising fast
New
First report within 6 hours
Rising
Still gathering discussion

Back to AI industry updates →