A team running eight different AI agents across support, sales, and internal tooling had a dashboard that showed uptime and call volume for each one. Every agent showed green. Every agent was, by that measure, working fine. What the dashboard did not show was that one of the eight agents, the one handling internal expense approvals, had a validator rejection rate nearly four times higher than any of the others, and had for three weeks. Nobody noticed, because "working fine" and "healthy" were being treated as the same thing, and the only metric on screen was whether requests completed, not whether they completed correctly.
This post is the product story behind relative ranking. The published 0–100 formula, score-drop alerts, and weekly digest live in the AI agent health score pillar.
Uptime tells you the agent ran. It does not tell you the agent was right
This is the gap that shows up in almost every team running more than one or two agents in production: engineering effort goes into keeping agents up and responding, and comparatively little goes into comparing how well they are actually doing their job relative to each other. An agent can have perfect uptime and terrible output quality, and from a standard infrastructure dashboard, the two look identical. The expense-approval agent in this case was not failing. It was approving things it should not have, at a rate nobody was tracking, because nothing was ranking agents against a quality signal, only an availability one.
Once the team went looking, the cause was mundane: a prompt update three weeks earlier had loosened language around what counted as a "routine" expense, and the model had drifted toward approving borderline requests more often. A validator was in place and catching a meaningful share of these, which is exactly why the rejection rate spiked, but the rejection rate itself was buried in per-agent logs nobody was cross-comparing. The signal existed. It just was not surfaced anywhere that made "this agent is worse than the others" visible at a glance.
What a reliability ranking actually needs to measure
The fix is not a leaderboard for the sake of having one. It is a ranking built on the metrics that actually distinguish a healthy agent from a struggling one: validator fail and warn rates, block rate, upstream errors, orchestration step mix, latency pressure, and optional cost per success when the data exists. Two agents with identical uptime can have wildly different scores here, and that gap is the entire point. It is the thing uptime alone will never show.
Comparing agents against each other, not just against their own historical baseline, matters because it catches problems a single-agent view misses. An agent whose rejection rate creeps up 2% a week for two months looks fine in isolation, gradual enough that no single week triggers an alert. Ranked next to seven other agents holding steady, that same agent stands out immediately, sitting at the bottom of the list for three straight weeks. Relative comparison surfaces slow drift that absolute thresholds are tuned to miss.
Ranking by risk, not just by count
Not every violation deserves equal weight in the ranking. An agent that occasionally produces a slightly verbose response is a different category of problem than one that occasionally approves an expense outside policy or discloses information it should not. A useful leaderboard weights violations by potential impact through capped penalties, not just by raw count, so an agent handling low-stakes formatting does not rank artificially better than one handling financial approvals just because its failures happen to be more frequent but individually less costly. The ranking should answer "which agent needs attention first," not just "which agent has the most flags."
ValGuard encodes that weighting in a single published formula (agent-health/1.1). Fail, warn, block, upstream, orch mix, latency, and cost each have a capped contribution. Self-heal success can add a small credit back. Missing cost data contributes zero, never an invented number. The formula drawer on Agents → Statistics shows the same breakdown the nightly rollup persists, so the league table and the sparkline do not disagree.
Relative ranking catches what absolute thresholds miss
Most teams that do track agent quality still do it with a fixed alert threshold per agent: page someone if this agent's rejection rate crosses 5%. That approach works for a sudden spike and fails for exactly the pattern in this incident, a slow crawl from 2% to 8% that never crosses a single-agent threshold in any given week. A leaderboard sidesteps this by making the comparison relative instead of absolute: if seven agents are holding steady around 2-3% and one is at 8% and climbing, that agent is visibly the outlier the moment the ranking is viewed, regardless of whether 8% happens to sit under whatever fixed threshold someone configured months ago. The ranking does not need to know in advance what a bad number looks like. It only needs to show that one agent looks different from its peers.
Score-drop alerts close the remaining gap. When today's score is at least fifteen points worse than seven days ago, with enough volume, the system emits a distinct score_drop event. That is different from a same-day rate spike. Spike alerts catch storms. Score-drop catches week-over-week decay. Both belong on the same health timeline.
Ranking needs to be a living view, not a monthly report
A leaderboard that gets generated once a month as a static report has most of the same blind spots as the uptime dashboard it is replacing, just on a longer delay. The value is in continuous ranking: every agent's position recalculated on a rolling window, visible at any time, not reconstructed after someone asks for it. That is what actually would have caught the expense-approval agent in week one of its drift instead of week three, the difference between a ranking that updates daily and one that gets pulled together for a quarterly review.
In practice that means three habits:
- Open Agents → Statistics after the daily rollup and sort worst-first when you are hunting risk.
- Download the league CSV when the platform team runs a Monday review, or turn on the optional weekly health digest.
- Follow a drop into the formula drawer, then into validation logs for the validators that dominate the penalty.
The weekly agent league table playbook turns that CSV or digest into a short ops brief. The health drop runbook is the companion when a single agent falls more than fifteen points in a day.
The takeaway
Uptime answers "is the agent running." A reliability ranking answers "is the agent doing its job as well as the others," and those are different questions with different failure modes. The expense-approval agent in this incident never went down. It just quietly got worse than its peers, for weeks, with no view that would have shown that comparison. Ranking agents against each other on the metrics that actually matter turns a silent drift into a visible signal before it becomes a policy incident.
ValGuard ships that ranking as the Agents → Statistics league table: every agent ordered by a transparent health score (0–100), with Δ7d, sparklines, formula breakdown, CSV export, and score-drop alerts when an agent slides versus a week ago. The pillar article publishes the exact formula used in production; this post is the product story of why relative ranking beats uptime alone.
For the deeper mechanics of catching change before it compounds, see deterministic drift detection. A league table and a drift detector answer related but different questions. One shows who is worst right now. The other shows who is getting worse. A mature dashboard needs both.
Book a technical walkthrough with our team → Start free in shadow mode — no credit card →