AI Agent Health Score: Measure, Rank, and Alert

An agent handling internal expense approvals kept a perfect uptime record for three straight weeks while its validator rejection rate quietly climbed to roughly four times the rate of every other agent on the same team's dashboard. Nobody paged on it. Uptime said the agent was fine. The rejection rate said otherwise, but it sat buried in per-agent logs that nobody was cross-comparing, because nothing on the dashboard was built to compare agents against each other in the first place.

That gap between "is it running" and "is it any good" is the gap a health score is built to close. This pillar publishes ValGuard's agent health score (agent-health/1.1): how it is calculated, where it surfaces in the dashboard, how ranking and score-drop alerts relate to each other, and how to run a weekly league review that would have caught the expense-approval drift above well before week three. The formula below matches production code in apps/web/lib/agent-health-score.ts and docs/observability.md.

For the fuller incident and the product case for relative ranking, see agent leaderboard. For the broader instrumentation story, drift, silent failures, audit trails, see LLM agent observability.

Why a health score beats a wall of charts

Teams running more than a couple of agents in production already have request volume, latency, and block-rate charts for each one individually. What they lack is a shared answer to a simple question: which agent needs attention first this week? Most teams approximate that answer with tribal knowledge, whoever happened to glance at a dashboard last, or whoever's agent generated the loudest Slack thread. A more disciplined team sets a fixed alert threshold per agent: page someone if this agent's rejection rate crosses 5 percent. That catches a sudden spike. It does not catch a slow crawl from 2 percent to 8 percent that never crosses the line in any single week, which is exactly the shape the expense-approval drift took. Neither approach scales past a handful of agents, and neither one produces a number a team can simply sort by.

A health score answers three questions with one number:

1. Measure — Is this agent healthy on a 0–100 scale given validation fails, warns, blocks, upstream errors, orchestration mix, latency, optional cost, and self-heal credit?
2. Rank — Where does it sit versus peers on Agents → Statistics (the league table)?
3. Alert — Did a rate spike, or did the score itself drop enough versus seven days ago?

Ranking is not a separate product. It is the sorted view of the same score, which is what keeps the league table and the alerts honest with each other.

Published formula (agent-health/1.1)

Score starts at 100. Each contributing problem subtracts a capped penalty, a self-heal credit adds a small amount back, and the result is clamped to 0–100. The same formula runs in exactly two places: nightly rollups write it to agent_daily_stats.health_score, and the dashboard's formula drawer recomputes it live for the UI. There is one definition of healthy, not a rollup number and a slightly different live number that drift apart from each other over time.

explainAgentHealthScore() returns the score plus this same per-component breakdown, which is what populates the formula drawer in the UI. Caps can sum above 100 on paper; the final score is still clamped to 0–100, so an agent failing on every dimension at once reads 0, not a negative number nobody asked for.

Fail, warn, and block rates use requests_total as the denominator. Upstream error rate and the orchestration step mix use orch_step_runs_total instead, since those two only mean something in the context of a multi-step flow, and the penalty is zero rather than inflated when an agent runs no orchestration steps at all.

Where the score shows up

The score is not a one-off report. It is wired into every view where an agent's standing actually matters:

• Agents list — Health (7d) badge and rate chips.
• Agents → Statistics — League table: rank, score, Δ7d, sparkline, formula drawer; Download CSV via GET /api/dashboard/agent-league?format=csv.
• Agent Statistics tab — Gauge, penalty breakdown, score sparkline from daily scores.
• Overview — At-risk timeline, rate spikes, score-drop events, compact heatmap (Growth+).

Sparklines everywhere use the persisted daily health_score, not a live recalculation with a different formula, so the number on the league table always matches the number in the chart next to it.

Hands-on setup: monitor your flows walks through turning on health alerts, running a league review, and enabling the weekly digest.

Score-drop vs rate-spike alerts

A single rate crossing a threshold and a sustained decline in the overall score are different failure shapes, and catching both needs two different alerts:

The expense-approval agent from the opening example is the second case exactly: a rejection rate climbing two or three points a week never tripped a same-day spike alert, because no single day looked unusual next to the day before it. A score-drop alert comparing today against seven days ago would have fired by the second week instead of surfacing in a quarterly review.

Both respect a minimum volume gate so quiet agents do not page you over three requests. Configure thresholds under Dashboard → Account → Notifications (health_alert_*, health_score_drop_*). Cron runs daily at 08:20 UTC.

Optional weekly league email (opt-in health_weekly_digest_enabled) lands Mondays 08:30 UTC with a ranked snapshot, useful for teams that want the review to happen whether or not anyone remembers to open the dashboard that week.

A weekly league review that actually sticks

The mechanics above are only useful if someone actually looks at them on a cadence. A five-step review turns the league table into a habit instead of a dashboard nobody opens between incidents:

flowchart LR
  A[Monday digest or CSV] --> B[Statistics league]
  B --> C{Bottom quartile?}
  C -->|Investigate| D[Formula drawer + logs]
  C -->|Stable| E[Record baseline]
  D --> F[Score-drop or rate-spike follow-up]
  F --> G[Fix validators / model / traffic]

1. Open Agents → Statistics (or the weekly digest).
2. Sort by score; note Δ7d for movers.
3. Expand the formula on the worst agent — which penalty dominated?
4. Jump to validation or request logs from the timeline when a score-drop or spike fired.
5. Export CSV for the ops channel or ticket if you need an offline ritual.

Playbooks that encode these rituals: weekly league table, health-drop runbook, bottom-quartile sunset, pre-model-upgrade baseline, and on-call health routing (seeded as orchestration templates).

What this is not

Worth being explicit about the boundaries, since a single 0–100 number invites assumptions it was not built to support:

• Not an LLM-as-judge quality score.
• Not a predictive “risk %.”
• Not a cross-org public leaderboard.
• Cost is never invented: missing cost data means zero cost penalty.

Ranking agents the way ops already ranks services

Ops teams have ranked services by error budget for years. Nobody argues a service is fine just because it is up. Agents deserve the same discipline, and a transparent, reproducible formula is what makes that ranking something a team will actually act on, instead of one more chart people learn to stop opening.

Pick an org with a few agents and real traffic, open Agents → Statistics, and confirm the ranks and scores match the formula table above, within rounding. Turn on score-drop alerts, and if your team reviews weekly, the digest. The expense-approval agent that drifted for three weeks under the old uptime-only view would have sat at the bottom of the league table by day four under this one.

Full ops reference: observability docs. Tutorial walkthrough: monitor your flows.

Book a technical walkthrough with our team → Start free in shadow mode — no credit card →