A compliance lead asked a question that sounded simple: what percentage of production traffic actually ran the full validator chain on the refunds agent last month? The team had a policy PDF, a catalog screenshot, and shadow-mode logs showing would-block counts. What they did not have was a denominator that separated “rules exist on the agent” from “every configured rule executed on this request.” Partial chains — early exit, skipped shadow rules, or routes that bypassed the proxy — looked like healthy traffic in volume charts.
That gap is validation coverage. ValGuard records per-request attribution on persist: how many rules were configured, how many executed, whether shadow rules would have blocked, and which fields were healed. The Validation coverage dashboard turns those columns into an org-level full-chain %, a gap list, and a CSV export for compliance worksheets. This pillar defines the metric, what it is not, and how to improve coverage without blocking production on day one.
For proof that rules ran when they did execute, see how to prove AI guardrails actually executed. For rollout discipline before enforce, see shadow mode rollout and shadow mode is not coverage.
Coverage is not a rule catalog
Most governance programs stop at configuration: validators enabled in the editor, playbooks attached, CI tests green. Configuration proves intent. It does not prove that every production request hit every configured rule in order.
Three buckets show up in real deployments:
| Signal | Meaning |
|---|---|
| Full chain | validation_rules_executed >= validation_rules_configured for requests with configured rules |
| Partial chain | Some rules configured but not all executed (early exit, bypass path, engine skip) |
| No validators | Request reached the org with zero configured rules on that agent |
Org full-chain % = full-chain requests ÷ requests with configured rules. It is intentionally not “validators ÷ all HTTP traffic.” High-traffic agents with no rules yet should not dilute the score; they show up under No validators so you can prioritize onboarding.
What the proxy writes (metadata only)
Migration 000105 adds coverage fields on each request row. The proxy fills them in RequestFinalize.WithCoverage — counts and paths, not prompt storage.
| Field | Role |
|---|---|
validation_rules_configured | Rules attached to the agent snapshot for this request |
validation_rules_executed | Rules that actually ran |
shadow_would_block | Shadow-mode signal: a block disposition would have fired |
heal_field_paths | Deterministic repair touched these JSON paths before validation |
Growth+ teams open Monitor → Validation coverage, pick a UTC window (7–90 days), and review summary cards, per-agent tables, and Coverage gaps (partial chains, no-validator traffic, heavy shadow signal). Export CSV uses the same query as the API (GET /api/validation-coverage?format=csv).
Shadow mode measures precision, not chain completeness
Shadow mode answers: “If this rule enforced, how often would it block?” It does not answer: “Did every configured rule run on this request?” An agent can run shadow validators on every call and still show partial chain if the engine short-circuited or a route skipped validation entirely.
Treat shadow would-block counts as rollout input, not as coverage. Pair them: high shadow signal on an agent that already has full chains is a candidate for simulate-then-enforce; high partial chains with low shadow counts is a wiring or catalog problem first.
Hands-on: close coverage gaps without blocking prod.
Close gaps in order (deterministic before enforce)
- Rank gaps — Sort the dashboard gap list by volume. Fix the top three agents before polishing long-tail routes.
- No validators — Confirm traffic uses
X-VG-Agent(or playbook steps) so requests attach to an agent with a rule pack. New agents: coverage for new agent. - Partial chain — Compare validator priority and early-exit behavior; look for mock mode, emergency bypass, or orchestration steps that call upstream without the proxy hop.
- Shadow-heavy agents — Run Simulate on candidate enforce rules (test policy before production), keep shadow for a window, then promote with shadow-to-enforce rollout.
- Export and review — Monthly CSV + human sign-off; playbook coverage gap remediation structures the 30-day ritual.
ValGuard prefers deterministic repair before re-ask when shape failures are fixable; healed paths appear in coverage export so you can see repair volume beside chain completeness.
Evals and offline benchmarks are not coverage
Monthly eval runs and golden datasets measure model quality on sampled inputs. Coverage measures production path completeness for configured rules. Both belong in a governance program; neither substitutes for the other. See evals vs validation in production.
Compliance language that survives review
Say:
- “For requests with validators configured, X% executed the full chain in window Y.”
- “These agent slugs had partial execution or no validators; remediation owners are listed in the export.”
- “Shadow would-block counts estimate enforce impact before flip.”
Avoid:
- “100% guarded” when the metric only covers requests with configured rules.
- Equating shadow logs with full-chain coverage.
- Claiming CSV export stores customer prompts (it stores agent-level aggregates and counts).
What this is not
- Not payload retention for audit (use evidence envelopes and retention settings separately).
- Not a Free-tier dashboard; coverage follows Growth+ analytics gating.
- Not a substitute for proof-of-execution signatures when an auditor asks for per-request integrity.
Start measuring this week
Pick one production agent with real volume, open Validation coverage for seven days, export CSV, and walk the gap list with engineering and compliance in the same room. If full-chain % is already high, use shadow counts to plan enforce; if partial chains dominate, fix routing before debating rule semantics.
Product detail: observability · What's new · Tutorial: close coverage gaps.
Book a technical walkthrough with our team → Start free in shadow mode — no credit card →