Deterministic LLM Validation for Production Agents

HTTP 200 on every step while customers receive wrong answers. Schema that parses but breaks downstream workflows. Citations that look authoritative and cite sources that do not exist.

Production AI needs inspectable policies. ValGuard offers 160 built-in validators and 116 agent templates for structure, compliance, grounding, and business rules. The rules you enable run on the agent's completions. Their verdict covers those checks, not overall answer quality.

For the engineering deep dive on structured outputs and tool calls, read the structured output validation pillar. For retrieval-specific failure modes, see RAG failure modes and validation gates.

Validation is a contract, not a vibe check

Each handoff between agents is a contract. The validator enforces it.

Passing that contract does not prove an answer is true, complete, or useful. A JSON reply can contain the required fields and no PII while stating the wrong refund deadline.

Grounding overlap can check a relationship with supplied text; it does not prove every claim follows from a source. Use domain-specific evals and human review for consequential semantic decisions. Model judges can supplement those checks, but need evaluation themselves.

A configured blocking gate can stop a detected violation before the next step. Warn, log, and shadow mode record violations without blocking them. An error outside the configured rules may still pass.

Browse the full catalog in validator documentation or fork a starting point from validator templates.

Deterministic rules beat another LLM critic

Teams often add a "critic" model to judge outputs. It is expensive, slow, and non-reproducible. Deterministic validators — JSON Schema, regex policy packs, arithmetic checks, grounding overlap — run at 1–50 µs p50 for typical packs on measured hardware (benchmarks).

Examples from published bench runs:

• Hot-path rule check (4 rules): ~1.1 µs p50
• RAG citation validation (4 rules): ~2.3 µs p50
• PII detection (5 compliance rules): ~16 µs p50
• Invoice extraction pack (12 business rules): ~52 µs p50

The playbook benchmarks use an 820 ms reference model call for comparison; that is an assumption, not a measured provider median. These engine measurements exclude model time and buffering. Block/re-ask waits for completion before releasing tokens; another model attempt adds more waiting. See methodology and ValGuard vs LLM-as-judge.

What validators cover

ValGuard groups validators into packs teams already reason about:

• Core — JSON Schema, enums, required fields, repair hints
• Compliance — HIPAA, PCI, GDPR, IBAN, PESEL, and regional packs
• Code & SQL — injection patterns, allowlisted statements
• RAG — citation overlap, source matching, chunk sufficiency
• Workflow — escalation triggers, human-handoff gates

Severity and on-fail behavior are configurable per agent: block, reask, warn, or log. Shadow mode lets you measure violation rates before you enforce — see shadow mode rollout.

Structured outputs and tool use

Free-text handoffs between agents lose information. Agent A narrates "customer seems angry about billing"; Agent B interprets differently than Agent A intended.

The pattern:

• Typed handoff objects — JSON Schema or Pydantic between every agent
• Validator on send and receive — catch drift before the next agent runs
• Prose generation only on the final customer-facing step after structured decisions pass

OpenAI structured outputs, LangChain with_structured_output, and ValGuard validators compose cleanly: parse first, validate second, route third. When JSON must be exact, read when JSON must be exact. The longer comparison is ValGuard vs structured outputs; the pillar guide is LLM output validation.

Tool calls need the same discipline. The Agentic Tool Call Validator playbook allowlists tools and argument shapes before execution — critical when prompt injection tries to smuggle instructions through retrieved text.

Response repair before re-ask

Re-ask is the expensive last resort. For structural failures — bad JSON shape, wrong enum, missing required fields — ValGuard prefers deterministic response repair first: rewrite the payload to satisfy the rule without another model call.

Order of operations on a failed check:

1. Validate — rule id, verdict, reason, metadata (no silent decisions)
2. Repair — deterministic fix when the failure is structural
3. Fallback / retry — cheaper model or alternate route when configured
4. Re-ask — only when repair cannot recover and another attempt is worth the tokens

Every repair and validation rule supports disabled, shadow, and enforce. Shadow measures would-repair / would-block rates before you enforce. Estimate savings with the response repair ROI calculator. Walkthrough: self-healing without re-ask. Product updates: What's new.

Re-ask vs block vs escalate

When validation still fails after repair (or repair is off), the response matters:

• Re-ask — one correction prompt with validator feedback; last option after deterministic repair
• Block — stop the pipeline; return a safe refusal to the user
• Escalate — route to human queue with audit context

Never retry deterministic failures blindly. An enum violation on the same model with the same temperature will fail again — and bill again. Pair validators with cost-aware routing so escalation means a different model or a human, not an infinite loop.

Measuring quality in production

Offline evals measure behavior on a chosen dataset. Runtime validation records the configured rule verdicts on requests. A pass rate is not an answer-accuracy score; compare it with reviewed outcomes before drawing quality conclusions.

Instrument:

• Validation pass rate per step and playbook version
• Violation rate by rule ID — not one blob "validation failed"
• Top violated rules week over week — prompt and schema tuning targets

The AI quality glossary defines terms like shadow mode, violation taxonomy, and grounding score. Measuring AI quality in production connects metrics to business outcomes.

Start with one high-risk output

Pick the step where a bad answer is expensive: refund approval, medical script, contract clause, SQL against production data. Attach the tightest validator pack there first.

Then expand upstream — classification and retrieval gates prevent garbage from reaching expensive synthesis steps. The text-to-SQL safety gate and catch PII before it leaves the team articles show focused starting points.

Validation is not a feature you add after launch. It is the layer that makes orchestration, cost control, and observability honest — because every downstream metric keys off the same deterministic verdict. Compare library and DIY options under library guardrails.