Deterministic LLM Validation for Production Agents

HTTP 200 on every step while customers receive wrong answers. Schema that parses but breaks downstream workflows. Citations that look authoritative and cite sources that do not exist.

Production AI needs inspectable policies, not black-box judges. ValGuard applies 160 built-in validators and 116 agent templates on every completion — structure, compliance, grounding, and business rules — in microseconds before the next agent or tool sees the output.

For the engineering deep dive on structured outputs and tool calls, read the structured output validation pillar. For retrieval-specific failure modes, see RAG failure modes and validation gates.

Validation is a contract, not a vibe check

Each handoff between agents is a contract. The validator enforces it.

A failure at step two never reaches step five. That is the difference between "we retry until something passes" and "we block, re-ask once, then escalate."

Browse the full catalog in validator documentation or fork a starting point from validator templates.

Deterministic rules beat another LLM critic

Teams often add a "critic" model to judge outputs. It is expensive, slow, and non-reproducible. Deterministic validators — JSON Schema, regex policy packs, arithmetic checks, grounding overlap — run at 1–50 µs p50 for typical packs on measured hardware (benchmarks).

Examples from published bench runs:

• Hot-path rule check (4 rules): ~1.1 µs p50
• RAG citation validation (4 rules): ~2.3 µs p50
• PII detection (5 compliance rules): ~16 µs p50
• Invoice extraction pack (12 business rules): ~52 µs p50

Compare that to a GPT-4 class completion at ~820 ms median. Validation is not the latency bottleneck — retries from bad outputs are.

What validators cover

ValGuard groups validators into packs teams already reason about:

• Core — JSON Schema, enums, required fields, repair hints
• Compliance — HIPAA, PCI, GDPR, IBAN, PESEL, and regional packs
• Code & SQL — injection patterns, allowlisted statements
• RAG — citation overlap, source matching, chunk sufficiency
• Workflow — escalation triggers, human-handoff gates

Severity and on-fail behavior are configurable per agent: block, reask, warn, or log. Shadow mode lets you measure violation rates before you enforce — see shadow mode rollout.

Structured outputs and tool use

Free-text handoffs between agents lose information. Agent A narrates "customer seems angry about billing"; Agent B interprets differently than Agent A intended.

The pattern:

• Typed handoff objects — JSON Schema or Pydantic between every agent
• Validator on send and receive — catch drift before the next agent runs
• Prose generation only on the final customer-facing step after structured decisions pass

OpenAI structured outputs, LangChain with_structured_output, and ValGuard validators compose cleanly: parse first, validate second, route third. When JSON must be exact, read when JSON must be exact.

Tool calls need the same discipline. The Agentic Tool Call Validator playbook allowlists tools and argument shapes before execution — critical when prompt injection tries to smuggle instructions through retrieved text.

Re-ask vs block vs escalate

When validation fails, the response matters:

• Re-ask — one correction prompt with validator feedback; cheap for transient formatting slips
• Block — stop the pipeline; return a safe refusal to the user
• Escalate — route to human queue with audit context

Never retry deterministic failures blindly. An enum violation on the same model with the same temperature will fail again — and bill again. Pair validators with cost-aware routing so escalation means a different model or a human, not an infinite loop.

Measuring quality in production

Offline evals tell you how the model behaved last Tuesday. Validation tells you what shipped to customers this second.

Instrument:

• Validation pass rate per step and playbook version
• Violation rate by rule ID — not one blob "validation failed"
• Top violated rules week over week — prompt and schema tuning targets

The AI quality glossary defines terms like shadow mode, violation taxonomy, and grounding score. Measuring AI quality in production connects metrics to business outcomes.

Start with one high-risk output

Pick the step where a bad answer is expensive: refund approval, medical script, contract clause, SQL against production data. Attach the tightest validator pack there first.

Then expand upstream — classification and retrieval gates prevent garbage from reaching expensive synthesis steps. The text-to-SQL safety gate and catch PII before it leaves the team articles show focused starting points.

Validation is not a feature you add after launch. It is the layer that makes orchestration, cost control, and observability honest — because every downstream metric keys off the same deterministic verdict.