LLM output validation: rules that run on every completion

LLM output validation means checking a completion against an explicit contract before the next agent, tool, or customer sees it. The contract can be schema, enums, arithmetic, policy phrases, PII patterns, or grounding overlap. The check should be deterministic: same input, same pass or block, with a rule ID in the log.

ValGuard is a runtime validation and policy-enforcement layer for LLM calls and agent steps. Deterministic rules run on every completion before the next model, tool, or customer-facing action. It includes a lightweight step graph so rules apply at handoffs. It is not a general workflow engine, connector platform, or durable-execution system.

This guide is the informational pillar for "how do I validate LLM output?" Product surface: Validation. Substitute pages: structured outputs, LLM-as-judge, Guardrails AI.

Validation is not an eval

Offline evals answer how a model behaved on a dataset last week. Runtime validation answers whether this request may proceed right now.

Teams often conflate the three. Keep all three. Do not let a float score authorize a refund. See evals vs validation.

What to validate

Start from the handoff, not from a generic "AI safety" checklist.

A failure at step two should never reach step five. That is the difference between retry-until-lucky and block / re-ask / escalate.

Browse packs in validator docs and marketplace templates.

Mechanism: proxy path

Point an OpenAI-compatible client at ValGuard. Attach rules to an agent. Choose on-fail: block, reask, warn, or log.

export VG_PROXY=https://api.valguard.ai
export VG_API_KEY=vg_live_...
export VG_AGENT=support-triage

curl -s "$VG_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $VG_API_KEY" \
  -H "X-VG-Agent: $VG_AGENT" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {"role": "system", "content": "Return JSON only: {\"team\",\"urgency\",\"confidence\"}"},
      {"role": "user", "content": "Billing dispute, card charged twice"}
    ]
  }'

Python:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.valguard.ai/v1",
    api_key=os.environ["VG_API_KEY"],
    default_headers={"X-VG-Agent": "support-triage"},
)
completion = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Billing dispute, card charged twice"}],
)
print(completion.choices[0].message.content)

Framework wiring: n8n, LangGraph, OpenAI Agents SDK.

Shadow mode before enforce

Shadow mode runs every rule on real traffic. Failures are logged. Nothing is blocked. It is available on every plan, including Free.

Rollout pattern:

1. Attach packs to one agent.
2. Enable shadow.
3. Watch would-block rate by rule ID.
4. Fix prompts or loosen brittle rules.
5. Switch to enforce on that agent.
6. Expand to the next step.

Tutorial: shadow mode rollout. Post: validate before you enforce.

Failure modes and on-fail behavior

Do not blind-retry deterministic failures on the same model and temperature. Pair validation with cost-aware routing when escalation means a different model or a human.

Structured outputs still help

OpenAI Structured Outputs, Pydantic, and Instructor reduce malformed JSON. They do not encode refund caps, cross-document identity, or org policy packs. Parse first when you can. Validate second. See vs structured outputs and when JSON must be exact.

Latency: three numbers

Blocking and re-ask rules buffer the full reply before emit. Warn, log, and shadow can pass tokens through. Always state the mode when you quote latency. Details: methodology, Trust, benchmarks.

What deterministic rules do not catch

• Fluent false claims with no rule signature
• Side effects that never pass through an LLM call
• Taste and tone (use a judge as an input, not as the only gate)
• MCP server runtime mounts you never put on the proxy path (see MCP security)

We do not claim to prevent every hallucination. We catch structural, policy, numeric, and grounding-overlap failures you encode as rules.

Portability and exit

Playbooks export as JSON (valguard-playbook-template/1), optionally with validators. Per-agent validator lists and audit export are available through the API. Export is per playbook and per agent today. An org-wide policy file is on the roadmap. Billing, keys, and dashboard settings are not part of the export.

Deployment

Default is hosted SaaS. There is no local or open-source runtime. Enterprise self-host (VPC / on-prem) is licensed. Production plan includes a 99.5% SLA; Free, Developer, and Growth are best effort. Pricing: Developer $69, Growth $149, Production $399, Enterprise from $1499. See pricing and Trust.

Worked failure

Without validation: the model returns urgency: "high". Downstream expects 1–5. The router mis-fires. A refund tool runs on the wrong branch.

With validation: enum/range fails → one reask → still failing → block → human queue with rule IDs. The tool never sees the bad label.

Related pillars and posts

• Validation capability
• LLM structured output and tool validation
• RAG failure modes
• Pydantic / OpenAI structured output
• Why we built ValGuard

FAQ

Is validation the same as guardrails? In marketing speech, often. In this product, validation means deterministic rules with on-fail actions and audit rows.

Can I start on Free? Yes. Shadow mode is included. Volume limits apply.

What if ValGuard is down? The request fails. We do not skip rules and pass traffic. Build client fallback deliberately.

Next step

Quickstart → validate LLM output → shadow mode.