ValGuard.ai logoValGuard.ai
Validation & safety~8 min read

How to test AI guardrails before production

Changing validator JSON is a deploy decision. The Simulate drawer on an agent’s validator editor lets you compare the saved policy against unsaved rules on fixtures—sample output, pasted JSON, or redacted quarantine snapshots—without calling a model. You get a pass/block diff and counts suitable for audit, then wire the same shapes into CI before merge.

By the end, you'll know:

  • When to use dashboard Simulate vs valguard test in CI
  • How to run a diff, read changed counts, and fix stale-state after edits
  • How quarantine fixtures work when Quarantine payloads is enabled for the org
  • Where to copy a GitHub Actions example for a pre-deploy gate
UI note: In current UI you may see both labels: Playbooks and Orchestration (same section, depending on plan/version). Agent creation is in Dashboard -> Agents -> Create agent. If a guide mentions Validation logs, open Dashboard -> Logs and apply filters.

  1. 1# How to test AI guardrails before production

    Production traffic should not be the first time a tighter validator meets real output shapes. Use the Simulate drawer on an agent’s validator editor to diff the saved policy against unsaved rules on fixtures—without calling a model.

  2. 2When to use Simulate vs CI

    PathBest for
    Simulate (dashboard)Editing rules interactively; quarantine snapshots; paste one-off outputs
    valguard test (OSS)Offline YAML policy + fixture JSON in git; no ValGuard account
    /internal/test-validators (hosted CI)Same rules as production agent; needs internal secret + agent_slug

  3. 3Step 1 — Open the simulator

    1. Go to Agents → pick an agent → Validators.
    2. Click Simulate (next to Save).
    3. Select fixtures:
      • Sample for this agent (template-aware),
      • Quarantine snapshots (org must opt in under Settings; payloads are redacted),
      • or paste an output JSON/text.

  4. 4Step 2 — Read the diff

    Each fixture shows Saved vs Candidate verdicts and failing rule names. The summary counts how many fixtures change outcome.

    If the agent runs in shadow mode, live traffic still logs failures without blocking—the simulator explains what enforce would do.

    Important: If you edit rules after a run, run Simulate again before save. The editor warns when the diff is stale.

  5. 5Step 3 — Roll out safely

    1. Tighten rules in the editor → Simulate until the diff matches intent.
    2. Save the agent.
    3. Optional: keep shadow mode for a week (shadow mode rollout).
    4. Add regression fixtures to CI (test validators in CI/CD).

  6. 6Offline CLI (same rule IDs, no network)

    For local gates, use the open-source valguard CLI with a valguard.yaml policy file:

    valguard check --policy valguard.yaml --file model-output.json
valguard test --dir conformance/fixtures

    See GitHub: valguard-ai/valguard for releases.

  7. 7Related

What's next?