Shadow mode: measure would-blocks before you enforce

Run every rule on live traffic, log would-blocks, and keep responses flowing until false-positive rates look safe.

Shadow mode runs every attached rule on live traffic and logs would-blocks without stopping the response. Use it before enforce so false positives show up as metrics, not outages.

Problem

Flipping new validators straight to block on production traffic is how teams create support spikes. You need a measured would-block rate before customers feel a deny.

Forces

  • Rules that look correct on fixtures fail on messy production phrasing.
  • Compliance wants evidence before a hard gate.
  • Redeploying app code to toggle a flag is slow across services.

Pattern

Attach packs to an agent. Set the agent (or rule) to shadow. Traffic still succeeds. ValGuard evaluates and writes audit rows with rule IDs. When would-block rates look sane, switch to enforce (block / reask).

flowchart LR
  C[Client] --> V[ValGuard]
  V --> M[Model]
  M --> V
  V -->|always deliver| C
  V -->|log would-block| A[Audit]

Shadow mode is available on every plan, including Free. Hosted shadow still sends the request to ValGuard. It is not offline or local evaluation. Trust.

Code

export VG_PROXY=https://api.valguard.ai
export VG_API_KEY=vg_live_...
export VG_AGENT=support-triage-shadow

curl -s "$VG_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $VG_API_KEY" \
  -H "X-VG-Agent: $VG_AGENT" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Classify urgency 1-5 as JSON"}]}'

In the dashboard, keep the agent in shadow until the would-block rate and sample payloads look acceptable. Then flip to enforce.

Trade-offs

BenefitCost
Safe measurement on real trafficBad outputs still reach the next step while shadowing
No app redeploy to change modeHosted path still processes content
Rule IDs in audit before enforceYou must watch dashboards or you never graduate

Anti-patterns

  • Leaving shadow forever on a high-risk payout path.
  • Enforcing without reading would-block samples.
  • Treating hosted shadow as a substitute for Enterprise VPC self-host when residency is the requirement.

Related

Next step

Quickstart. Put one high-risk agent in shadow for a week of traffic before enforce.