How to validate streaming responses

1. 1How It Works

   When your application calls the API with stream: true, the ideal case is exactly what you'd expect from calling the upstream LLM directly: tokens flow to your client as they're generated, with minimal added latency from ValGuard sitting in the middle.

   But validators, by nature, need the complete response text to evaluate most rules meaningfully. You can't check required_fields on a JSON object that's only half-written. This creates two fundamentally different modes of operation:

   Pass-through streaming. When an agent has no rules configured with on_fail: block or on_fail: reask (for example, an agent using only warn/log rules, or no output validators at all), there's nothing that could need to intervene after the fact. So the engine forwards the upstream SSE stream directly to your client as it arrives, with validators still running against the accumulated text for logging purposes, but never in a position to change what the client already received.

   Buffered validation. When an agent has at least one rule that could block or reask, the engine fundamentally cannot forward tokens as they arrive and also guarantee it can stop delivery if validation fails, once a token reaches your client, you can't un-send it. In this case, orchestration execution (X-VG-Flow) does not support streaming at all, and a single agent with blocking rules will not stream token-by-token the way a pass-through agent does; the client receives the response once validation completes.

   This is the core tradeoff you're managing: you can have full real-time token streaming, or you can have a hard guarantee that a bad response never reaches the client. But for a single request through a single validated step, you cannot have unconditional streaming and a blocking guarantee simultaneously. Understanding this is more valuable than any specific configuration setting, because it tells you where to make the actual product decision.

2. 2Prerequisites

   • An agent (or a step you're considering) where the trade-off between streaming UX and blocking guarantees actually matters, typically a customer-facing chat surface
   • A client capable of consuming Server-Sent Events (most OpenAI-compatible SDKs handle this natively when you pass stream: true)
   • Clarity on your product requirement: is it more important that users see partial output as it's generated, or that a validation-failing response never reaches them at all? Different agents in the same product can reasonably answer this differently.

3. 3Step-by-Step Setup

   Step 1: Decide, per agent, which guarantee matters more

   For a customer support chat agent where a slightly awkward phrasing is a minor UX issue but a multi-second delay feels broken, streaming UX usually wins. Configure output rules as warn/log rather than block. For an agent generating something like a legally reviewed contract summary, where a single wrong response has real consequences, the blocking guarantee should win, and you accept buffered delivery.

   Step 2: For streaming-priority agents, remove blocking rules from the streaming path

   Review the agent's validators and change any rule currently set to block or reask to warn or log, if you've decided this agent should prioritize streaming.

   

   Be deliberate here. This means a rule violation on this agent will never prevent a bad response from reaching the user in real time. It will still be logged, so you retain visibility, but not enforcement in the moment.

   Step 3: For validation-priority agents, accept buffered delivery and design the client accordingly

   If you decided the blocking guarantee matters more, keep block/reask rules as-is. Design your client's loading state around "the response may take slightly longer to start appearing" rather than expecting immediate token flow.

   

   Step 4: Call the streaming-priority agent with stream: true

   curl -N -si "$VG_PROXY/v1/chat/completions" \
     -H "Authorization: Bearer $VG_API_KEY" \
     -H "X-VG-Agent: support-reply-streaming" \
     -H "Content-Type: application/json" \
     -d '{"model":"openai/gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"What is your refund policy?"}]}'
   

   Step 5: Build a client that consumes the SSE stream correctly

   const response = await fetch(`${process.env.VG_PROXY}/v1/chat/completions`, {
     method: "POST",
     headers: {
       Authorization: `Bearer ${process.env.VG_API_KEY}`,
       "X-VG-Agent": "support-reply-streaming",
       "Content-Type": "application/json",
     },
     body: JSON.stringify({
       model: "openai/gpt-4o-mini",
       stream: true,
       messages: [{ role: "user", content: "What is your refund policy?" }],
     }),
   });
   
   const reader = response.body.getReader();
   const decoder = new TextDecoder();
   let buffer = "";
   
   while (true) {
     const { done, value } = await reader.read();
     if (done) break;
     buffer += decoder.decode(value, { stream: true });
   
     const lines = buffer.split("\n");
     buffer = lines.pop();
     for (const line of lines) {
       if (!line.startsWith("data: ") || line === "data: [DONE]") continue;
       const chunk = JSON.parse(line.slice(6));
       const token = chunk.choices?.[0]?.delta?.content;
       if (token) appendToUI(token);
     }
   }
   

   Step 6: Build a client path for the non-streaming, buffered agent

   const response = await fetch(`${process.env.VG_PROXY}/v1/chat/completions`, {
     method: "POST",
     headers: {
       Authorization: `Bearer ${process.env.VG_API_KEY}`,
       "X-VG-Agent": "contract-summary-validated",
       "Content-Type": "application/json",
     },
     body: JSON.stringify({
       model: "anthropic/claude-3-5-sonnet",
       messages: [{ role: "user", content: "Summarize this contract's liability clauses..." }],
     }),
   });
   
   if (response.status === 403) {
     const error = await response.json();
     showValidationBlockedMessage(error.error?.message ?? "Response blocked by validation");
   } else {
     const completion = await response.json();
     renderFullResponse(completion.choices[0].message.content);
   }
   

   Step 7: Never call X-VG-Flow with stream: true expecting token-by-token delivery

   Orchestration execution does not support streaming. If your client sends stream: true alongside X-VG-Flow, the API returns HTTP 400 with error code orchestration_stream_unsupported. Route multi-step playbook calls through the standard buffered request path in your client.

   Step 8: Test both paths under realistic network conditions

   Streaming UX benefits are most visible (and most worth protecting) on slower connections or longer responses. Test your streaming-priority agent's perceived responsiveness on a throttled connection, not just on a fast local network where the difference between streaming and buffered is barely noticeable.

4. 4Diagram / Flow

5. 5Configuration Examples

   Streaming-priority agent (validators set to warn/log, safe for pass-through streaming):

   {
     "slug": "support-reply-streaming",
     "validators": [
       { "name": "no_refusal_phrases", "enabled": true, "on_fail": "warn" },
       { "name": "toxicity_pattern_filter", "enabled": true, "on_fail": "log" }
     ]
   }
   ```
   
   **Validation-priority agent (blocking rules, streaming not applicable):**
   
   ```json
   {
     "slug": "contract-summary-validated",
     "validators": [
       { "name": "contract_risk_pattern", "enabled": true, "on_fail": "block" },
       { "name": "required_fields", "enabled": true, "on_fail": "block" }
     ]
   }
   ```
   
   **Example SSE chunk from a pass-through streaming agent:**
   
   ```
   data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":"Our"},"index":0}]}
   
   data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":" refund"},"index":0}]}
   
   data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":" policy"},"index":0}]}
   
   data: [DONE]

6. 6Testing and Verification

   1. Confirm streaming is actually streaming. Use curl -N (disable buffering) against your streaming-priority agent and visually confirm output appears incrementally rather than all at once at the end. A common configuration mistake is leaving a block rule enabled without realizing it silently forces buffered delivery.
   2. Confirm the validation-priority agent never leaks partial content. Send an input designed to fail validation and confirm your client never renders any partial text before the block response arrives. If your client is naively rendering as bytes arrive without checking the final status, this is worth testing explicitly.
   3. Measure time-to-first-token on the streaming-priority agent versus time-to-full-response on the validation-priority agent, on the same network conditions, to have real numbers when explaining the tradeoff to product stakeholders.

   

7. 7Troubleshooting

   "I set stream: true but the response still arrived all at once." Check the agent's validators for any rule still set to block or reask. Even one such rule forces buffered delivery for that agent, regardless of the stream parameter in your request.

   "My playbook call with X-VG-Flow and stream: true isn't streaming." This is expected. Orchestration execution doesn't support token-by-token streaming at all currently. Design your playbook-calling client around a buffered response, and reserve streaming for single-agent calls.

   "Users on slow connections say the streaming agent doesn't feel faster." Check your time-to-first-token specifically, not just total response time. If there's a large fixed overhead before the first token arrives (for example, a slow upstream provider connection setup), the perceived benefit of streaming shrinks. This is often an upstream provider or network issue rather than a ValGuard configuration issue.

   "I want streaming AND a hard guarantee that PII never reaches the client." This is exactly the tension this tutorial describes, and there's no configuration that fully satisfies both for a single agent. You have to choose. A workaround some teams use is a lightweight streaming-priority agent for the visible chat and a separate, buffered, block-configured audit call in parallel purely for compliance logging. But the streamed content the user saw cannot be un-sent if the audit call fails, so this only gives you visibility after the fact, not prevention.

   "Reask on a streaming agent behaves unexpectedly." Any rule set to reask implies the engine might need to retry the entire upstream call, which is fundamentally incompatible with having already streamed the first attempt's tokens to the client. Agents with reask rules should be treated the same as block rules for streaming purposes: expect buffered delivery, not token-by-token.

8. 8Best Practices

   • Make the streaming-vs-validation tradeoff an explicit product decision, not a default you fell into. Decide per agent, based on what a bad response actually costs you in that specific context.
   • Never mix block/reask rules onto an agent you're relying on for real-time streaming UX without accepting that it will silently stop streaming. If you need both to some degree, consider whether the task can be split into a fast, low-stakes streaming agent and a separate, buffered, high-stakes validated agent.
   • Design your client to handle both response shapes from the start, even if only one agent currently streams. Product requirements shift, and a client that already knows how to handle both an SSE stream and a buffered JSON response with a possible 403 validation block will adapt far more easily than one hardcoded to a single path.
   • Measure time-to-first-token, not just total latency, when evaluating whether streaming is actually delivering its intended UX benefit.
   • Keep validators on streaming agents in log/warn mode intentionally, and review those logs regularly. You're trading enforcement for speed, and that tradeoff is only safe if you're actually watching what would have been caught.

9. 9Advanced Options

   Client-side soft validation as a stopgap. For streaming agents where you can't use server-side blocking, some teams implement lightweight client-side pattern checks (obvious profanity, format sanity checks) on the accumulating stream purely as a UX safety net. This is not a substitute for ValGuard's deterministic validation and shouldn't be treated as an audit-grade control, but can catch the most egregious cases faster than waiting for a post-hoc log review.

   Splitting a single logical response into a fast draft and a validated final. For some use cases (like a support agent that shows a streaming draft answer immediately, then silently swaps in a validated version if the draft would have failed a check), you can architect two calls: a fast streaming-priority agent for the initial UX, and a parallel buffered validated agent whose result replaces the draft if it becomes available and differs meaningfully. This is a genuinely advanced pattern and adds real complexity, so only reach for it when the UX payoff is clearly worth it.

   Monitoring the streaming/buffered mix across your agent fleet. If you operate many agents, it's worth periodically auditing which ones are actually streaming versus silently buffered due to an added block rule nobody revisited. A rule added for a good reason six months ago can quietly change an agent's UX characteristics without anyone noticing until a user complains about "the chat feels slower than it used to."

10. 10Summary and Next Steps

    Streaming and hard validation guarantees pull in opposite directions by nature. You can't un-send a token, so a rule that might need to block a response fundamentally can't coexist with unconditional token-by-token delivery on that same call. The productive move isn't looking for a configuration that magically resolves this tension; it's making the tradeoff explicit per agent, based on what a bad response actually costs in that specific context, and building your client to handle both shapes correctly.

    From here, revisiting the shadow-to-enforce rollout guide is useful specifically for streaming-priority agents, since warn/log rules on those agents are exactly the kind of "watching without enforcing" pattern that rollout discipline was designed for. You're not shadow-testing toward eventual enforcement in this case, you're deliberately keeping visibility without ever intending to block, and it's worth being explicit with your team about which of those two situations you're actually in.