LLM-as-Judge With a Deterministic Validation Layer

A lot of serious teams already run an LLM-as-judge. A second model scores the first. The rubric is in a prompt. The score is a float. Sometimes a human still glances at the low scores. That design is not a mistake. It is how you handle tone, edge-case policy, and "does this summary still sound like us?"

The mistake is treating the judge as the last authority before money moves, PHI leaves the clinical system, or a customer is told their MRI is approved.

ValGuard is not an argument against judges. It is a deterministic validation layer you put around the stack you already have, so the probabilistic parts stay probabilistic and the parts that must not drift stay code.

What you already built, honestly

LangGraph, Crew, Semantic Kernel, a homegrown state machine: the shape is the same. Step N produces JSON. Step N+1 consumes it. Somewhere you added if score < 0.7: escalate. That score came from a model. It will move when you change the judge prompt, the judge model, or the date of the snapshot. An auditor will ask what "0.7" meant in March. You will not have a clean answer unless you versioned the rubric and frozen the model, which almost nobody does for every weekly tweak.

None of that means you should delete the judge. It means the judge should output a field, and a deterministic step should decide whether that field is allowed to authorize a tool. How ValGuard complements LangGraph, CrewAI, and AutoGen is the same idea at the framework boundary: keep the orchestrator, add the gate.

Two different questions

Judge question. "Is this apology the right temperature for a customer who already had two outages this quarter?" Language. Context. Taste. A model is the right tool.

Rule question. "Is gross_amount equal to the sum of lines within 0.01?" "Does the patient email contain approval language?" "Do passport date_of_birth and application date_of_birth match after ISO parse?" "Is the refund less than or equal to the captured charge?" Arithmetic, lists, schema. A model is the wrong tool. It will be right often. Production is a volume business. We already made that argument.

ValGuard answers the second class on every request, in microseconds, with a rule ID. You can still send the first class to a judge. You put the judge inside the graph, not after the side effect.

A flow that keeps both

Call center wrap-up is a clean example. The model drafts a summary and a wrap-up code. A judge (or a specialist model) can score whether the summary matches the transcript's sentiment. Then a pack such as call_wrap_up_code_agent (9 rules, 2.6 µs p50) checks that the code is in the enum, that required fields exist, that the payload is JSON rather than a fenced essay. If the code is illegal, you do not write it to the WFM system because a judge said the prose was kind. Call-center wrap-up validation is that graph without the speech.

Prior auth is the same sandwich with higher stakes. Keep your clinical LLM. Keep a judge that flags uncertain medical necessity for a nurse. Then run the patient-facing guard so "Good news, your MRI has been approved" cannot ship. That guard is 19.1 µs. The prior authorization playbook is this pattern without requiring you to invent the graph.

KYC can use a vision model and even a fraud-scoring model. The cross-document playbook still compares names and dates as data. A judge that says "probably the same person" is a useful input to a reviewer queue. It is a bad authorization to open the account if DOB disagrees by a decade.

Invoice three-way match: let a model extract messy PDFs. Let a judge comment on vendor-name near-duplicates if you want. Do not pay the invoice until cross_branch_field_match sees PO total vs invoice total. That check is exact on purpose.

Where the judge sits in ValGuard

You do not rewrite the orchestrator to "become deterministic." You keep step types you already understand:

1. Extract / generate (your model).
2. Optional judge (your second model), writing risk_score, tone_ok, rationale.
3. Validate (ValGuard): schema on the payload, plus thresholds you chose. Example: risk_score must be a number; if risk_score >= 0.8 then route to review_handoff; if forbidden_content hits, block regardless of score.
4. Act only on on_validation_pass.

The judge payload is just another JSON object. Schema-check it the same way you schema-check an extraction:

{
  "wrap_up_code": "billing_followup",
  "summary": "Customer asked about a duplicate charge on March invoice.",
  "risk_score": 0.42,
  "tone_ok": true,
  "rationale": "Summary matches billing intent; no refund language."
}

Rules on that object are boring on purpose. wrap_up_code is an enum. risk_score is a number in [0, 1]. tone_ok is a boolean. If risk_score >= 0.8, the graph goes to review_handoff even when tone_ok is true. If wrap_up_code is missing or invented, the write to WFM does not happen. The judge can be wrong in a soft way. The rule still refuses to call refund().

Shadow mode on the rule tells you how often the judge and the rule disagree, which is the most useful weekly metric in this design. If they always agree, the judge is expensive duplication. If they always disagree, your rubric or your rule is mis-specified. That conversation is impossible when the judge is also the gate.

Latency stays in the band we publish for graphs, plus however long your judge takes. The 3-step linear playbook is 2.14 ms p50 with mocked agents. How we measure ValGuard latency is the stopwatch. Your judge is the line item. We will not pretend otherwise. What we will not add is a third model to grade the judge.

Stability is a property of the gate, not the prose

People hear "deterministic" and think we want to replace agents with regex. That is not the product. Agents remain the right way to read messy PDFs, route a conversation, and draft a reply. Determinism is the property of the check: same JSON in, same pass/block out, tomorrow and next quarter, after you bump the model.

That is also why playbooks are boring on purpose. Ticket routing, FNOL, wire review, HR policy answers: the graph is explicit. Branch conditions are validation outcomes or JSON fields, not "the model decided it was billing." You can still have a classification model. You validate its team enum before the router runs.

For the person who has to defend this in a design review

Expect the objection: "We already have a judge, this is another hop." Answer with the 0.36 ms HTTP path and the µs packs, then with the failure you actually fear. The hop is not the cost. The un-auditable 0.73 score is the cost.

Expect: "Won't rules be brittle?" Yes, if you encode taste as a regex. Do not. Encode totals, enums, required IDs, banned claims, country allowlists. Leave taste to the judge and the human.

Expect: "Can't the judge output JSON with a score and we schema-check that?" Yes. That is the sandwich. Schema on the judge payload is the first ValGuard rule. Business limits on what that score is allowed to authorize is the second. You already almost built this. The missing piece is usually fail-closed enforcement and a rule ID in the log.

The standard orchestration story does not end. You keep LangGraph or the in-house graph. You keep the judge where nuance is real. You add a layer that cannot be sweet-talked, does not burn tokens, and does not change its mind when the prompt team ships on Friday. That is the whole thesis.