How to prevent hallucinations

1. 1How It Works

   Hallucination prevention is layered, because no single check catches every failure mode. The layers, roughly from cheapest to most specific:

   Structural separation. If your prompt asks for a bare prose answer, there's nothing for a validator to check except the prose itself. String matching against expected content is brittle and easy to game. Requiring a structured response (a JSON object with an answer field and a separate sources array, for example) gives validators something concrete to check independently: does every claim have an associated source reference, is the sources array non-empty when the prompt required it, are the source IDs actually ones that were supplied.

   Context grounding. For question-answering and RAG use cases, the validator's job is to check that the answer is consistent with the context you supplied, not to independently verify factual truth (validators have no access to the outside world, only to what's in the request). rag_source_presence checks that the response actually references at least one supplied source. rag_source_match checks that claimed source citations correspond to sources that were genuinely provided, not invented IDs.

   Citation format validation. citation_format_valid enforces that citations follow a parseable, consistent format ([source: doc_3], footnote-style, or whatever your application expects), since an answer with unparseable or missing citation markers is unusable by downstream systems even if the underlying content happens to be accurate.

   Refusal handling. When context genuinely doesn't support an answer, the correct output is an explicit "I don't know" or "not covered by the provided material," not a plausible-sounding invention. no_refusal_phrases might sound like it's the opposite of what you want here. It's actually about blocking unwanted generic refusals ("I can't help with that") on requests that should be answerable, which is a different problem from making sure a genuinely unanswerable question gets a genuine refusal instead of a hallucination. The distinction matters: you want refusals exactly when they're warranted, and confident answers exactly when they're warranted, and validators check each direction separately.

2. 2Prerequisites

   • An agent used for question-answering, summarization, or any task where invented content is a real risk
   • If your use case involves retrieval (RAG), a way to pass retrieved chunks into the prompt. ValGuard validates the generated answer against supplied context, it does not perform retrieval itself
   • Familiarity with attaching validators from the validate-llm-output tutorial

3. 3Step-by-Step Setup

   Step 1: Require structured output first

   Before adding any grounding-specific validator, make sure the agent's prompt asks for a structured response with separable claim and evidence fields, for example, {"answer": "...", "sources": ["doc_2", "doc_5"]}, and attach required_fields and json_schema_validate to enforce that shape.

   

   Step 2: Pass retrieved context explicitly into the request

   Wherever your retrieval happens (your own vector DB or retrieval pipeline, outside ValGuard), pass the retrieved chunks into the request, either in the system prompt, the user message, or (for playbook-based flows) as rag_sources on the internal test/validation payload.

   {
     "messages": [
       { "role": "system", "content": "Answer only using the provided sources. Cite each source by ID." },
       { "role": "user", "content": "What is our parental leave policy?\n\nSource doc_2: ...\nSource doc_5: ..." }
     ]
   }
   

   Step 3: Add rag_source_presence

   Attach rag_source_presence to require that the answer references at least one of the supplied sources, catching the case where the model answers confidently without grounding in anything you actually gave it.

   

   Step 4: Add rag_source_match

   Attach rag_source_match to check that cited source IDs actually correspond to sources present in the request, catching the more subtle failure where the model invents a plausible-looking source ID that was never supplied.

   

   Step 5: Add citation_format_valid

   Attach citation_format_valid and configure the expected citation pattern for your application (bracketed IDs, footnote markers, etc.) so downstream rendering can reliably parse citations out of the response.

   

   Step 6: Guard against thin or truncated answers

   If your use case penalizes suspiciously short answers when good context was available, apply the RAG Answer Completeness agent template (rag_answer_completeness_agent in Marketplace → Templates) to the generation step. It's not a single validator. It's a preset bundle of min_length, required_sections, no_truncation_markers, and word_count_range, tuned together to catch an answer that under-uses the supplied material without penalizing genuinely short-but-correct responses.

   Step 7: Test with a deliberately ungrounded prompt

   In the Playground, paste a completion that answers without using the supplied context (a plausible-sounding but unsupported claim), and confirm rag_source_presence or rag_source_match catches it.

   

   Step 8: Test the genuinely-unanswerable case

   Paste a completion where the model correctly says "the provided material doesn't cover this" and confirm your rule set doesn't incorrectly block a legitimate refusal. A validator set tuned only around "must always answer" will misfire here.

   Step 9: Roll out through shadow mode

   Grounding validators, like any new rule set, should observe real traffic in shadow mode before enforcing. See the shadow-mode-rollout tutorial. Hallucination patterns are often subtler in real traffic than in the test cases you think to write yourself.

   Step 10: Review shadow logs for genuinely ungrounded answers caught

   The most valuable output of the shadow window isn't just a would-block percentage. It's actually reading a sample of the specific answers that would have been blocked, to confirm the rule set is catching real hallucinations and not just formatting noise.

4. 4Diagram / Flow

5. 5Configuration Examples

   Grounding validator set:

   {
     "validators": [
       { "name": "required_fields", "on_fail": "block", "params": { "fields": ["answer", "sources"] } },
       { "name": "rag_source_presence", "on_fail": "block", "params": {} },
       { "name": "rag_source_match", "on_fail": "block", "params": {} },
       { "name": "citation_format_valid", "on_fail": "warn", "params": { "pattern": "\\[source:\\s*\\w+\\]" } }
     ]
   }
   

   Example completion that should pass:

   {
     "answer": "Parental leave is 12 weeks paid, per the HR policy [source: doc_2].",
     "sources": ["doc_2"]
   }
   

   Example completion that should fail rag_source_match:

   {
     "answer": "Parental leave is 16 weeks paid [source: doc_9].",
     "sources": ["doc_9"]
   }
   

   (doc_9 was never among the sources supplied in the request.)

6. 6Testing and Verification

   1. Grounded-answer pass case. Confirm a well-formed answer citing a genuinely supplied source passes every rule.
   2. Invented-source fail case. Confirm an answer citing a source ID not present in the request fails rag_source_match.
   3. Legitimate-refusal pass case. Confirm a correct "not covered by the provided material" response is not incorrectly blocked by any rule in your set.

   

7. 7Troubleshooting

   "Legitimate answers are getting blocked for not matching a source format." Check your citation_format_valid pattern against real model output. A too-strict regex is a common cause; loosen it to match the actual citation style your prompt elicits, or adjust the prompt to produce a more consistent format.

   "The model's genuine refusals are being blocked." Review whether any rule assumes every response must contain a substantive answer. A refusal response might legitimately have an empty or minimal sources array, which a naive required_fields check could misinterpret as incomplete rather than as a correct refusal. Consider a distinct response shape for the refusal case.

   "rag_source_match passes even on answers that clearly weren't grounded in the sources." This validator checks that cited IDs were supplied, not that the answer's content actually reflects those sources' text. It catches invented citations, not subtly misrepresented ones. For deeper content-level grounding checks, you may need a custom validator comparing answer text against source text more directly.

   "I don't have a retrieval pipeline. Can ValGuard do the retrieval for me?" No. ValGuard validates the generated answer against context you supply; it doesn't replace your retriever. Pair your own vector search or retrieval system with these validators on the generation step.

8. 8Best Practices

   • Require structured output before anything else. Every grounding validator here works better, and often only works at all, when claims and sources are separable fields rather than free text.
   • Test the refusal case as carefully as the answer case. A validator set that only ever tests "does it answer correctly" will silently start penalizing correct refusals.
   • Treat citation format and grounding correctness as separate concerns. A citation that's correctly formatted but points to a fabricated source is a worse failure than one that's real but slightly malformed. Weight on_fail severity accordingly.
   • Roll out in shadow and actually read a sample of caught cases, not just the aggregate rate. This is where you'll discover whether your rule set is catching real hallucinations or just cosmetic issues.
   • Revisit the rule set periodically as your prompt evolves. A prompt change that alters how the model formats citations can silently break a previously well-tuned citation_format_valid pattern.

9. 9Advanced Options

   Combining with the RAG flow validation tutorial. This tutorial focuses on the validators themselves; the RAG flow validation tutorial covers the full three-step playbook pattern (retrieve, generate, validate) these rules typically sit inside.

   Custom content-level grounding checks. For domains where citation-ID matching isn't strict enough (you need to confirm the content of a claim is actually supported by the cited source's text, not just that the ID exists), a custom validator comparing answer substrings against source text is the natural extension; see the custom validator tutorial.

   Combining with reask for self-correction. A response that fails rag_source_match is sometimes fixable by giving the model another attempt with a reminder to only cite supplied sources. Consider on_fail: reask for this specific rule rather than an immediate block, and compare the reask success rate against a hard block over a real traffic sample.

10. 10Summary and Next Steps

    Preventing hallucinations isn't one validator. It's a layered set: structural separation of claims from evidence, grounding checks against supplied context, citation format validation, and deliberate handling of the legitimate-refusal case so it isn't mistaken for a failure. None of these checks verify truth in some absolute sense; they verify that the model's answer is actually consistent with what you gave it, which is the practical, checkable version of "don't make things up" that a deterministic validator can enforce.

    From here, building a RAG flow with validation puts these validators into the full retrieve-generate-validate playbook pattern, and writing a custom validator is worth exploring if your domain needs content-level grounding checks beyond citation-ID matching.