Modern RAG systems can appear confident while remaining surprisingly fragile. Changing the order of the same evidence, rearranging a prompt, or presenting supporting facts in a slightly different sequence can cause the model to produce a different answer. This is not always a retrieval problem; it is often a context presentation problem, and it represents one of the most common and most fixable causes of RAG hallucinations.
This matters beyond correctness: reasoning models do not resolve these structural problems at no cost. When a poorly structured prompt forces a model to reason its way around missing delimiters or inconsistent ordering, that reasoning consumes reasoning tokens, and most providers bill reasoning tokens at the output token rate, approximately five times the cost of input tokens. Correcting a prompt’s structure is a one-time cost. Consuming reasoning tokens to compensate for poor structure recurs on every call, at scale, across every user. Correcting the input eliminates the ongoing cost of compensating for it.
This article examines three practical solutions, each isolating a single variable: whether repeating a prompt reduces hallucinations, how to structure RAG prompts so that identical chunks produce the correct answer, and how to evaluate RAG answers automatically, without requiring manual review of each one. All code referenced below is available in the GitHub repository and can be cloned and executed directly.
Part 1: Does Prompt Repetition Reduce LLM Hallucinations?
GitHub Link: Prompt Repetition
This section begins with a specific research finding: “Prompt Repetition Improves Non-Reasoning LLMs.” The claim is almost deceptively simple: repeating an entire prompt verbatim, back-to-back, can measurably improve accuracy on certain tasks when using a non-reasoning model, without any architectural changes
The test
A classic test case involves identifying the single name that appears between two other names within a long, repetitive list.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="not-needed") prompt = """ Here's a list (potentially with repetitions) of names: Carlos Davis, Dale Sims, Carlos Davis, Dale Sims, Stephen Cruz, Dale Sims, Finnian Ross, Stephen Cruz, Stephen Cruz, Gregory Collins, Dale Sims, Stephen Cruz, Carlos Davis, Stephen Cruz, Dale Sims, Dale Sims, Stephen Cruz, Stephen Cruz, Leonard Kalman, Bruce Phillips, Raymond Roberts, Dale White, Leonard Kalman, Finnian Ross, James Wright, Finnian Ross, Raymond Roberts, Dale Sims, Dale Sims, Leonard Kalman, Dale Sims, Carlos Davis, Leonard Kalman, Bruce Phillips, Dale Sims, Raymond Roberts, Gregory Collins, Gregory Collins, Dale Sims, Finnian Ross What is the single name that appears right between Carlos Davis and Bruce Phillips? """ messages = [{"role": "user", "content": prompt}] response = client.chat.completions.create( model="local-llm", messages=messages, temperature=0.0, ) print(response.choices[0].message.content) |
Doubled prompt (client_double.py): identical content, pasted twice back-to-back into one message, before the question is asked a final time.
The result
When run against Qwen 2.5 7B locally, the results were as follows:
- Single pass: “Dale Sims” (incorrect)
- Doubled prompt: “Leonard Kalman” (correct)
The model, question, and list remained identical in both cases. The only variable that changed was repeating the input once.
Why prompt repetition works
Non-reasoning LLMs are, fundamentally, language models that process text left to right. In a single pass, by the time the model reaches the question at the end of a long block, earlier tokens exert less influence on the final answer than tokens positioned closer to the question. Repeating the entire block gives the model a second exposure to every fact immediately before it must reason the question, without requiring additional reasoning tokens.
This also explains why reasoning-enabled models are considerably less affected by this technique: they already perform a second, third, or even tenth pass internally through chain-of-thought reasoning, albeit at the cost of additional tokens, increased latency, and higher API expenditure.
The implication for RAG systems is significant: if ordering and repetition alone can alter an answer, then permutation-induced hallucinations, producing a different answer solely because retrieved evidence arrived in a different order, constitute a real, structural risk in any RAG pipeline rather than an isolated edge case.
Part 2: How to Structure RAG Prompts to Prevent Hallucinations
GitHub Link: Reconstructing RAG Prompts
If prompt structure alone can alter an answer for a plain list, the same principle should apply to retrieved RAG context. This was tested directly by holding retrieval, the model, and the question constant while varying only how the prompt was constructed.
The dataset
The test used four simulated Apple 10-Q filing chunks, deliberately returned out of quarter order, in the manner a naive chunker might split a financial table row by row instead of by quarter:
|
1 2 3 4 5 6 7 8 |
retrieved_chunks = [ {"origin": "Apple Inc. 10-Q filing - Q3 FY2025 - Revenue", "text": "Total revenue for the quarter was $94.9 billion."}, {"origin": "Apple Inc. 10-Q filing - Q4 FY2025 - Revenue", "text": "Total revenue for the quarter was $102.5 billion."}, {"origin": "Apple Inc. 10-Q filing - Q4 FY2025 - Gross Margin", "text": "Gross margin for the quarter was 47.3 percent."}, {"origin": "Apple Inc. 10-Q filing - Q3 FY2025 - Gross Margin", "text": "Gross margin for the quarter was 46.9 percent."}, ] question = "What was Apple's gross margin for the quarter with $94.9 billion in revenue?" |
Neither sentence identifies its own quarter; that attribution exists only in the origin field, outside the text itself. Whether the model can use this attribution depends entirely on how the prompt is constructed.
Mistake #1: Cross-Chunk Bleeding
GitHub link: bad_structure.py
|
1 2 3 4 5 6 7 8 |
# bad_structure.py undelimited_context = " ".join(chunk["text"] for chunk in retrieved_chunks) messages = [ {"role": "system", "content": "You are a financial question-answering component..."}, {"role": "user", "content": f"""Question: {question}\n\nRetrieved context:\n{undelimited_context}"""}, ] |
Without boundary markers, the model must infer which margin sentence corresponds to which revenue sentence. In practice, it pairs them positionally, matching the first revenue figure with the first margin figure, similar to merging two parallel lists, and confidently returns 47.3 percent, Q4’s margin, in response to a question about Q3.
Mistake #2: Causal Misalignment (“Premature Interrogation”)
The prompt above also poses the question before presenting the context. Decoder-based LLMs rely on causal, left-to-right attention, meaning that whatever appears last in the prompt remains most salient as the model begins generating its response. Placing the question first means the model encounters it before receiving any grounding information; by the time it reaches the supporting facts, the question is no longer fresh in its attention.
The fix
GitHub link: good_structure.py
|
1 2 3 4 5 6 7 8 9 10 11 |
# good_structure.py bounded_context = "\n".join( f'<chunk id="{chunk["id"]}" source="{chunk["origin"]}">\n{chunk["text"]}\n</chunk>' for chunk in retrieved_chunks ) messages = [ {"role": "system", "content": "You are a financial question-answering component..."}, {"role": "user", "content": f"""Retrieved context:\n{bounded_context}\n\nQuestion: {question}"""}, ] |
Two changes were applied to the same data: each chunk was wrapped in an explicit, closed, labeled tag so that attribution could be read directly rather than inferred, and the question was repositioned to the end, following all context.
The result
| Bad structure | Good structure | |
|---|---|---|
| Chunk boundaries | None (raw concatenation) | <chunk id="..." source="..."> tags |
| Question position | First | Last |
| Model’s answer | 47.3% (wrong) | 46.9% (correct) |
Neither retrieval nor the model changed in this comparison. Formatting alone altered the resulting answer.
Why XML beats JSON for RAG prompt formatting
LLMs function as language models rather than parsers. Heavy JSON or structured payloads introduce token bloat that measurably interferes with reasoning, according to “Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models,” which reports failure rates 5 to 20 percent higher with rigid structured output compared to human-readable formats. The solution is not additional structure, but rather structure that remains readable.
Part 3: How to Evaluate RAG Answers Without a Human in the Loop
GitHub Link: Prompt Evaluation
Ensuring correct input addresses only half of the problem. The remaining half involves determining whether the output can be trusted, without requiring manual review of every answer. Two complementary checks were used for this purpose.
Check 1: LLM-as-judge for faithfulness and groundedness
GitHub link: evaluate_answer.py
|
1 2 3 4 5 6 7 8 9 10 11 12 |
def llm_judge_faithfulness(question, context, answer):[KM12] judge_messages = [ {"role": "system", "content": ( "You are a strict evaluation judge for a RAG pipeline. Given " "CONTEXT and ANSWER, decide whether every factual claim in " "ANSWER is directly supported by CONTEXT. Respond in exactly " "this format:\nVERDICT: SUPPORTED or UNSUPPORTED\nREASON: <one short sentence>" )}, {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION:\n{question}\n\nANSWER TO EVALUATE:\n{answer}"}, ] response = client.chat.completions.create(model="local-llm", messages=judge_messages, temperature=0.0) return response.choices[0].message.content |
This method involves a second LLM call that evaluates the first model’s answer against the retrieved context, functioning as a claim-level entailment check.
Check 2: Citation Overlap, a hallucination detection heuristic
GitHub link: evaluate_answer.py
|
1 2 3 4 5 6 7 |
_WORD_RE = re.compile(r"[A-Za-z0-9.%]+")[KM13] def citation_overlap_score(answer, context_chunks): answer_tokens = {t.lower() for t in _WORD_RE.findall(answer) if len(t) > 2} source_text = " ".join(c["text"] for c in context_chunks).lower() matched = [t for t in answer_tokens if t in source_text] return len(matched) / len(answer_tokens) |
This is a deterministic check that requires no model call, measuring what fraction of the answer’s words and numbers can be located in the retrieved text. It does not detect cleverly paraphrased hallucinations, but it reliably identifies the most common failure mode: a number that does not appear anywhere in the context.
Using Real OpenSearch Retrieval
GitHub link: evaluate_answer.py
Here, in part 3 we retrieve context live, through an actual kNN vector search against an OpenSearch index:
|
1 2 3 |
embedder = EmbeddingModel(settings) query_vec = to_list(embedder.encode([question])[0]) hits = knn_search(search_client, settings.opensearch_index, query_vec, k=2) |
Good answer vs. bad answer, Side by Side
Two candidate answers were evaluated side by side against the same Q3/Q4 Apple dataset introduced in Part 2:
|
1 2 3 4 |
CANDIDATE_ANSWERS = { "good": "Apple's gross margin for the $94.9 billion quarter (Q3 FY2025) was 46.9 percent.", "bad": "Apple's gross margin for that quarter was 47.3 percent, reflecting strong iPhone demand in international markets.", } |
| “Good” answer | “Bad” answer | |
|---|---|---|
| Judge verdict | SUPPORTED | UNSUPPORTED |
| Citation overlap | 92% of tokens traceable to source | 47% – barely half |
The incorrect answer’s wrong-quarter figure (47.3 percent) is, in fact, present in the source text; it is simply attributed to the wrong quarter, an error the overlap check alone would not catch. It is the fabricated claim regarding “iPhone demand,” absent from the retrieved data, that lowers the score and triggers the UNSUPPORTED verdict from the judge. This illustrates precisely why both checks should be run together rather than relying on either one in isolation.
For production: RAGAS vs. DeepEval vs. TruLens
This hand-built evaluator serves as an instructional tool rather than a production recommendation. At scale, the following frameworks are recommended:
- RAGAS: provides the same faithfulness and precision checks and is considered an industry standard
- DeepEval: designed for automated testing within CI pipelines
- TruLens: tracks groundedness across an entire application
Conclusion
Across all three parts, a consistent pattern emerges: structure and evaluation matter as much as retrieval quality. Repeating a prompt corrected a reasoning gap without any architectural changes. Labeling chunks and reordering context resolved cross-chunk bleeding and causal misalignment. Two complementary checks, one LLM-based and one purely lexical, identified a hallucinated answer without requiring human review.
None of these solutions are exotic. They represent structural habits available to anyone building RAG systems on OpenSearch today, requiring no change to the underlying retrieval architecture.
For more on the economics behind this (why burning reasoning tokens to compensate for bad prompts quietly drains your API budget), check out this LinkedIn article.
Ready to try this yourself? All the code is open – clone it, point it at your own OpenSearch cluster, and see which of these failure modes are already hiding in your pipeline. Don’t have an OpenSearch cluster yet? Spin one up in minutes on NetApp Instaclustr and skip straight to the fun part.
