# Will Your Cassandra Database Project Succeed?: The New Stack
[Blog](/blog/)>[Technology](/blog/category/technical/)>Fix RAG retrieval in OpenSearch: Hybrid search and filters
Fix RAG retrieval in OpenSearch: Hybrid search and filters
==========================================================
October 28, 2026 | By [ Ramya Ravi](https://www.instaclustr.com/blog/author/ramya-ravi/)
[ ](https://x.com/intent/tweet?text=Fix%20RAG%20retrieval%20in%20OpenSearch:%20Hybrid%20search%20and%20filters&url=https://www.instaclustr.com/blog/fix-rag-retrieval-in-opensearch-hybrid-search-and-filters/) [ ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.instaclustr.com/blog/fix-rag-retrieval-in-opensearch-hybrid-search-and-filters/&title=&summary=Fix%20RAG%20retrieval%20in%20OpenSearch:%20Hybrid%20search%20and%20filters&source=)
***Get the code:*** [*OpenSearch/fix-your-retrieval*](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster)
**Why RAG answers go wrong: retrieval, not the model**
------------------------------------------------------
When a retrieval-augmented generation (RAG) system gives a wrong answer, the model usually gets the blame. Often the real problem happens earlier: the retrieval step never found the right document, so the model had nothing correct to work with. If the right document never reaches the prompt, no model can give the right answer.
This article follows one question through four versions of a RAG recipe assistant built on OpenSearch. Each version fixes the failure of the one before it and exposes a new one, moving from vector search to BM25 keyword search, then OpenSearch hybrid search, and finally hard filters with k-NN filtering. These failures surface most clearly at full scale, where a realistic corpus has enough near-misses to expose each one. We finish with how GPU-accelerated vector index builds on [NetApp Instaclustr](https://www.instaclustr.com/) keep these fixes fast at production scale.
**Retrieval basics: lexical, semantic, and hybrid search**
----------------------------------------------------------
A RAG system has two parts: a retriever that searches a document store, and a generator (the LLM) that writes an answer from what the retriever returns. The generator never sees a document the retriever did not surface, so retrieval quality is a hard ceiling on answer quality.
Three retrieval strategies account for most of the failures and fixes in this article:
- **Lexical search (BM25)** ranks documents by term overlap. It is exact and auditable: a term either matches, or it does not, but it has no sense of meaning beyond the literal words.
- **Semantic search** (vector embeddings with nearest-neighbor search) ranks documents by meaning. A model converts each document into a numeric vector, and a question is ranked against those vectors by distance. This tolerates paraphrasing, but it has little sense of exact identifiers, since embeddings capture meaning, not literal text.
- **Hybrid search** combines both signals in one query. It fixes the main weakness of each approach on its own, but introduces a new problem: how to weigh and merge two differently scaled ranking systems.
**The question that broke the RAG assistant**
---------------------------------------------
Ask a recipe assistant a simple question: *“Does the thegratefulgirlcooks.com Fish and Chips recipe contain sulfites?”*
The correct answer is no, because that recipe lists no allergen cautions. But the corpus has another recipe with the same name, from womensweeklyfood.com.au, and it does contain sulfites from the beer in its batter. For someone with a sulfite sensitivity, mixing them up is a safety problem, not a trivial mistake.
The assistant searches 39,445 real recipes from the [datahiveai/recipes-with-nutrition](https://huggingface.co/datasets/datahiveai/recipes-with-nutrition) dataset. It uses OpenSearch for retrieval and Claude Haiku 4.5 on Amazon Bedrock to generate answers.
The following sections show what happens to that question as the retrieval approach changes.
**Stage 1: Why vector search returns the wrong document**
---------------------------------------------------------
View the demo: [demos/01\_baseline\_vector\_recipes](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/demos/01_baseline_vector_recipes)
The baseline embeds every recipe with all-MiniLM-L6-v2 and retrieves the top 5 using k-NN (k-nearest neighbor) vector search. The model produces 384-dimension vectors, which are L2-normalized, so L2 distance ranks results the same way cosine similarity would. The index uses a faiss HNSW (Hierarchical Navigable Small World) graph.
For the thegratefulgirlcooks.com Fish and Chips question, the right recipe is not in the top 5 at all.
![Fix RAG retrieval in OpenSearch: Hybrid search and filters screenshot 1]()
**Figure 1.** Stage 1 vector search for the thegratefulgirlcooks.com Fish and Chips question. None of the top five hits is the recipe the question names, so the assistant answers “I don’t know.”
The reason is that the question embeds closer to the other Fish and Chips recipe than to the one it names, and other fish and chips content outranks both. To an embedding model, a domain name like thegratefulgirlcooks.com carries almost no meaning, so the shared dish name dominates. This is not specific to recipe names. Any identifier with low semantic content, such as a product SKU, a case number, or a username, is diluted the same way inside an embedding space, because the surrounding words carry more of the semantic weight the model was trained to capture.
The system prompt tells the model to say “I don’t know” when the context does not support an answer, so this run declines instead of guessing. This is the best-case outcome. With a looser prompt, the same retrieval gap produces a confident wrong answer about an allergen.
**Lesson:** Similarity is not identity. When a question names a specific thing, you need exact matching.
**Stage 2: BM25 keyword search with named entity recognition**
--------------------------------------------------------------
View the demo: [demos/02\_baseline\_bm25\_ner\_recipes](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/demos/02_baseline_bm25_ner_recipes)
A straightforward fix is to add keyword search. BM25, the standard keyword-ranking algorithm in OpenSearch, scores documents by term overlap with no vectors involved, so the literal term thegratefulgirlcooks.com either matches or it does not.
This stage adds a spaCy named entity recognition (NER) service. It tags entities in each recipe at ingest time and in each question at query time, and matching entities get a boost in the BM25 query. One test result shaped the design: spaCy’s small model finds no useful entity in thegratefulgirlcooks.com, and it tags the word “chips” as an organization. As a result, the recipe’s source and `recipe_name` are copied directly into the entities field. That metadata is already known, so there is no reason to rely on a model to find it.
**Result:** the assistant now answers “no sulfites” for the thegratefulgirlcooks.com Fish and Chips recipe and “yes, sulfites” for the womensweeklyfood.com.au recipe.
![Fix RAG retrieval in OpenSearch: Hybrid search and filters screenshot 2]()
**Figure 2.** Stage 2 BM25 and NER retrieval for the same question. The assistant now finds the named recipe and correctly answers that it lists no sulfites.
The cost is a separate microservice that has to be running for both ingest and every query, just to tell two Fish and Chips recipes apart. Keyword search also gives up the meaning-based recall that made vector search useful in the first place. This pattern, pairing keyword search with entity extraction, applies anywhere a system must match a named thing exactly, such as a product code, a person’s name, or a legal citation. It trades meaning-based recall for identifier precision, which is the right trade whenever exactness matters more than tolerance for paraphrasing.
**Lesson:** Exact matching fixes identity, but bolting on a second system to get it adds operational weight.
**Stage 3: OpenSearch hybrid search and fusion weights**
--------------------------------------------------------
View the demo: [demos/03\_hybrid\_native\_opensearch](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/demos/03_hybrid_native_opensearch)
Stage 3 keeps exact matching but drops the NER service. Hybrid search combines keyword and vector search in one query. OpenSearch runs BM25 and k-NN together in a single hybrid query, with a search pipeline that merges the results on the server:
1. `normalization-processor` rescales each sub-query’s scores to a range of 0 to 1 using `min_max`, because raw BM25 scores and vector scores are on completely different scales.
2. `arithmetic_mean` combines the normalized scores with weights.
PUT /\_search/pipeline/hybrid-search-pipeline { "phase\_results\_processors": \[{ "normalization-processor": { "normalization": { "technique": "min\_max" }, "combination": { "technique": "arithmetic\_mean", "parameters": { "weights": \[0.6, 0.4\] } } } }\] }
1
2
3
4
5
6
7
8
9
10
11
12
PUT /\_search/pipeline/hybrid-search-pipeline
{
"phase\_results\_processors": \[{
"normalization-processor": {
"normalization": { "technique": "min\_max" },
"combination": {
"technique": "arithmetic\_mean",
"parameters": { "weights": \[0.6, 0.4\] }
}
}
}\]
}
View this exact setup: [common/opensearch\_client.py, ensure\_hybrid\_pipeline()](https://github.com/instaclustr/code-samples/blob/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/common/opensearch_client.py#L195-L242)
### Why fusion weights act as a ceiling
Weights follow the order of the sub-queries: 0.6 for BM25 and 0.4 for vector. The recipe’s source domain is part of the indexed text, so BM25 can match it without an NER service.
The key finding is that fusion weights are ceilings, not preferences. A document found only by BM25 can never score higher than the BM25 weight. On the full corpus, the thegratefulgirlcooks.com Fish and Chips recipe is not in the vector top 50, so BM25 is the only retriever carrying it. With BM25 weighted at 0.3, recipes with similar wording outranked it on similarity alone, and it came last in the top 5. At 0.6, it moves up to 2nd. On the 260-recipe sample, 0.3 worked fine, so tune weights against the corpus that will actually be served. This is a general property of min-max normalization combined with weighted fusion, not an OpenSearch-specific quirk: pinning each sub-query’s best hit to 1.0 is what turns a weight into a ceiling.
### Why the candidate window matters
The candidate window matters too, and it applies to both sub-queries. Each one, BM25 and vector, is searched deeper than the final result size (`pagination_depth` of 50 against a size of 5). If the window were only 5, a recipe that either retriever ranks 6th would never reach the combiner.
View this exact query: [common/opensearch\_client.py, hybrid\_search()](https://github.com/instaclustr/code-samples/blob/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/common/opensearch_client.py#L245-L280)
![Fix RAG retrieval in OpenSearch: Hybrid search and filters screenshot 3]()
**Figure 3.** Stage 3 hybrid search on the full corpus. The thegratefulgirlcooks.com Fish and Chips recipe is back at rank 2, carried entirely by the BM25 side of the query.
**Lesson:** Hybrid search is not “vector search, slightly improved.” It is two independent retrievers, and either one can carry a document the other misses entirely, as long as the weights and window give it room.
**Stage 4: Hard filters with k-NN filtering**
---------------------------------------------
View the demo: [demos/04\_hybrid\_ground\_refine](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/demos/04_hybrid_ground_refine)
Now the same dish, but asked by a user who must avoid sulfites: *“Suggest a British fish and chips recipe.”*
81% of the corpus (31,949 recipes) carries a sulfite caution. A ranking can push those recipes down, but it cannot guarantee their removal. “No sulfites” and “contains sulfites” can embed close together, and BM25 has no concept of negation.
The fix is to stop treating safety as a ranking problem. Stage 4 uses ground then refine:
1. Ground: a BM25 plus entity query against the keyword index, with exclusions applied as boolean `must_not` clauses. When an exclusion is set, the 25-candidate set is guaranteed not to contain excluded recipes, not just ranked lower.
2. Refine: a k-NN search against the vector index, limited to the grounding set’s IDs, with the same exclusions applied again as a second layer of protection. This reranks the safe candidates by meaning.
View the full retrieval logic: [demos/04\_hybrid\_ground\_refine/query.py](https://github.com/instaclustr/code-samples/blob/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/demos/04_hybrid_ground_refine/query.py)
{ "query": { "knn": { "embedding": { "vector": \[ ... \], "k": 25, "filter": { "bool": { "filter": \[ { "terms": { "recipe\_id": \["<grounded ids>"\] } } \], "must\_not": \[ { "terms": { "cautions": \["Sulfites"\] } } \] } } } } } }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"query": {
"knn": {
"embedding": {
"vector": \[ ... \],
"k": 25,
"filter": {
"bool": {
"filter": \[ { "terms": { "recipe\_id": \["<grounded ids>"\] } } \],
"must\_not": \[ { "terms": { "cautions": \["Sulfites"\] } } \]
}
}
}
}
}
}
**What changes with the exclusion filter**
Without a filter, the results lean toward recipes with a sulfite caution, which is expected when 81% of the corpus carries one. With `--exclude allergen=Sulfites`, every sulfite recipe is gone, and the assistant recommends the thegratefulgirlcooks.com recipe.
![Fix RAG retrieval in OpenSearch: Hybrid search and filters screenshot 4]()
**Figure 4.** Stage 4 without a filter. The grounding set includes both Fish and Chips recipes, but the refined results and the answer are all recipes with a sulfite caution.
![Fix RAG retrieval in OpenSearch: Hybrid search and filters screenshot 5]()
**Figure 5.** Stage 4 with `--exclude allergen=Sulfites`. Every recipe in the grounding and refined sets has no cautions, and the assistant recommends the thegratefulgirlcooks.com Fish and Chips recipe.
### Why filter placement matters
Where the filter goes matters. Wrapping the knn clause in a surrounding bool query causes the filter to run after the search: OpenSearch returns the nearest neighbors across all 39,445 recipes, then intersects them with the roughly 25 grounded IDs, and the intersection is usually empty. With the filter inside the knn clause, OpenSearch filters during graph traversal, and falls back to exact search when the filter is selective enough to make that cheaper. On a small sample, both approaches look the same, because the global neighbors and the filtered set overlap heavily.
View this exact logic: [common/opensearch\_client.py, build\_recipe\_vector\_query()](https://github.com/instaclustr/code-samples/blob/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster/common/opensearch_client.py#L503-L554)
This ground-then-refine pattern generalizes beyond allergen safety. Any requirement that must hold with certainty, such as a data residency rule, an age restriction, or a jurisdiction limit, needs the same treatment: enforce it as a boolean filter inside the retrieval query, rather than relying on a ranking model to respect it.
**Lesson:** Relevance is a score, and safety is a rule. A score should not be allowed to enforce a rule.
With retrieval now correct at every stage, the remaining question is whether the index can be built fast enough to keep up as the corpus grows. If your corpus is large, that is where GPU-accelerated index builds come in.
**GPU-accelerated vector index builds in OpenSearch**
-----------------------------------------------------
The four fixes above matter most at full scale, where a realistic corpus has enough near-misses to expose the failure each one addresses, and that same scale is what makes the index slow to build. Approximate nearest-neighbor graphs, including the one OpenSearch builds for vector search, get more expensive to construct as the number of documents and their dimensionality grow.
OpenSearch’s remote vector index build on NetApp Instaclustr moves graph construction to a GPU in response to that cost. OpenSearch sends the build request over HTTP, and the vectors and the finished faiss HNSW graph move through S3.
A segment is only sent to the GPU once its vector data passes `index.knn.remote_index_build.size.min` (50MB by default). At 384 dimensions, that is about 34,000 documents. The 260-recipe sample never reaches it, but the full corpus does. Each ingest ends with a force merge to one segment, so the GPU builds one large graph.
In our [benchmarks on NetApp Instaclustr](https://www.instaclustr.com/blog/build-opensearch-vector-indexes-faster-with-netapp-instaclustr-and-nvidia-cuvs/), GPU-accelerated builds were up to 6.7× faster for total index builds and 24× faster for force merges.
**Key takeaways for RAG retrieval**
-----------------------------------
Every failure in this article happened at the retrieval step, not in the model. The following summarizes what happened to the Fish and Chips question at each stage:
- Vector search found similar recipes but missed the thegratefulgirlcooks.com recipe the question named.
- BM25 with NER told the two Fish and Chips recipes apart but needed an extra service to do it.
- Hybrid search found the right recipe in one query, but only once the fusion weights and candidate window were set for the real corpus size.
- Ground then refine turned “no sulfites” into a boolean filter, so a sulfite recipe cannot come back by scoring well.
At full scale, GPU index builds keep the vector side fast enough to reindex as the corpus grows. The theme across all four stages is the same: test retrieval using production-sized data, because small samples hide the failures that matter.
**Try it on NetApp Instaclustr**
--------------------------------
All four stages run on a single OpenSearch cluster on [NetApp Instaclustr](https://console2.instaclustr.com/signup). The code, with setup steps for each stage and the GPU build settings, is at [OpenSearch/fix-your-retrieval](https://github.com/instaclustr/code-samples/tree/main/OpenSearch/2026-opensearchcon-fix-your-retrieval-gpu-cluster).
### About the author
**[Ramya Ravi](https://www.instaclustr.com/blog/author/ramya-ravi/)** | Developer Advocate
Ramya Ravi is a Developer Advocate at NetApp, specializing in open source and AI technologies. She connects Instaclustr supported technologies with emerging AI use cases, creating technical content, demos, and guides to help developers build and deploy AI applications. She has previously worked for Intel and TCS. In her previous role at Intel, she drove AI/ML ecosystem growth by developing tutorials and resources that supported thousands of AI developers across Reddit, the PyTorch community, and developer ecosystem forums.
[ Add Instaclustr as a preferred source on Google ](https://google.com/preferences/source?q=instaclustr.com)
![mail icon]()#### Get the latest articles for open sourceIn your inbox
### Related content
[ Zero Downtime Migration to Instaclustr
Yes, we can migrate existing Cassandra clusters to Instaclustr without any downtime. Here's what to expect from the process...
](https://www.instaclustr.com/blog/zero-downtime-migration-to-instaclustr/)
[ Workflow Comparison: Uber Cadence vs Netflix Conductor
When choosing what’s right for your company’s opensource workflow needs it is important to know the difference and similarities ...
](https://www.instaclustr.com/blog/workflow-comparison-uber-cadence-vs-netflix-conductor/)
[ Will Your Cassandra Database Project Succeed?: The New Stack
Open source Apache Cassandra® continues to stand out as an enterprise-proven solution for organizations seeking high availability...
](https://www.instaclustr.com/blog/will-your-cassandra-database-project-succeed-the-new-stack/)
×Sign upto ourNewsletter
-----------------------