# Introduction to similarity search with word embeddings: Part 1–Apache Cassandra® 4.0 and OpenSearch®
[Blog](/blog/)>[Technology](/blog/category/technical/)>Introduction to similarity search with word embeddings: Part 1–Apache Cassandra® 4.0 and OpenSearch®
Introduction to similarity search with word embeddings: Part 1–Apache Cassandra® 4.0 and OpenSearch®
====================================================================================================
March 05, 2025 | By [ Murilo Miranda](https://www.instaclustr.com/blog/author/murilo-miranda/)
[ ](https://x.com/intent/tweet?text=Introduction%20to%20similarity%20search%20with%20word%20embeddings:%20Part%201%E2%80%93Apache%20Cassandra%C2%AE%204.0%20and%20OpenSearch%C2%AE&url=https://www.instaclustr.com/blog/introduction-to-similarity-search-with-word-embeddings-part-1/) [ ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.instaclustr.com/blog/introduction-to-similarity-search-with-word-embeddings-part-1/&title=&summary=Introduction%20to%20similarity%20search%20with%20word%20embeddings:%20Part%201%E2%80%93Apache%20Cassandra%C2%AE%204.0%20and%20OpenSearch%C2%AE&source=)
Word embeddings have revolutionized how we approach tasks like natural language processing, search, and recommendation engines.
They allow us to convert words and phrases into numerical representations (vectors) that capture their meaning based on the context in which they appear. Word embeddings are especially useful for tasks where traditional keyword searches fall short, such as finding semantically similar documents or making recommendations based on textual data.
**For example**: a search for **“Laptop”** might return results related to **“Notebook”** or **“MacBook”** when using embeddings (as opposed to something like **“Tablet”)** offering a more intuitive and accurate search experience.
As applications increasingly rely on AI and machine learning to drive intelligent search and recommendation engines, the ability to efficiently handle word embeddings has become critical. That’s where databases like Apache Cassandra come into play—offering the scalability and performance needed to manage and query large amounts of vector data.
In Part 1 of this series, we’ll explore how you can leverage word embeddings for similarity searches using Cassandra 4 and OpenSearch. By combining Cassandra’s robust data storage capabilities with OpenSearch’s powerful search functions, you can build scalable and efficient systems that handle both metadata and word embeddings.
Cassandra 4 and OpenSearch: A partnership for embeddings
--------------------------------------------------------
Cassandra 4 doesn’t natively support vector data types or specific similarity search functions, but that doesn’t mean you’re out of luck. By integrating Cassandra with OpenSearch, an open-source search and analytics platform, you can store word embeddings and perform similarity searches using the k-Nearest Neighbors (kNN) plugin.
This hybrid approach is advantageous over relying on OpenSearch alone because it allows you to leverage Cassandra’s strengths as a high-performance, scalable database for data storage while using OpenSearch for its robust indexing and search capabilities.
Instead of duplicating large volumes of data into OpenSearch solely for search purposes, you can keep the original data in Cassandra. OpenSearch, in this setup, acts as an intelligent pointer, indexing the embeddings stored in Cassandra and performing efficient searches without the need to manage the entire dataset directly.
This approach not only optimizes resource usage but also enhances system maintainability and scalability by segregating storage and search functionalities into specialized layers.
### Deploying the environment
To set up your environment for word embeddings and similarity search, you can leverage the [Instaclustr Managed Platform](https://www.instaclustr.com/), which simplifies deploying and managing your Cassandra cluster and OpenSearch. Instaclustr takes care of the heavy lifting, allowing you to focus on building your application rather than managing infrastructure. In this configuration, Cassandra serves as your primary data store, while OpenSearch handles vector operations and similarity searches.
Here’s how to get started:
1. **Deploy a managed Cassandra cluster**: Start by [provisioning your Cassandra 4 cluster](https://www.instaclustr.com/support/documentation/cassandra/getting-started-with-cassandra/creating-a-cluster/) on the Instaclustr platform. This managed solution ensures your cluster is optimized, secure, and ready to store non-vector data.
2. **Set up OpenSearch with kNN plugin**: Instaclustr also offers a fully managed OpenSearch service. You will need to [deploy OpenSearch](https://www.instaclustr.com/support/documentation/opensearch/getting-started-with-opensearch/creating-an-opensearch-cluster/), with the [kNN plugin enabled](https://www.instaclustr.com/support/documentation/opensearch/using-opensearch-plugins/knn-plugin/), which is critical for handling word embeddings and executing similarity searches.
By using Instaclustr, you gain access to a robust platform that seamlessly integrates Cassandra and OpenSearch, combining Cassandra’s scalable, fault-tolerant database with OpenSearch’s powerful search capabilities. This managed environment minimizes operational complexity, so you can focus on delivering fast and efficient similarity searches for your application.
### Preparing the environment
Now that we’ve outlined the environment setup, let’s dive into the specific technical steps to prepare Cassandra and OpenSearch for storing and searching word embeddings.
#### **Step 1: Setting up Cassandra**
In Cassandra, we’ll need to create a table to store the metadata. Here’s how to do that:
1. **Create the Table**:
Next, create a table to store the embeddings. This table will hold details such as the embedding vector, related text, and metadata:CREATE KEYSPACE IF NOT EXISTS aisearch WITH REPLICATION = {‘class’: ‘SimpleStrategy’, ‘
CREATE KEYSPACE IF NOT EXISTS aisearch WITH REPLICATION = {'class': 'SimpleStrategy', ' replication\_factor': 3}; USE file\_metadata; DROP TABLE IF EXISTS file\_metadata; CREATE TABLE IF NOT EXISTS file\_metadata ( id UUID, paragraph\_uuid UUID, filename TEXT, text TEXT, last\_updated timestamp, PRIMARY KEY (id, paragraph\_uuid) );
1
2
3
4
5
6
7
8
9
10
11
12
13
14
CREATE KEYSPACE IF NOT EXISTS aisearch WITH REPLICATION = {'class': 'SimpleStrategy', '
replication\_factor': 3};
USE file\_metadata;
DROP TABLE IF EXISTS file\_metadata;
CREATE TABLE IF NOT EXISTS file\_metadata (
id UUID,
paragraph\_uuid UUID,
filename TEXT,
text TEXT,
last\_updated timestamp,
PRIMARY KEY (id, paragraph\_uuid)
);
#### **Step 2: Configuring OpenSearch**
In OpenSearch, you’ll need to create an index that supports vector operations for similarity search. Here’s how you can configure it:
1. **Create the index:**
Define the index settings and mappings, ensuring that vector operations are enabled and that the correct space type (e.g., L2) is used for similarity calculations.
{ "settings": { "index": { "number\_of\_shards": 2, "knn": true, "knn.space\_type": "l2" } }, "mappings": { "properties": { "file\_uuid": { "type": "keyword" }, "paragraph\_uuid": { "type": "keyword" }, "embedding": { "type": "knn\_vector", "dimension": 300 } } } }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
"settings": {
"index": {
"number\_of\_shards": 2,
"knn": true,
"knn.space\_type": "l2"
}
},
"mappings": {
"properties": {
"file\_uuid": {
"type": "keyword"
},
"paragraph\_uuid": {
"type": "keyword"
},
"embedding": {
"type": "knn\_vector",
"dimension": 300
}
}
}
}
This index configuration is optimized for storing and searching embeddings using the k-Nearest Neighbors algorithm, which is crucial for similarity search.
With these steps, your environment will be ready to handle word embeddings for similarity search using Cassandra and OpenSearch.
### Generating embeddings with FastText
Once you have your environment set up, the next step is to generate the word embeddings that will drive your similarity search. For this, we’ll use FastText, a popular library from Facebook’s AI Research team that provides pre-trained word vectors. Specifically, we’re using the **crawl-300d-2M** model, which offers 300-dimensional vectors for millions of English words.
#### Step 1: Download and load the FastText model
To start, you’ll need to download the pre-trained model file. This can be done easily using Python and the requests library. Here’s the process:
**1. Download the FastText model**: The FastText model is stored in a zip file, which you can [download from the official FastText](https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip) website. The following Python script will handle the download and extraction:
import requests import zipfile import os # Adjust file\_url and local\_filename variables accordingly file\_url = <a href="https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip">'https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip'</a> local\_filename = '/content/gdrive/MyDrive/0\_notebook\_files/model/crawl-300d-2M.vec.zip' extract\_dir = '/content/gdrive/MyDrive/0\_notebook\_files/model/' def download\_file(url, filename): with requests.get(url, stream=True) as r: r.raise\_for\_status() os.makedirs(os.path.dirname(filename), exist\_ok=True) with open(filename, 'wb') as f: for chunk in r.iter\_content(chunk\_size=8192): f.write(chunk) def unzip\_file(filename, extract\_to): with zipfile.ZipFile(filename, 'r') as zip\_ref: zip\_ref.extractall(extract\_to) # Download and extract download\_file(file\_url, local\_filename) unzip\_file(local\_filename, extract\_dir)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import requests
import zipfile
import os
\# Adjust file\_url and local\_filename variables accordingly
file\_url = <a href="https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip">'https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip'</a>
local\_filename = '/content/gdrive/MyDrive/0\_notebook\_files/model/crawl-300d-2M.vec.zip'
extract\_dir = '/content/gdrive/MyDrive/0\_notebook\_files/model/'
def download\_file(url, filename):
with requests.get(url, stream=True) as r:
r.raise\_for\_status()
os.makedirs(os.path.dirname(filename), exist\_ok=True)
with open(filename, 'wb') as f:
for chunk in r.iter\_content(chunk\_size=8192):
f.write(chunk)
def unzip\_file(filename, extract\_to):
with zipfile.ZipFile(filename, 'r') as zip\_ref:
zip\_ref.extractall(extract\_to)
\# Download and extract
download\_file(file\_url, local\_filename)
unzip\_file(local\_filename, extract\_dir)
**2. Load the model**: Once the model is downloaded and extracted, you’ll load it using Gensim’s KeyedVectors class. This allows you to work with the embeddings directly:
from gensim.models import KeyedVectors # Adjust model\_path variable accordingly model\_path = "/content/gdrive/MyDrive/0\_notebook\_files/model/crawl-300d-2M.vec" fasttext\_model = KeyedVectors.load\_word2vec\_format(model\_path, binary=False)
1
2
3
4
5
from gensim.models import KeyedVectors
\# Adjust model\_path variable accordingly
model\_path = "/content/gdrive/MyDrive/0\_notebook\_files/model/crawl-300d-2M.vec"
fasttext\_model = KeyedVectors.load\_word2vec\_format(model\_path, binary=False)
#### Step 2: Generate embeddings from text
With the FastText model loaded, the next task is to convert text into vectors. This process involves splitting the text into words, looking up the vector for each word in the FastText model, and then averaging the vectors to get a single embedding for the text.
Here’s a function that handles the conversion:
import numpy as np import re def text\_to\_vector(text): """Convert text into a vector using the FastText model.""" text = text.lower() words = re.findall(r'\\b\\w+\\b', text) vectors = \[fasttext\_model\[word\] for word in words if word in fasttext\_model.key\_to\_index\] if not vectors: print(f"No embeddings found for text: {text}") return np.zeros(fasttext\_model.vector\_size) return np.mean(vectors, axis=0)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import numpy as np
import re
def text\_to\_vector(text):
"""Convert text into a vector using the FastText model."""
text = text.lower()
words = re.findall(r'\\b\\w+\\b', text)
vectors = \[fasttext\_model\[word\] for word in words if word in fasttext\_model.key\_to\_index\]
if not vectors:
print(f"No embeddings found for text: {text}")
return np.zeros(fasttext\_model.vector\_size)
return np.mean(vectors, axis=0)
This function tokenizes the input text, retrieves the corresponding word vectors from the model, and computes the average to create a final embedding.
#### Step 3: Extract text and generate embeddings from documents
In real-world applications, your text might come from various types of documents, such as PDFs, Word files, or presentations. The following code shows how to extract text from different file formats and convert that text into embeddings:
import uuid import mimetypes import pandas as pd from pdfminer.high\_level import extract\_pages from pdfminer.layout import LTTextContainer from docx import Document from pptx import Presentation def generate\_deterministic\_uuid(name): return uuid.uuid5(uuid.NAMESPACE\_DNS, name) def generate\_random\_uuid(): return uuid.uuid4() def get\_file\_type(file\_path): # Guess the MIME type based on the file extension mime\_type, \_ = mimetypes.guess\_type(file\_path) return mime\_type def extract\_text\_from\_excel(excel\_path): xls = pd.ExcelFile(excel\_path) text\_list = \[\] for sheet\_index, sheet\_name in enumerate(xls.sheet\_names): df = xls.parse(sheet\_name) for row in df.iterrows(): text\_list.append((" ".join(map(str, row\[1\].values)), sheet\_index + 1)) # +1 to make it 1 based index return text\_list def extract\_text\_from\_pdf(pdf\_path): return \[(text\_line.get\_text().strip().replace('\\xa0', ' '), page\_num) for page\_num, page\_layout in enumerate(extract\_pages(pdf\_path), start=1) for element in page\_layout if isinstance(element, LTTextContainer) for text\_line in element if text\_line.get\_text().strip()\] def extract\_text\_from\_word(file\_path): doc = Document(file\_path) return \[(para.text, (i == 0) + 1) for i, para in enumerate(doc.paragraphs) if para.text.strip()\] def extract\_text\_from\_txt(file\_path): with open(file\_path, 'r') as file: return \[(line.strip(), 1) for line in file.readlines() if line.strip()\] def extract\_text\_from\_pptx(pptx\_path): prs = Presentation(pptx\_path) return \[(shape.text.strip(), slide\_num) for slide\_num, slide in enumerate(prs.slides, start=1) for shape in slide.shapes if hasattr(shape, "text") and shape.text.strip()\] def extract\_text\_with\_page\_number\_and\_embeddings(file\_path, embedding\_function): file\_uuid = generate\_deterministic\_uuid(file\_path) file\_type = get\_file\_type(file\_path) extractors = { 'text/plain': extract\_text\_from\_txt, 'application/pdf': extract\_text\_from\_pdf, 'application/vnd.openxmlformats-officedocument.wordprocessingml.document': extract\_text\_from\_word, 'application/vnd.openxmlformats-officedocument.presentationml.presentation': extract\_text\_from\_pptx, 'application/zip': lambda path: extract\_text\_from\_pptx(path) if path.endswith('.pptx') else \[\], 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': extract\_text\_from\_excel, 'application/vnd.ms-excel': extract\_text\_from\_excel } text\_list = extractors.get(file\_type, lambda \_: \[\])(file\_path) return \[ { "uuid": file\_uuid, "paragraph\_uuid": generate\_random\_uuid(), "filename": file\_path, "text": text, "page\_num": page\_num, "embedding": embedding } for text, page\_num in text\_list if (embedding := embedding\_function(text)).any() # Check if the embedding is not all zeros \] # Replace the file path with the one you want to process file\_path = "../../docs-manager/Cassandra-Best-Practices.pdf" paragraphs\_with\_embeddings = extract\_text\_with\_page\_number\_and\_embeddings(file\_path)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
import uuid
import mimetypes
import pandas as pd
from pdfminer.high\_level import extract\_pages
from pdfminer.layout import LTTextContainer
from docx import Document
from pptx import Presentation
def generate\_deterministic\_uuid(name):
return uuid.uuid5(uuid.NAMESPACE\_DNS, name)
def generate\_random\_uuid():
return uuid.uuid4()
def get\_file\_type(file\_path):
\# Guess the MIME type based on the file extension
mime\_type, \_ = mimetypes.guess\_type(file\_path)
return mime\_type
def extract\_text\_from\_excel(excel\_path):
xls = pd.ExcelFile(excel\_path)
text\_list = \[\]
for sheet\_index, sheet\_name in enumerate(xls.sheet\_names):
df = xls.parse(sheet\_name)
for row in df.iterrows():
text\_list.append((" ".join(map(str, row\[1\].values)), sheet\_index + 1)) \# +1 to make it 1 based index
return text\_list
def extract\_text\_from\_pdf(pdf\_path):
return \[(text\_line.get\_text().strip().replace('\\xa0', ' '), page\_num)
for page\_num, page\_layout in enumerate(extract\_pages(pdf\_path), start=1)
for element in page\_layout if isinstance(element, LTTextContainer)
for text\_line in element if text\_line.get\_text().strip()\]
def extract\_text\_from\_word(file\_path):
doc = Document(file\_path)
return \[(para.text, (i == 0) + 1) for i, para in enumerate(doc.paragraphs) if para.text.strip()\]
def extract\_text\_from\_txt(file\_path):
with open(file\_path, 'r') as file:
return \[(line.strip(), 1) for line in file.readlines() if line.strip()\]
def extract\_text\_from\_pptx(pptx\_path):
prs = Presentation(pptx\_path)
return \[(shape.text.strip(), slide\_num) for slide\_num, slide in enumerate(prs.slides, start=1)
for shape in slide.shapes if hasattr(shape, "text") and shape.text.strip()\]
def extract\_text\_with\_page\_number\_and\_embeddings(file\_path, embedding\_function):
file\_uuid = generate\_deterministic\_uuid(file\_path)
file\_type = get\_file\_type(file\_path)
extractors = {
'text/plain': extract\_text\_from\_txt,
'application/pdf': extract\_text\_from\_pdf,
'application/vnd.openxmlformats-officedocument.wordprocessingml.document': extract\_text\_from\_word,
'application/vnd.openxmlformats-officedocument.presentationml.presentation': extract\_text\_from\_pptx,
'application/zip': lambda path: extract\_text\_from\_pptx(path) if path.endswith('.pptx') else \[\],
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': extract\_text\_from\_excel,
'application/vnd.ms-excel': extract\_text\_from\_excel
}
text\_list = extractors.get(file\_type, lambda \_: \[\])(file\_path)
return \[
{
"uuid": file\_uuid,
"paragraph\_uuid": generate\_random\_uuid(),
"filename": file\_path,
"text": text,
"page\_num": page\_num,
"embedding": embedding
}
for text, page\_num in text\_list
if (embedding := embedding\_function(text)).any() \# Check if the embedding is not all zeros
\]
\# Replace the file path with the one you want to process
file\_path = "../../docs-manager/Cassandra-Best-Practices.pdf"
paragraphs\_with\_embeddings = extract\_text\_with\_page\_number\_and\_embeddings(file\_path)
This code handles extracting text from different document types, generating embeddings for each text chunk, and associating them with unique IDs.
With FastText set up and embeddings generated, you’re now ready to store these vectors in OpenSearch and start performing similarity searches.
Performing similarity searches
------------------------------
To conduct similarity searches, we utilize the k-Nearest Neighbors (kNN) plugin within OpenSearch. This plugin allows us to efficiently search for the most similar embeddings stored in the system. Essentially, you’re querying OpenSearch to find the closest matches to a word or phrase based on your embeddings.
For example, if you’ve embedded product descriptions, using kNN search helps you locate products that are semantically similar to a given input. This capability can significantly enhance your application’s recommendation engine, categorization, or clustering.
This setup with Cassandra and OpenSearch is a powerful combination, but it’s important to remember that it requires managing two systems. As Cassandra evolves, the introduction of built-in vector support in Cassandra 5 simplifies this architecture. But for now, let’s focus on leveraging both systems to get the most out of similarity searches.
### Example: Inserting metadata in Cassandra and embeddings in OpenSearch
In this example, we use Cassandra 4 to store metadata related to files and paragraphs, while OpenSearch handles the actual word embeddings. By storing the paragraph and file IDs in both systems, we can link the metadata in Cassandra with the embeddings in OpenSearch.
We first need to store metadata such as the file name, paragraph UUID, and other relevant details in Cassandra. This metadata will be crucial for linking the data between Cassandra, OpenSearch and the file itself in filesystem.
The following code demonstrates how to insert this metadata into Cassandra and embeddings in OpenSearch, make sure to run the previous script, so the “paragraphs\_with\_embeddings” variable will be populated:
from tqdm import tqdm # Function to insert data into both Cassandra and OpenSearch def insert\_paragraph\_data(session, os\_client, paragraph, keyspace\_name, index\_name): # Insert into Cassandra cassandra\_result = insert\_with\_retry( session=session, id=paragraph\['uuid'\], paragraph\_uuid=paragraph\['paragraph\_uuid'\], text=paragraph\['text'\], filename=paragraph\['filename'\], keyspace\_name=keyspace\_name, max\_retries=3, retry\_delay\_seconds=1 ) if not cassandra\_result: return False # Stop further processing if Cassandra insertion fails # Insert into OpenSearch opensearch\_result = insert\_embedding\_to\_opensearch( os\_client=os\_client, index\_name=index\_name, file\_uuid=paragraph\['uuid'\], paragraph\_uuid=paragraph\['paragraph\_uuid'\], embedding=paragraph\['embedding'\] ) if opensearch\_result is not None: return False # Return False if OpenSearch insertion fails return True # Return True on success for both # Process each paragraph with a progress bar print("Starting batch insertion of paragraphs.") for paragraph in tqdm(paragraphs\_with\_embeddings, desc="Inserting paragraphs"): if not insert\_paragraph\_data( session=session, os\_client=os\_client, paragraph=paragraph, keyspace\_name=keyspace\_name, index\_name=index\_name ): print(f"Insertion failed for UUID {paragraph\['uuid'\]}: {paragraph\['text'\]\[:50\]}...") print("Batch insertion completed.")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
from tqdm import tqdm
\# Function to insert data into both Cassandra and OpenSearch
def insert\_paragraph\_data(session, os\_client, paragraph, keyspace\_name, index\_name):
\# Insert into Cassandra
cassandra\_result = insert\_with\_retry(
session=session,
id=paragraph\['uuid'\],
paragraph\_uuid=paragraph\['paragraph\_uuid'\],
text=paragraph\['text'\],
filename=paragraph\['filename'\],
keyspace\_name=keyspace\_name,
max\_retries=3,
retry\_delay\_seconds=1
)
if not cassandra\_result:
return False \# Stop further processing if Cassandra insertion fails
\# Insert into OpenSearch
opensearch\_result = insert\_embedding\_to\_opensearch(
os\_client=os\_client,
index\_name=index\_name,
file\_uuid=paragraph\['uuid'\],
paragraph\_uuid=paragraph\['paragraph\_uuid'\],
embedding=paragraph\['embedding'\]
)
if opensearch\_result is not None:
return False \# Return False if OpenSearch insertion fails
return True \# Return True on success for both
\# Process each paragraph with a progress bar
print("Starting batch insertion of paragraphs.")
for paragraph in tqdm(paragraphs\_with\_embeddings, desc="Inserting paragraphs"):
if not insert\_paragraph\_data(
session=session,
os\_client=os\_client,
paragraph=paragraph,
keyspace\_name=keyspace\_name,
index\_name=index\_name
):
print(f"Insertion failed for UUID {paragraph\['uuid'\]}: {paragraph\['text'\]\[:50\]}...")
print("Batch insertion completed.")
#### Performing similarity search
Now that we’ve stored both metadata in Cassandra and embeddings in OpenSearch, it’s time to perform a similarity search. This step involves searching OpenSearch for embeddings that closely match a given input and then retrieving the corresponding metadata from Cassandra.
The process is straightforward: we start by converting the input text into an embedding, then use the k-Nearest Neighbors (kNN) plugin in OpenSearch to find the most similar embeddings. Once we have the results, we fetch the related metadata from Cassandra, such as the original text and file name.
Here’s how it works:
1. **Convert text to embedding**: Start by converting your input text into an embedding vector using the FastText model. This vector will serve as the query for our similarity search.
2. **Search OpenSearch for similar embeddings**: Using the KNN search capability in OpenSearch, we find the top k most similar embeddings. Each result includes the corresponding file and paragraph UUIDs, which help us link the results back to Cassandra.
3. **Fetch metadata from Cassandra**: With the UUIDs retrieved from OpenSearch, we query Cassandra to get the metadata, such as the original text and file name, associated with each embedding.
The following code demonstrates this process:
import uuid from IPython.display import display, HTML def find\_similar\_embeddings\_opensearch(os\_client, index\_name, input\_embedding, top\_k=5): """Search for similar embeddings in OpenSearch and return the associated UUIDs.""" query = { "size": top\_k, "query": { "knn": { "embedding": { "vector": input\_embedding.tolist(), "k": top\_k } } } } response = os\_client.search(index=index\_name, body=query) similar\_uuids = \[\] for hit in response\['hits'\]\['hits'\]: file\_uuid = hit\['\_source'\]\['file\_uuid'\] paragraph\_uuid = hit\['\_source'\]\['paragraph\_uuid'\] similar\_uuids.append((file\_uuid, paragraph\_uuid)) return similar\_uuids def fetch\_metadata\_from\_cassandra(session, file\_uuid, paragraph\_uuid, keyspace\_name): """Fetch the metadata (text and filename) from Cassandra based on UUIDs.""" file\_uuid = uuid.UUID(file\_uuid) paragraph\_uuid = uuid.UUID(paragraph\_uuid) query = f""" SELECT text, filename FROM {keyspace\_name}.file\_metadata WHERE id = ? AND paragraph\_uuid = ?; """ prepared = session.prepare(query) bound = prepared.bind((file\_uuid, paragraph\_uuid)) rows = session.execute(bound) for row in rows: return row.filename, row.text return None, None # Input text to find similar embeddings input\_text = "place" # Convert input text to embedding input\_embedding = text\_to\_vector(input\_text) # Find similar embeddings in OpenSearch similar\_uuids = find\_similar\_embeddings\_opensearch(os\_client, index\_name=index\_name, input\_embedding=input\_embedding, top\_k=10) # Fetch and display metadata from Cassandra based on the UUIDs found in OpenSearch for file\_uuid, paragraph\_uuid in similar\_uuids: filename, text = fetch\_metadata\_from\_cassandra(session, file\_uuid, paragraph\_uuid, keyspace\_name) if filename and text: html\_content = f""" <div style="margin-bottom: 10px;"> <p><b>File UUID:</b> {file\_uuid}</p> <p><b>Paragraph UUID:</b> {paragraph\_uuid}</p> <p><b>Text:</b> {text}</p> <p><b>File:</b> {filename}</p> </div> <hr/> """ display(HTML(html\_content))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
import uuid
from IPython.display import display, HTML
def find\_similar\_embeddings\_opensearch(os\_client, index\_name, input\_embedding, top\_k=5):
"""Search for similar embeddings in OpenSearch and return the associated UUIDs."""
query = {
"size": top\_k,
"query": {
"knn": {
"embedding": {
"vector": input\_embedding.tolist(),
"k": top\_k
}
}
}
}
response = os\_client.search(index=index\_name, body=query)
similar\_uuids = \[\]
for hit in response\['hits'\]\['hits'\]:
file\_uuid = hit\['\_source'\]\['file\_uuid'\]
paragraph\_uuid = hit\['\_source'\]\['paragraph\_uuid'\]
similar\_uuids.append((file\_uuid, paragraph\_uuid))
return similar\_uuids
def fetch\_metadata\_from\_cassandra(session, file\_uuid, paragraph\_uuid, keyspace\_name):
"""Fetch the metadata (text and filename) from Cassandra based on UUIDs."""
file\_uuid = uuid.UUID(file\_uuid)
paragraph\_uuid = uuid.UUID(paragraph\_uuid)
query = f"""
SELECT text, filename
FROM {keyspace\_name}.file\_metadata
WHERE id = ? AND paragraph\_uuid = ?;
"""
prepared = session.prepare(query)
bound = prepared.bind((file\_uuid, paragraph\_uuid))
rows = session.execute(bound)
for row in rows:
return row.filename, row.text
return None, None
\# Input text to find similar embeddings
input\_text = "place"
\# Convert input text to embedding
input\_embedding = text\_to\_vector(input\_text)
\# Find similar embeddings in OpenSearch
similar\_uuids = find\_similar\_embeddings\_opensearch(os\_client, index\_name=index\_name, input\_embedding=input\_embedding, top\_k=10)
\# Fetch and display metadata from Cassandra based on the UUIDs found in OpenSearch
for file\_uuid, paragraph\_uuid in similar\_uuids:
filename, text = fetch\_metadata\_from\_cassandra(session, file\_uuid, paragraph\_uuid,
keyspace\_name)
if filename and text:
html\_content = f"""
<div style="margin-bottom: 10px;">
<p><b>File UUID:</b> {file\_uuid}</p>
<p><b>Paragraph UUID:</b> {paragraph\_uuid}</p>
<p><b>Text:</b> {text}</p>
<p><b>File:</b> {filename}</p>
</div>
<hr/>
"""
display(HTML(html\_content))
This code demonstrates how to find similar embeddings in OpenSearch and retrieve the corresponding metadata from Cassandra. By linking the two systems via the UUIDs, you can build powerful search and recommendation systems that combine metadata storage with advanced embedding-based searches.
Conclusion and next steps: A powerful combination of Cassandra 4 and OpenSearch
-------------------------------------------------------------------------------
By leveraging the strengths of Cassandra 4 and OpenSearch, you can build a system that handles both metadata storage and similarity search. Cassandra efficiently stores your file and paragraph metadata, while OpenSearch takes care of embedding-based searches using the k-Nearest Neighbors algorithm. Together, these two technologies enable powerful, large-scale applications for text search, recommendation engines, and more.
Coming up in Part 2, we’ll explore how Cassandra 5 simplifies this architecture with built-in vector support and native similarity search capabilities.
Ready to try vector search with Cassandra and OpenSearch? [Spin up your first cluster](https://console2.instaclustr.com/signup?_gl=1*i2o9gc*_gcl_au*MjUwNzcwNzUxLjE3MzI0OTE2NzU.*_ga*MTE4NDE2NDQ4NC4xNzE2ODU2NDc1*_ga_4NBQSJMP6D*MTczODg5MzgyOS40NTQuMS4xNzM4ODk0ODc0LjYwLjAuMA..&_ga=2.156842172.665239454.1738889425-1184164484.1716856475) for free on the Instaclustr Managed Platform and explore the incredible power of vector search.
### About the author
**[Murilo Miranda](https://www.instaclustr.com/blog/author/murilo-miranda/)** | Professional Services Consultant
Murilo has demonstrated exceptional expertise in open source technologies, contributing significantly to projects like Apache Cassandra. Murilo's dedication to open source has not only advanced the field but also fostered a collaborative and innovative environment within NetApp.
[ Add Instaclustr as a preferred source on Google ](https://google.com/preferences/source?q=instaclustr.com)
![mail icon]()#### Get the latest articles for open sourceIn your inbox
### Related content
[ Zero Downtime Migration to Instaclustr
Yes, we can migrate existing Cassandra clusters to Instaclustr without any downtime. Here's what to expect from the process...
](https://www.instaclustr.com/blog/zero-downtime-migration-to-instaclustr/)
[ Workflow Comparison: Uber Cadence vs Netflix Conductor
When choosing what’s right for your company’s opensource workflow needs it is important to know the difference and similarities ...
](https://www.instaclustr.com/blog/workflow-comparison-uber-cadence-vs-netflix-conductor/)
[ Will Your Cassandra Database Project Succeed?: The New Stack
Open source Apache Cassandra® continues to stand out as an enterprise-proven solution for organizations seeking high availability...
](https://www.instaclustr.com/blog/will-your-cassandra-database-project-succeed-the-new-stack/)
×Sign upto ourNewsletter
-----------------------