Search has evolved vastly in a short time, and there are now multiple search paradigms to learn, like the difference between lexical search and semantic search. To build modern applications, it’s essential to understand the core differences in search paradigms, especially when comparing lexical search vs semantic search.

This article explains the differences between these two search methods. We will explore what each type of search is, how it works behind the scenes, and its ideal use cases. We’ll also cover the idea of hybrid search techniques that leverage the abilities of both lexical and semantic search to create even better and faster search results.

What is lexical search?

Lexical search is, in essence, the traditional keyword search you think of when you think of early search engines, like library catalogs. They operate by matching keywords or tokens against a tokenized version of the data to be searched. In lexical search, all the search knows is the key token it is searching for, and the tokens of the data to be searched, and the key token must at least approximate the results you’re looking for—there’s no context to use to clear any questions up.

How does lexical search work?

Lexical search works using traditional databases like SQL and NoSQL document databases; the search looks through each chunk of data in every relevant row for the token it’s searching for. This can limit it when it comes to misspellings; you can use techniques to work around them a bit, but you can’t work around the lack of context; for instance, if you search ‘BBQ’ a lexical search will not know to also search for ‘barbeque’.

What is lexical search good at?

What lexical search lacks in context and flexibility, it makes up for in speed. Between database technologies and search algorithms evolving over the years, it’s very fast at what it does! Another benefit of lexical search is it’s easier to conceptualize and talk about than lexical search—as you’ll see shortly. The lack of flexibility can be seen as a pure simplicity. On that note, lexical search is less computationally and resource-expensive, by a mile, than semantic search. It’s cheaper to run a lexical search application than a semantic one of the same size.

When do you use lexical search?

Lexical search is the perfect tool when precision is paramount, and the search terms are specific. When you know the name or value exactly of the item you’re looking for, like the title of a book or a group of books under a Dewey decimal system number, lexical search is worth considering. Also, when you don’t need similar items—just the exact item you’re looking for—this is where lexical search shines.

Consider using it in scenarios such as:

  • Searching for a specific product code or SKU in an e-commerce inventory
  • Looking up a person by their exact name or user ID in a database
  • Finding documents that contain a specific legal phrase or technical term
  • Filtering logs or records based on exact error codes or identifiers

What is semantic search?

Earlier I mentioned that lexical search lacked context; this is where semantic search comes in. Semantic search uses Large Language Models (LLMs) and ML/AI to find not just the exact thing you’re searching for, but items that are conceptually like it, or similar to it, or related in some other way.

Semantic search is search that, using ML/AI technology, gives a search engine the ability to parse the query not just syntactically but semantically (hence the name). This allows semantic search queries to return results that don’t match syntactically with the result but are relevant regardless. This would in fact return ‘barbeque’ results for a query of ‘BBQ’.

How does semantic search work?

The magic behind semantic search lies in vector embeddings. To understand how it works, we first need to cover what vectors are in this context.

Vectors and vector embeddings

Semantic search heavily relies on the concept of vectors and vector embeddings. The short explanation of these is a vector is a numerical representation of a direction and magnitude (or length), and a vector embedding is a vector representation of non-numeric data, like words, concepts, etc. These embeddings are created by putting the data through an ML/AI model that turns the data point into a vector. If you’re feeling confused, check out this educational article to learn more about vectors and vector embeddings.

The semantic search process

When you make a semantic search query, you typically first turn your query into an embedding that you can use to compare against existing embeddings within the vector database. You would most likely use an ML/AI model to do so; this way your query has the same context and dimensions as your data vectors.

Once you have your query as a vector your vector database will use an algorithm that compares vectors in space against each other using distance algorithms to see which vectors it should include in the result. Several algorithms exist, such Knn or K nearest neighbor, and ANN (Approximate nearest neighbor). You can learn more about these algorithms here.

Why distance?

Because the AI/ML models used to create the vector embeddings are designed to place items that are contextually related close to each other. For instance, LLMs or Large Language Models parse human words into vector embeddings with dimensions that allow searchers to find words related to their query but not explicitly included.

Once the vector database has the result set of vectors, it passes them back as their respective data points. For instance, items with ‘BBQ’ would show up in a search for ‘barbeque’; if the model correctly categorizes these items as related, they’ll be close in vector space.

What is semantic search good at?

Semantic search is great any time you want your results to have some form of context. Its key strength is the ability to relate concepts to one another when searching and finding relevant results. If you want your search to know more than a sequence of characters matches, you’ll look to semantic search.

When to use semantic search

Semantic search is ideal for any application where user intent might be broad or not easily expressed with specific keywords.

Common use cases include:

  • E-commerce. Recommending products that complement items in a user’s cart or finding products based on descriptive queries like “warm jacket for hiking.”
  • Content discovery. Suggesting articles, videos, or music based on a user’s viewing history and interests.
  • Customer support. Powering chatbots or knowledge bases to find answers to questions phrased in natural language.
  • Travel and hospitality. Recommending local attractions or restaurants based on a user’s travel plans and preferences.

Again, any time you want contextual results, look to semantic.

That being said, a new technique is emerging in the search scene: creating a hybrid search using both semantic and lexical search.

Hybrid search: The best of both worlds

The idea of hybrid search appeals to those who want the performance and reliability of lexical search results, but also contextual results that could be relevant, but the results aren’t useless if they’re not. There are at least a few ways you could implement something like this—here are a few examples:

  • A lexical search gets the tourist and other destinations for a given city, then a semantic search filters these for dimensions the user specifies in their query, returning the destinations that best suit the user’s needs.
  • A semantic search grabs everything it can regarding a given topic, and a lexical search could be used to reduce this down into a usable list, like a web search.
  • When a user searches a shopping site for a particular brand, a lexical search could return all the items of that brand, and a second semantic search would be used to find recommendations for other items.

There are many things you can use hybrid search techniques for, and you can tune your use of each to get more accurate results, as fast as reasonably possible.

Understanding the differences between lexical and semantic search is no longer just an academic exercise—it’s a crucial step in building powerful, modern applications. We’ve seen that lexical search offers unmatched speed and precision for exact-match queries, making it a reliable tool when you know exactly what you’re looking for. On the other hand, semantic search unlocks a deeper, contextual understanding of user intent, delivering relevant results that go beyond simple keywords.

The true power, however, often lies in combining these approaches. Hybrid search techniques offer the best of both worlds: the pinpoint accuracy of lexical search and the intelligent, context-aware capabilities of semantic search. By leveraging both, you can create a search experience that is not only fast and efficient but also incredibly intuitive and responsive to your users’ needs.

Ready to dive deeper and explore how advanced search technologies can transform your applications? Check out these additional resources:

If you’d like to play with search via the OpenSearch search framework, get started for free today.