Tech Corner

Understanding cognitive search, its abilities, and its scope

Anwesh Roy
last edited on
October 25, 2023
2-3 mins

Table of contents

Automate your business at $5/day with Engati

Switch to Engati: Smarter choice for WhatsApp Campaigns 🚀
What is cognitive search

During these tough times, remote workers find themselves struggling to serve customers, handle employees, and complete tasks to keep businesses afloat. Imagine having quick access to relevant information. Read on to explore ways to handle these challenges.

The problem

When employees were working in-office, they know how to get the information they need to perform their job. They also have access to tools that enabled these tasks. However, when remote, this flexibility is absent. This could lead to lowered productivity, and could cause a dip in efficiency.

One of the ways technology can come to the rescue is by enabling fast access to relevant information that aligns with human thinking. Cognitive Search is one such technology.

The solution – Cognitive Search

What is Cognitive Search?

A new generation of enterprise search solutions that employ AI technologies such as natural language processing and machine learning to ingest, understand, organize, and query digital content from multiple data sources.
- Forrester

The need for quick information has been heightened by Google and the emergence of sophisticated Ecommerce search engines. Now users demand for more information in shorter periods of time. Natural Language Processing (NLP) powers the next generation search technologies. This has been possible due to remarkable breakthroughs in using advanced Deep Learning technologies to process natural language and understand it (NLU).

The picture below depicts the evolution of ‘Search’ technology the value it brings to business.

The evolution of search, from keyword to cognitive search
Cognitive search

As you can see, Cognitive Search takes human-computer interaction (HCI) to the next level. And for the first time, it enables humans to interact with the machine in the most natural way possible. Gone are the days of confusing menus. The traditional UI is giving way to Conversational AI, which forms the backbone of chatbot technologies like Engati.

A history of Cognitive Search

In the last couple of years, we’ve seen companies like Google, Microsoft, Open AI, Facebook, and many more attempting to produce improved language models. The reason? To enable contextual and sematic search which have now become the building blocks for Cognitive Search- in addition to fast inference and scale.

Last year, both Google and Microsoft announced that they are using “BERT" to take Google and Bing respectively to the next level. BERT, an advanced deep learning-based language model, enables more contextual and meaning-based matches of documents to user queries, which are not keyword based. This advanced search technology has proven that it can now be used in other use cases like ecommerce, document search, enterprise search, and so on. The possibilities are endless.

At Engati, we have chosen this advanced AI/NLP technology to enable Cognitive Search via Chatbots. We’ve broken down all barriers to information access to truly provide information at your fingertips.

How Cognitive Search gives you the upper hand

Cognitive search holds the promise of augmenting user’s intelligence. Be it employees or customers, cognitive search enables faster and smarter decisions and additionally, enhances productivity. Not only that, but it will also provide a continuous supply of relevant information to cater to the insatiable thirst for information.

Enabling cognitive search

In simple terms, we're trying to find the most relevant documents and passages in those documents that have a high contextual and semantic similarity to an input search query.

Below is a reference architecture of a typical Question/Answer system that can be utilized for building a Cognitive Search system.

architecture for question/answer system in cognitive search
How Cognitive Search systems works

The starting point is to setup a document store that contains the searchable documents with some sort of indexes to quicken the search. If the order of such documents is over a million, then it makes sense to use a text-based search engine like Elasticsearch or Solr to generate inverted indexes at ‘word’ level and store the passages as searchable documents.

The retriever is responsible for using the query to search for similar documents in the document store. It then retrieves the top K document passages based on a predefined cutoff threshold.

For getting the top K matches from a text indexing system, algorithms like TF-IDF or BM25 are used. These algorithms focus on exact word, or phrase matches based on weighting factor that focus on unique terms in the documents.

The shortcomings of using this approach is that the context of words, meaning of the query or passage, polysemy occurrences are not handled. This could lead to losing out some of the relevant documents in the initial shortlisting.  

If the goal is to quickly get to the candidate documents- this is one of the best ways of doing that.

The reader is a powerful neural model that reads through the top K candidate passages and uses contextual and semantic similarity to create a shortlist of the most relevant matches between the query vector and passage/sentence vectors using cosine similarity.

Nowadays, most of the reader models are large pre-trained language models like BERT, Roberta, XLNet, ALBERT or distilBERT. All of these are transformer-based language models that have been trained for next word, next sentence prediction.  

Using transfer learning techniques these pre-trained models can also be fine-tuned to get domain specific encoders.

A ranking algorithm can be used to rank the output of the reader to arrange the most relevant documents and corresponding passages first.  

The above mechanism to build a cognitive search engine is an unsupervised technique. It will require heuristic rules in the ranker to prune out false positives or perform re-ranking to get better results.

Supervised technique can be in the form of a Learning to Rank (LTR) algorithm to personalise the ranking, based on user’s feedback.

The reader is computationally intensive and will need a GPU server for larger models or multi-core CPUs for smaller models.  

The scope for Cognitive Search

Cognitive Search will open up access to thousands of documents which are available in an enterprise and allow users to lookup these documents within minutes rather than reading through those documents to look for an answer to their question or enhance their learning.

Any use case that needs shifting through thousands of voluminous text documents in an intelligent way that would mimic how human beings read and comprehend information is prime for using Cognitive Search.

Cognitive Search can also be used to enhance results generated from current text-based search engines like Elasticsearch or SOLR. By encoding context and intent into search, E-commerce, news websites, search engines research websites etc. can employ cognitive search to enhance the quality of search results and improve user engagement.

The future of search is going to be exciting, so stay tuned! Until then, start adopting modern AI solutions with Engati’s solutions.

Anwesh Roy

Anwesh is the Senior Vice President of Engati. Driven by a passion to deliver value through AI-driven solutions, Anwesh is on a mission to mainstream Natural Language Processing (NLP), Natural Language Understanding (NLU), Natural Language Generation (NLG) and Data Analytics applications.

Close Icon
Request a Demo!
Get started on Engati with the help of a personalised demo.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
*only for sharing demo link on WhatsApp
Thanks for the information.
We will be shortly getting in touch with you.
Oops! something went wrong!
For any query reach out to us on
Close Icon
Congratulations! Your demo is recorded.

Select an option on how Engati can help you.

I am looking for a conversational AI engagement solution for the web and other channels.

I would like for a conversational AI engagement solution for WhatsApp as the primary channel

I am an e-commerce store with Shopify. I am looking for a conversational AI engagement solution for my business

I am looking to partner with Engati to build conversational AI solutions for other businesses

Close Icon
You're a step away from building your Al chatbot

How many customers do you expect to engage in a month?

Less Than 2000


More than 5000

Close Icon
Thanks for the information.

We will be shortly getting in touch with you.

Close Icon

Contact Us

Please fill in your details and we will contact you shortly.

This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
Thanks for the information.
We will be shortly getting in touch with you.
Oops! Looks like there is a problem.
Never mind, drop us a mail at