<!-- JSON-LD markup generated by Google Structured Data Markup Helper. --><script type="application/ld+json">{  "@context" : "http://schema.org",  "@type" : "Article",  "name" : "Introducing Cognitive Search",  "author" : {    "@type" : "Person",    "name" : "Anwesh Roy"  },  "image" : "https://global-uploads.webflow.com/5ef788f07804fb7d78a4127a/5f2d69ab4b36880316ab254c_Introducing-Cognitive-Search.png",  "articleSection" : "What is Cognitive Search?",  "articleBody" : [ "A new generation of enterprise search solutions that employ AI technologies such as natural language processing and machine learning to ingest, understand, organize, and query digital content from multiple data sources. Cognitive search holds the promise of augmenting user’s intelligence. Be it employees or customers, cognitive search enables faster and smarter decisions and additionally, enhances productivity. Not only that, but it will also provide a continuous supply of relevant information to cater to the insatiable thirst for information." ],  "url" : "https://www.engati.com/blog/introducing-cognitive-search",  "publisher" : {    "@type" : "Organization",    "name" : "Engati"  }}</script>

Tech Corner

Understanding cognitive search, its abilities, and its scope

Anwesh Roy
.
Aug 4
.
2-3 mins

Table of contents

Key takeawaysCollaboration platforms are essential to the new way of workingEmployees prefer engati over emailEmployees play a growing part in software purchasing decisionsThe future of work is collaborativeMethodology

During these tough times, remote workers find themselves struggling to serve customers, handle employees, and complete tasks to keep businesses afloat. Imagine having quick access to relevant information. Read on to explore ways to handle these challenges.

The problem

When employees were working in-office, they know how to get the information they need to perform their job. They also have access to tools that enabled these tasks. However, when remote, this flexibility is absent. This could lead to lowered productivity, and could cause a dip in efficiency.

One of the ways technology can come to the rescue is by enabling fast access to relevant information that aligns with human thinking. Cognitive Search is one such technology.

The solution – Cognitive Search

What is Cognitive Search?

A new generation of enterprise search solutions that employ AI technologies such as natural language processing and machine learning to ingest, understand, organize, and query digital content from multiple data sources.
- Forrester

The need for quick information has been heightened by Google and the emergence of sophisticated Ecommerce search engines. Now users demand for more information in shorter periods of time. Natural Language Processing (NLP) powers the next generation search technologies. This has been possible due to remarkable breakthroughs in using advanced Deep Learning technologies to process natural language and understand it (NLU).

The picture below depicts the evolution of ‘Search’ technology the value it brings to business.

The evolution of search, from keyword to cognitive search
Cognitive search

As you can see, Cognitive Search takes human-computer interaction (HCI) to the next level. And for the first time, it enables humans to interact with the machine in the most natural way possible. Gone are the days of confusing menus. The traditional UI is giving way to Conversational AI, which forms the backbone of chatbot technologies like Engati.

A history of Cognitive Search

In the last couple of years, we’ve seen companies like Google, Microsoft, Open AI, Facebook, and many more attempting to produce improved language models. The reason? To enable contextual and sematic search which have now become the building blocks for Cognitive Search- in addition to fast inference and scale.

Last year, both Google and Microsoft announced that they are using “BERT" to take Google and Bing respectively to the next level. BERT, an advanced deep learning-based language model, enables more contextual and meaning-based matches of documents to user queries, which are not keyword based. This advanced search technology has proven that it can now be used in other use cases like ecommerce, document search, enterprise search, and so on. The possibilities are endless.

At Engati, we have chosen this advanced AI/NLP technology to enable Cognitive Search via Chatbots. We’ve broken down all barriers to information access to truly provide information at your fingertips.

How Cognitive Search gives you the upper hand

Cognitive search holds the promise of augmenting user’s intelligence. Be it employees or customers, cognitive search enables faster and smarter decisions and additionally, enhances productivity. Not only that, but it will also provide a continuous supply of relevant information to cater to the insatiable thirst for information.

Enabling cognitive search

In simple terms, we're trying to find the most relevant documents and passages in those documents that have a high contextual and semantic similarity to an input search query.

Below is a reference architecture of a typical Question/Answer system that can be utilized for building a Cognitive Search system.

architecture for question/answer system in cognitive search
How Cognitive Search systems works

The starting point is to setup a document store that contains the searchable documents with some sort of indexes to quicken the search. If the order of such documents is over a million, then it makes sense to use a text-based search engine like Elasticsearch or Solr to generate inverted indexes at ‘word’ level and store the passages as searchable documents.

The retriever is responsible for using the query to search for similar documents in the document store. It then retrieves the top K document passages based on a predefined cutoff threshold.

For getting the top K matches from a text indexing system, algorithms like TF-IDF or BM25 are used. These algorithms focus on exact word, or phrase matches based on weighting factor that focus on unique terms in the documents.

The shortcomings of using this approach is that the context of words, meaning of the query or passage, polysemy occurrences are not handled. This could lead to losing out some of the relevant documents in the initial shortlisting.  

If the goal is to quickly get to the candidate documents- this is one of the best ways of doing that.

The reader is a powerful neural model that reads through the top K candidate passages and uses contextual and semantic similarity to create a shortlist of the most relevant matches between the query vector and passage/sentence vectors using cosine similarity.

Nowadays, most of the reader models are large pre-trained language models like BERT, Roberta, XLNet, ALBERT or distilBERT. All of these are transformer-based language models that have been trained for next word, next sentence prediction.  

Using transfer learning techniques these pre-trained models can also be fine-tuned to get domain specific encoders.

A ranking algorithm can be used to rank the output of the reader to arrange the most relevant documents and corresponding passages first.  

The above mechanism to build a cognitive search engine is an unsupervised technique. It will require heuristic rules in the ranker to prune out false positives or perform re-ranking to get better results.

Supervised technique can be in the form of a Learning to Rank (LTR) algorithm to personalise the ranking, based on user’s feedback.

The reader is computationally intensive and will need a GPU server for larger models or multi-core CPUs for smaller models.  

The scope for Cognitive Search

Cognitive Search will open up access to thousands of documents which are available in an enterprise and allow users to lookup these documents within minutes rather than reading through those documents to look for an answer to their question or enhance their learning.

Any use case that needs shifting through thousands of voluminous text documents in an intelligent way that would mimic how human beings read and comprehend information is prime for using Cognitive Search.

Cognitive Search can also be used to enhance results generated from current text-based search engines like Elasticsearch or SOLR. By encoding context and intent into search, E-commerce, news websites, search engines research websites etc. can employ cognitive search to enhance the quality of search results and improve user engagement.

The future of search is going to be exciting, so stay tuned! Until then, start adopting modern AI solutions with Engati’s solutions.

Share
Share

Anwesh Roy

Anwesh is the Senior Vice President of Engati. Driven by a passion to deliver value through AI-driven solutions, Anwesh is on a mission to mainstream Natural Language Processing (NLP), Natural Language Understanding (NLU), Natural Language Generation (NLG) and Data Analytics applications.

Andy is the Co-Founder and CIO of SwissCognitive - The Global AI Hub. He’s also the President of the Swiss IT Leadership Forum.

Andy is a digital enterprise leader and is transforming business strategies keeping the best interests of shareholders, customers, and employees in mind.

Follow him for your daily dose of AI news and thoughts on using AI to improve your business.

Catch our interview with Andy on AI in daily life

Continue Reading