Tech Corner

Here's why sentence similarity is a tough NLP problem

Anwesh Roy
.
last edited on
.
October 26, 2023
3-4 mins

Table of contents

Automate your business at $5/day with Engati

REQUEST A DEMO
Switch to Engati: Smarter choice for WhatsApp Campaigns 🚀
TRY NOW
Sentence similarity is a tough NLP problem

With the advent of chatbots, training computers to read, understand, and write language has become a big business. The training may seem easy at first but as you start your Natural Language Processing (NLP) journey, you realize that surmounting the challenges is no easy task. That's why sentence similarity is amongst the toughest NLP problems.

Why is it so hard to teach language to a computer?

After all, a child picks up languages intuitively. That’s because computers are good at number crunching but are not capable of reading or understanding human language directly. For example, when we say, ‘An apple a day keeps you healthy’, how do you teach the computer that ‘Apple’ in this context is a fruit and not the company, 'Apple Inc.'

We take great pains to transform words and sentences into numerical representations and then train the computers with these numerical representations of language.

These representations have to capture the meaning of words (semantics), how they occur in a sentence (syntax), the contextual conversation, and the intertwining of words.

A new phase

NLP got a big boost when Tomas Mikolov at Google invented word2vec. Word2vec is a method to convert a word into a representation in an n-dimensional vector space which is referred to as word embedding. Glove is another method from Stanford that creates vector representations of words.

One of the popular and new methods is fastText from Facebook. These methods make it easy for the computer to compare words based on the context and the meaning. Armed with this technique, let’s see if we can extend this comparison to complete sentences or find similarities between sentences.

Spacy, a popular open-source NLP engine, provides an out-of-the-box feature to find similarity between sentences.

Spacy sentence similarity model
Spacy helps find similarity between sentences

The numbers represent sentence similarity. The greater the similarity value, the more similar the sentences are. The first two sentences are more similar since a city and country occur in them. However, why do we get a 0.65 and 0.55 similarity with the third one?

1

Very similar sentences

https://dandelion.eu/semantic-text/text-similarity-demo/?text1=Apple+sells+iOS+smartphones.&text2=Google+sells+Android+smartphones&lang=auto&exec=true

The sentences used are: ‘Apple sells iOS smartphones.’ and ‘Google sells Android smartphones.’

The semantic similarity is 83%, which is decent.

Let’s fool the model by stating something that is not a fact.

https://dandelion.eu/semantic-text/text-similarity-demo/?text1=Apple+sells+iOS+smartphones.&text2=Google+sells+iOS+smartphones.&lang=auto&exec=true

The sentences used are: ‘Apple sells iOS smartphones.’ and ‘Google sells iOS smartphones.’

The semantic similarity is 87% since we have similar sentences; however, the second sentence is factually incorrect.

Notice how replacing Android with iOS in the second sentence enhanced the similarity from 83% to 87%.
The model used here has not been trained with ‘facts’ but trained to compare sentences semantically.
Would a human perform better? Yes, only if the human has the knowledge that Google does not sell iOS phones.
So, by imparting this knowledge to the model it can perform at the same level as a human.

2

Somewhat similar sentences

https://dandelion.eu/semantic-text/text-similarity-demo/?text1=Apple+invented+iOS.&text2=Google+bought+Android.&lang=auto&exec=true

The sentences used are: ‘Apple invented iOS.’ and ‘Google bought Android.’

The semantic similarity is 75%, which may be acceptable.

3x Your Revenue With Chatbot And Live Chat
install for free

3

Dissimilar sentences

https://dandelion.eu/semantic-text/text-similarity-demo/?text1=Apple+invented+iOS.&text2=Apple+a+day+keeps+you+healthy%2C+as+shown+in+an+iOS+application&lang=auto&exec=true

The sentences used are: ‘Apple invented iOS.’ and ‘Apple a day keeps you healthy, as shown in an iOS application.’

The semantic similarity is 92%, which is very high for sentences that are dissimilar.
The ‘Apple’ and ‘iOS’ words are common between the two sentences, but the meaning of both sentences is quite different.

The results from the above tests prove that we have a long way to go for coming up with models that can do an acceptable job.

By default, most semantic similarity models take an average of the word vectors in a sentence to come up with a sentence-level vector or a vector that represents the meaning of the sentence in an n-dimensional vector space.

It then compares the sentence level vectors of the two sentences by using the cosine similarity method to come up with the similarity %.

Cosine in sentence similarity

It is a measurement of similarity between two non-zero vectors of an inner product space that measure the cosine of the angle between them.

Isn’t this non-intuitive? Would a human compare sentences in the same manner as this?

Recent developments in Deep Learning have shown promise that semantic similarity at a sentence level can be solved with better accuracy using recurrent and recursive neural networks. This will be the subject of discussion in a future post.

In text mining, sentence similarity is used as a criterion to discover unseen knowledge from textual databases.
- Sentence similarity study on Researchgate.

Get the Engati advantage

At Engati, we have some of the best in the industry at our office who can help you learn more about NLP, machine learning or chatbot technology. Our expertise will better help you understand the technology and make use of it at work.

Businesses can build their own chatbot for free and can upgrade as per their need and requirement. Engati comes with 50+ languages, and businesses can select their preferred language. Therefore, get started with chatbot technology. In fact, begin at the earliest because the chatbot only gets better with time because it becomes mature with more data.

Ever wondered what is stopping your business from reaching its full potential? Register with Engati today and start building your free chatbot!

3x your revenue with Chatbots and Live Chat
Schedule a demo

Anwesh Roy

Anwesh is the Senior Vice President of Engati. Driven by a passion to deliver value through AI-driven solutions, Anwesh is on a mission to mainstream Natural Language Processing (NLP), Natural Language Understanding (NLU), Natural Language Generation (NLG) and Data Analytics applications.

Close Icon
Request a Demo!
Get started on Engati with the help of a personalised demo.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
*only for sharing demo link on WhatsApp
Thanks for the information.
We will be shortly getting in touch with you.
Oops! something went wrong!
For any query reach out to us on contact@engati.com
Close Icon
Congratulations! Your demo is recorded.

Select an option on how Engati can help you.

I am looking for a conversational AI engagement solution for the web and other channels.

I would like for a conversational AI engagement solution for WhatsApp as the primary channel

I am an e-commerce store with Shopify. I am looking for a conversational AI engagement solution for my business

I am looking to partner with Engati to build conversational AI solutions for other businesses

continue
Finish
Close Icon
You're a step away from building your Al chatbot

How many customers do you expect to engage in a month?

Less Than 2000

2000-5000

More than 5000

Finish
Close Icon
Thanks for the information.

We will be shortly getting in touch with you.

Close Icon

Contact Us

Please fill in your details and we will contact you shortly.

This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
This is some text inside of a div block.
Thanks for the information.
We will be shortly getting in touch with you.
Oops! Looks like there is a problem.
Never mind, drop us a mail at contact@engati.com