Bag of words Model

What is bag-of-words model?

The bag-of-words model is used to preprocess the text by converting it into a bag of words or fixed-length vectors, machine learning algorithms. It is the simplest form of text representation in numbers. It is extremely easy, both to understand and to implement, and is used for language modeling and document classification.

It is a way to extract features from text to be used in modeling. 

A bag-of-words includes a vocabulary of known words and a measure of the presence of known words. It describes the occurrence of words in a document.

The model only bothers only about whether known words show up in the document. It does not care where they show up in the document, only that they do show up.

It tries to learn about the meaning of a document from its content alone and assumes that if documents have similar content, they are similar to each other.

We cannot directly feed text into algorithms applied in NLP. They work on numbers. The model converts the text into a bag-of-words. The bag-of-words keeps a count of the occurrences of the most frequently occurring words in that text.

The model counts the number of times each word appears and turns text into fixed-length vectors.

Where is bag-of-words used?

The bag-of-words model is widely used in natural language processing and information retrieval (IR). It is also used rather often in methods of document classification where the frequency of occurrence of every word is used as a feature for training a classifier. The bag-of-words model is also used for computer vision.

The most common practical application of the bag-of-words model is as a tool for feature generation. After you transform the text into a "bag of words", it becomes possible for you to calculate several different measures that can be used to characterize the text. 

The most common kind of characteristic, or feature calculated from the Bag-of-words model is term frequency, which is essentially the number of times a term appears in the text. Term frequency is not necessarily the best representation for the text, but it still does find successful applications in areas like email filtering. Term frequency isn’t the best representation of the text because common words such as "the", "a", "to" are almost always the terms with highest frequency in the text. This shows that having a high raw count does not necessarily indicate that the corresponding word is more important. The most popular way to deal with this issue is to "normalize" the term frequencies by weighting a term by the inverse of document frequency, or tf–idf. In addition to this, for the task of classification, there have been supervised algorithms developed in order to to account for the class label of a document. Even binary (presence/absence or 1/0) weighting is used instead of frequencies for a few problems. For example, this is used in the WEKA machine learning software system).

How does the bag-of-words model work? How do you implement a bag-of-words model?

Here are the steps involved in implementing the bag-of-words model:

The first step is to pre-process the data. The text needs to be converted into lower case, all non-word characters need to be removed, and all punctuations need to be removed.

In the next step, we have to find the most frequent words in the text. The vocabulary must be defined, each sentence must be tokenized to words, and then the number of times the word occurs must be counted.

After that, the model is constructed. A vector is built to determine whether a word is a frequent word. If it is a frequent word, it is set as 1 and if not, it is set as 0.

And now you get your output.

‍

What is the biggest advantage of the bag-of-words model?

The most significant advantage of the bag-of-words model is its simplicity and ease of use. It can be used to create an initial draft model before proceeding to more sophisticated word embeddings.

‍

‍

What are the limitations and disadvantages of the bag-of-words model?

The bag-of-words model is rather easy to understand and implement, but it does have a few limitations and drawbacks. 

The vocabulary/dictionary needs to be designed very carefully. Considering its size has an impact on the sparsity of the document representations and must be managed well.

The bag of words model ignores context by discarding the meaning of the words and focusing on the frequency of occurrence. This can be a major problem because the arrangement of the words in a sentence can completely change the meaning of the sentence and the model cannot account for this.

Another major drawback of this model is that it is rather difficult to model sparse representations. This is due to informational reasons as well as computational reasons. The model finds it difficult to harness a small amount of information in a vast representational space.

‍

Bag-of-word representation

‍

What is the difference between bag-of-words and TF-IDF?

Term frequency-inverse document frequency (TF-IDF) is a numerical statistic the purpose of which is to reflect how important a word is to a document in a collection or corpus.” 

Term Frequency (TF) is a measure of how frequently a term, t, appears in a document. Inverse Document Frequency (IDF) is a measure of how important a term is. The IDF value is important because simply computing just the TF is not sufficient enough to understand the importance of words.

The main differences between bag-of-words and TF-IDF are that:

Bag of Words only creates a set of vectors that contains the count of word occurrences in the document (reviews). The TF-IDF model, on the other hand, holds information on the more important words and the less important ones as well.

It is rather easy to interpret bag-of-words vectors. But TF-IDF generally tends to perform better in machine learning models.