In the previous post, we learned what are probabilistic classifiers definition. In this post, we introduce Simple Classifiers like Naive Bayes and in the next post we introduce Linear classifiers. These two are probabilistic classifiers.
Introducing Naive Bayes Probabilistic Classifier
First, we should know what is Bayes Theorem before learning the Naive Bayes Probabilistic Classifier
What is Bayes’s Theorem?
Before we know Bayes Theorem in the context of NLP, please note that NLP Language Models, especially classifiers, are used to predict the next word or predict the probability of a word OR phrase coming before and/or current word/current phrase.
Bayes Theorem is the probability of an event, based on a prior dataset OR prior knowledge of conditions that might be related to the event. Here event is predicting the text in the NLP context.
Bayes’ Theorem is useful when working with conditional probabilities (read the below example).
What are Naive Bayes Probabilistic Classifiers?
Naive Bayes classifiers mean applying the Bayes Theorem.
In the previous posts, we mentioned Sports, Politics, Entertainment, Science, Technology, Humanity, etc. are possible categories of a sentence or a phrase or set of words. Classifying to which category a sentence or phrase or set of words belongs is the job of a classifier.
If we take a sentence and split it into say one word each or two words together as one unit each, then each split is called a feature ( i.e. one word OR one unit (combined words) ).
The first thing we need to do when creating a machine learning model is to decide what to use as features. In the case of NLP where it deals with text (words, sentences, phrases), we use word frequencies as features
So all naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature.
Understanding Probability in Naive Bayes Classifiers:
Assume you have a large number of word documents. You can consider these documents as your training data.
Then probability is just counting the training data. That means, for a given sentence in the above training data, the probability of each word “W” i.e P (W) is calculated. That’s easy enough.
Then, calculating P ( “Another Word AW ” | Given W) means counting how many times another word “AW” appears in a sentence that word “W“ divided by the total number of words with word W
In the below equation, A = “Another Word AW” and B = “Given word W”
Understanding Probability with an Example:
| Text | Classified As |
|---|---|
| A great cinema | Entertainment |
| The cricket match was over | Sports |
| Very clean national party election manifesto | Politics |
| A forgettable rocket launch | Technology |
Now, which classification does the sentence “A very close story” have?
Since Naive Bayes Classifier is a probabilistic classifier, we calculate the probability that the sentence “A very close story” is Entertainment and also calculated the probability that it is Not Entertainment.
From the above two probabilities, we take whichever value is higher.
Written mathematically, what we want is P (Entertainment | “A very close story“) — the probability that the category of a sentence is Entertainment given that the sentence is “A very close story“.
How do we calculate the above probabilities?
To calculate probabilities for the given set of word documents, as mentioned above, we calculate word frequencies in the above set of word documents.
As mentioned above, Bayes’ Theorem is useful when working with conditional probabilities (read the below example). In the current example, the condition is given as “A very close story” what is the probability that it is related to the Entertainment category?
P ( Entertainment | “A very close story“) =
P (“A very close story” | Entertainment) X P ( Entertainment) / P (“A very close story“)
There could be a problem though: “A very close story” may OR may not appear in our training data, so this probability could be zero. Unless every sentence that we want to classify appears in our training data, the above equation or model won’t be very useful.
So here comes the Naive part: we assume that every word in the sentence “A very close story” is independent of the other ones. This means that we’re no longer looking at the entire sentence, but rather at individual words in the sentence “A very close story”.
So we write the sentence “A very close story” as:
P (“A very close story“) = P(a) * P(very) * P(close) * P(story)
So
P (“A very close story” | Entertainment) =
P(a | Entertainment ) * P(very | Entertainment ) * P(close | Entertainment) * P(story | Entertainment)
Since all of the above words ( a, very, close, story) generally show up several times in word documents i.e. training data, we can calculate each word’s probability.
Calculating the above probabilities – Advanced Techniques for Unseen Words
In the above probability P (“A very close story“) = P(a) * P(very) * P(close) * P(story)
what if we get a word or words or phrase or sentence that does not existied in your training data. Then probability of that will be zero.
Say the phrase is “A very good city in Domestika“, but your input documents used for training does not have the word “Domestika”. Then P(“Domestika”) will be zero. Since we are multiplying each probability in the above i.e.
P(“A very good city in Domestika“) = P(a) * P(very) * P(good) * P(city) * P(in) * P(Domestika).
Since P(Domestika) is zero the whole P(“A very good city in Domestika”) = 0 and this will be a problem.
To overcomes this problem following are the advanced techniques that can be applied:
Technique 1: Laplace smoothing where we add 1 to every count so it’s never zero.
Technique 2: Removing stopwords
Technique 3: Lemmatizing words
Technique 4: Using n-grams
Technique 5: Using TF-IDF