In the previous post, Regularization Techniques in Machine Learning are discussed. In this post, we discuss the Smoothing Regularization technique for NLP Language Models.

Quick Review of Regularization Technique – Regularization technique prevents the overfitting of machine learning models and there are various techniques to achieve this. 

Introducing the Smoothing Regularization Technique

Smoothing is one of the regularization techniques for NLP Language Models to reduce prediction errors.

Let us understand its definition through a story

Story START

Consider all the previous blog posts on this website. There is no mention of the word “football” but the word “Learn” is mentioned multiple times. 

So, Probability of (word = “football”) = 0 and Probability of (word = “Learn”) ~~ 1

Another example where “p” below is the probability

p(Website mentioned football) = p(Website|•) p(mentioned|Website) p(football|mentioned) p(•|football) p(•|word)

Does it mean that when someone provides the “football” word as input text and your NLP Language Model did not see before, does your NLP Language Model not predict next anything?

Another way of saying above is does your NLP Language Model “overfitted” to your training data i.e. have overfitted to the words that have seen in your training data while building the NLP Language Model?

An NLP Learning Model should be designed to predict even when an unseen word or phrase or text is provided.

Just because “a word” has never been observed in the training data of your NLP Language Model “does not mean it cannot occur in test data”.

So Smoothing technique is also about preventing the overfitting of models and supporting the unseen words.

Story END

Definition of Smoothing

Smoothing is a technique that adjusts the probabilities used in our language model so that our model can perform more accurately and even handle the words absent in the training set.

A quick way to understand the Smoothing technique is if P (unseen words) = 0 then make their probability = 1 i.e. Add-One value technique.

Types of Smoothing

Add-One Smoothing	We simply add 1 to all the counts of words so that we never incur a 0 value. i.e. we add 1 to all counts of words. Also known as Laplace smoothing
Additive smoothing	Similar to Laplace smoothing. Instead of 1, we add a δ value to all the counts of words so that we never incur a 0 value
Absolute discounting
Kneser-Ney smoothing	You discount an absolute discounting value, d from observed N-grams and distribute it to unseen N-grams.
Good-Turing estimate	Instead of adding 1 value to all word count, we estimate a better number that we can add to the words count. Use N-Gram (i.e. more than one word together), to estimate how much to add for zero occurring N-Grams. Determining the estimate is computationally expensive Improves the perplexity of your model
Jelinek-Mercer smoothing (interpolation)	Try a mixture of (multiple) n-gram models
Katz smoothing (backoff)	Start with n-gram, -> If insufficient observations, check (n-1)gram -> If insufficient observations, check (n-2)gram Is computationally more efficient Provides better perplexity and usage
Witten-Bell smoothing

 

Related Topics:

1. Natural Language Processing Primer
2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
7. NLP Learnings 07 – Introducing NLP Language Models
8. NLP Learnings 08 – Language Models – Probability Types
9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
10. NLP Learnings 10 – Language Models – Defining Word Boundaries
11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
15. NLP Learnings 15 – Language Models – Classifiers Introduction
16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
21. NLP Learnings 21 – Language Models – Word Embeddings
22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider