In the previous post(s), we discussed the Frequency of Words, Word Boundaries, One-Hot Representation of words, and Dense Arrays as some of the approaches to how you can represent your business text data.

In this post, we discuss while building your own NLP Language Model(s), what Regularization techniques you should apply to your business text data.

For readers of this post, who like quick answers, Regularization is a machine learning (ML) technique that avoids overfitting of an ML model and reduces prediction errors, especially for unseen words in the context of the NLP Language Model.

Introduction to Regularization in Machine Learning

Firstly, Regularization is a Machine Learning technique.

Instead of mentioning the definition of Regulairziation too early, let me build a story and let the story be the intuition for you to understand the meaning of Regularization.

Remember the purpose of Language Models

Language models are important as they help to predict the next word in a sentence or the most likely or accurate sequence for a sentence.

Story START

Consider all the previous blog posts on NLP on this website. There is no mention of the word COVID in any of the previous blog posts. The probability of occurrence of the word COVID will be zero.

i.e. P (COVID word occurrence in previous blog posts) = 0

The reason why the above probability is zero is that the COVID word is not mentioned in any of the previous blog posts on this website. That means COVID is an unseen word.

Does this mean that, if you develop a Language Model using all the previous posts on this website as your input data or training data when an unseen word is provided to your Language Model then your Language Model cannot predict the next word? (re-read again the purpose of the Language Model at the top of this post).

Main Motto of Your Language Models Should Be:

Your Language Model should have the ability to Predict even when UnSeen Word is provided as input

We cannot say that your Language Model cannot predict the unseen word. If your Language Model has a weakness of not being able to predict the unseen word then you made your Language Model understand more on the training data or raw data your model was developed on. Basically, you Overfitted Your Language Model, which is not good.

With the above “Overfitting” story I believe you understand the meaning of Regularization in Machine Learning

Story END 

Definition of Regularization in Machine Learning

Regularization is a technique in machine learning where you reduce prediction errors of your machine learning model by fitting the machine learning model appropriately on the given training set and avoiding overfitting.

At a High Level, How Regularization Techniques Work in Machine Learning Models:

Firstly we all know a Machine Learning Model is nothing but a function or an equation. As an example, let us take the below function or equation:

y = f (x) = w0 + w1 * x1 + w2 * x2 ….. wn * xn

where  w1, w2, w3, … wn are weights or coefficients  AND x1, x2, x3… xn are features or observations

Regularization Techniques penalize the coefficients. For readers with deep learning, it means penalizing the weight matrices.

Listed below are various Regularization Techniques. That means how each technique penalizes the coefficients.

Regularization Techniques

L1 regularization	Also called LASSO(Least Absolute Shrinkage and Selection Operator) regression.  Penalizes by adding “sum of absolute values of the coefficients” to the Loss Function
L2 regularization	Also called Ridge regression Penalizes by adding “sum of squares of the coefficients” to the Loss Function
Dropout regularization	Mostly used in Deep Learning. During each iteration of Deep Learning, it drops the nodes OR incoming/outgoing connections
Early stopping	During each iteration of your model training, if the performance of your model is worsening i.e. loss function performance, then immediately stop training the model

Regularization in the context of NLP Language Models

Smoothing is one of the regularization techniques of NLP Language Models. We discuss more on smoothing in the next blog post on this website.

 

Related Topics:

1. Natural Language Processing Primer
2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
7. NLP Learnings 07 – Introducing NLP Language Models
8. NLP Learnings 08 – Language Models – Probability Types
9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
10. NLP Learnings 10 – Language Models – Defining Word Boundaries
11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
15. NLP Learnings 15 – Language Models – Classifiers Introduction
16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
21. NLP Learnings 21 – Language Models – Word Embeddings
22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider