In the series of NLP Leanings, in all the previous posts just before this post, we learned Linguistics and Linguistic Categories. In this post, we start with NLP Language models.

Language Models became very fundamental in Natural Language Processing (NLP).

Introducing Language Models

Language models are the way to measure and score a text.

(look below for how to measure sentences)

The text could be a simple sentence or phrase in a sentence or a paragraph or multiple paragraphs and beyond it could be, large text (also known as corpus) as big as a book.

Language models are important as they help to predict the next word in a sentence or the most likely or accurate sequence for a sentence.

What is the use of Score output by Language Models?

The score generated by Language Models helps us to decide how good a text or sub-text or words or corpus is. 

What are the real applications of Language Models?

Language Models can be applied to a number of natural language processing (NLP) requirements, including the correction of spelling mistakes, speech recognition, and machine translation.

Using the scoring output by Language Models, we can develop the following applications:

1. Rank two or more sentences – and pick the best based on the rank – may be in
   • search box OR
   • while typing software recommending better sentences to use OR
   • select the best grammatical sentence with a similar meaning while you are typing
2. Create Predictive software systems say for text completion or text 
3. Language Translations,  Speech Recognition, Language Generation

How do we measure two or more Sentences?

Below are some of the many techniques one can use to measure two or more sentences.

1. Frequency of words i.e. number of occurrences of a word or a phrase in a sentence or corpus

2. Stylistics (does it have casual words and formal words)

3. Probability of words occurring in different contexts or scenarios

Probability = Number of occurrences / Total Number of words in the corpus

In NLP we generally write the Probability as

p(X = x) is the probability that the random variable X having the value x

example: p(X = ‘equation’) means X having the value ‘equation’

 

Related Topics:

1. Natural Language Processing Primer
2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
7. NLP Learnings 07 – Introducing NLP Language Models
8. NLP Learnings 08 – Language Models – Probability Types
9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
10. NLP Learnings 10 – Language Models – Defining Word Boundaries
11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
15. NLP Learnings 15 – Language Models – Classifiers Introduction
16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
21. NLP Learnings 21 – Language Models – Word Embeddings
22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider