In the previous posts, we learned various classifiers. In this post, we discuss how to evaluate those classifiers.

Why do we Evaluate Classifiers?

After you develop a machine learning model based on training data, you should be able to explain about the model. Evaluation Metrics are one of the ways to explain the model.

Specific to NLP text classifiers, using the metrics you should be able to inform users of your model how well it classifies the new text.

List of Common Metrics

Following are the common metrics you can use to evaluate your model.

Before learning the below list you need to consider these in your thoughts:

1. You created a machine-learning model based on training data
2. You have an ML model equation i.e. above created ML model (this is a repeat of above point 1 in a different way)
3. Now using programming languages like Python or R you applied various techniques to build the ML model (like cross-validation etc, feature engineering, etc.)
4. Using the above techniques you obtained metrics either in percentages or in probability values (between 0 to 1 values) or in the list of values (like true OR false OR politics OR entertainment etc.)
5. How to convert that metric into an English sentence is the below list of common metrics

To undersetand below, let us assume we have a basket of 10 furits with Apple and Oranges combined. You developed a ML model to predict Apple and Orange. With this context we can understand below definitions:

Before definition understanding,

“No of Observations or Fruits = 10” ;

“No of Apples in the basket = 6”;

“No of Oranges in the basket = 4” 

Accuracy

Fraction of times the model makes a correct prediction as compared to the total predictions it makes.

So Accuracy = ( Total Correct Orange Predicted + Total Correct Apled Predicted ) / Total No of fruits in the bask

Problem with Accuracy metric:

If we have 9 Orangages and 1 Apple in the basket, then ML model might predict Orangage accurately and might not predict Apple. This is a problem with this Accuracy metric.

Precision

One metric that could solve avove Accuracy metric problem is Precision.

Precision = True Positive /  (True Positive + False Positive)

Precision = No of Correct Positive Results / No of Predicted Positive Results

So Precision = Total Correct Apples Predicted  “on the Apples side” / Total No Of Predicted Apples “Side” of the Observations In The Basket

You can relace Aples with Oranages in the above question. Please observe or stress the world “on the Apple side”. This means we consider total in one side of the class and within that class what is the % of our preodiction

It does not matter Apples or Ornages, basically you consider 1 class (Apple or Oranage) and find out how much your ML model predicted right it is Apple (or Orange) from the list of Apples (or Oranages)

Recall

Recal = No of Correct Positive Results / No of Actual Postivie Results

Recall = TP / TP + FN

So Precision = Total Correct Apples Predicted in the bask / Total No Of Apples  In The Basket

F-Score

First we need to know what is Harmonic mean because F-score is the Harmonica mean of Precision and Recall. Harmonic mean is “the reciprocal of the arithmetic mean of the reciprocals“

F-Score = 2 /  {  ( 1 / Precision ) + ( 1 / Recall ) }

F-Score = 2 *  (Precision * Recall ) / (Precision + Recall)

AUC

MRR

RMSE

METEOR

ROUGE

Perplexity

 

 

Related Topics:

1. Natural Language Processing Primer
2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
7. NLP Learnings 07 – Introducing NLP Language Models
8. NLP Learnings 08 – Language Models – Probability Types
9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
10. NLP Learnings 10 – Language Models – Defining Word Boundaries
11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
15. NLP Learnings 15 – Language Models – Classifiers Introduction
16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
21. NLP Learnings 21 – Language Models – Word Embeddings
22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider