In all the previous posts, we have learned about NLP Models, Classifiers, and evaluating the NLP classifiers. In this post, we learn a new topic and it is Feature Selection in NLP Text processing.
Introduction to Feature Selection in Text Processing
Choosing and representing features in the right way from the input documents is a very important part of the classification. The best way to represent features is to convert them into numerics i.e. convert text into numbers.
Stepwise, Perturb, Random Forest, and Dropout are some of the techniques for choosing features.
Representing Words As Numbers And Also As Features
If we consider a large document and consider each word in it. Make each word a number and the number of occurrences of each word as the number. Then we can say that we represented words as numbers.
Alternatively, if we consider all words in the document, put them in an Excel sheet with each word as the column name. Then take all sentences in the document and put them as rows. Now in each row, split each sentence into words and put 1 if the word in the sentence matches with the word that is used as a column name. Put 0 if the word in the sentence does not match the word that is used as a column name.
Above two approaches can be considered as representing the words as numbers.
In either of the above approach, you can even consider n-gram as one unit and do counts (1st approach) or 0/1 (second approach).
You can also classify each sentence based on a subject like Sports, Entertainment, Politics, Books, Humanity, etc. Then count the occurrence of each of those subjects with each subject name as the column names in the Excel sheet and count as a numeric representation of the sentence or text.
Many Features – Challenge 1 – Computationally Expensive:
Above we discussed two approaches and used Excel to populate the features (i.e. words) and their counts (i.e. numerical representation of words). A document generally contains many words and if we put each word in the Excel column name, then Excel becomes very huge. This also causes performing computations like sum, mean, and average across the columns in a single Excel sheet. This is what mean by Computationally Expensive.
Many Features – Challenge 2 – Time-Consuming
Having many features in a single row and processing the counts, and statistical functions across all the rows with each row having so many columns is time-consuming.
Many Features – Challenge 3 – Spurious Correlation
Spurious English word meaning means – “not being what it purports to be”
Spurious Correlation means – “refers to a connection between two variables that appears to be causal but is not“
Solution for the above Challenges – Representing The Features The Right – Title of this Blog Post
We have seen the above challenges having many features (words or n-gram or subjects) as column names in each row of an Excel sheet. Below are some of the solutions for the above challenges:
Step 1: Identify the features that have the better correlation
You can find the above through Linear Regression of all the rows with all the features (i.e. words as column names in each row)
Remove the features that do not have a better correlation. i.e. from the above Linear Regression if most of the features have the same correlation value then remove them.
in addition to the same correlation value, you can sometimes consider features having “almost” the same correlation value and remove them.
Step 2: Identify the features that have Spurious Correlation
For example, the number of occurrences of common words like “the”, “this” etc. With these words as features (i.e. column names in the Excel sheet), remove them as features i.e. remove them as columns in the Excel sheet.
To implement the above solutions there are many techniques in NLP
Techniques to Feature Selection Right Way
Technique 1: Stepwise Feature Selection
This process is computationally expensive because you consider each feature in the Excel sheet.
Technique 2: Perturbation
Ignore some of the features
Technique 2: Random Forest
Take a random subset of features and build an ML model using that subset.
From the above subset ML model, again take another subset and build an ML model.
Get a GOOD prediction with this combination of ML models.
Technique 3: Dropout (main in Neural Networks ML models)
Ignore all the features in a node by assigning zero (0) to all features in that node.
Related Topics:
- Natural Language Processing Primer
- NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
- NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
- NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
- NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
- NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
- NLP Learnings 07 – Introducing NLP Language Models
- NLP Learnings 08 – Language Models – Probability Types
- NLP Learnings 09 – Language Models – Measuring Text Based On Probability
- NLP Learnings 10 – Language Models – Defining Word Boundaries
- NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
- NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
- NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
- NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
- NLP Learnings 15 – Language Models – Classifiers Introduction
- NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
- NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
- NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
- NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
- NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
- NLP Learnings 21 – Language Models – Word Embeddings
- NLP Learnings 22 – Language Models – Sentence and Document Embeddings
- NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider