In this post, we learn Word Embedding briefly.

Word Embeddings Introduction

In simple understanding, Word Embeddings are vector representations of a word.

In physics, Scalar means a value or physical quantity.

In Latin, Vecotr means carrier.

In mathematics and physics, a vector refers to some quantities or numbers or magnitude that should be expressed by a single number (a scalar) and direction (carrier from Point A to Point B).

Introduction on Embeddings

Embeddings have been one of the most influential techniques in NLP. Encoding information i.e. converting words into numbers, i.e. into a low-dimensional vector representation, that can be easily integrable in pre-trained or traditional machine learning models or tools, has played a central role in the advancement of NLP.

How do we generate Word Embeddings?

Consider the following similar sentences:

Sentence 1: This is an amazing post

Sentence 2: This is a great post.

The above two sentences hardly have different meanings. Now, if we represent words into numbers by putting each word in each sentence in one line as shown below:

Sentence 1:  This, is, an, amazing, post

Sentence 2:  This, is, a, great, post

So the above two sentences can be represented as (SR1, SR2 where SR1 and SR2 are sentence  row 1, sentence row 2 respectively)

SR1 = {This, is, an, amazing, post}

SR2 = {This, is, a, great, post}

Now if we place both SR1 and SR2 into a variable say V (V stands for vocabulary)

Vocabulary V = Vector V = V = list of SR1, SR2  i.e. V = [ S1, S2 ] where element within this brackets [] should be treated as rows

So above V have two rows

Let us now generate Word embeddings with the above V

One-Hot encoding of rows in Vocabulary V

This = [1,0,0,0,0]`; is=[0,1,0,0,0]` ; an=[0,0,1,0,0]` ; amazing=[0,0,0,1,0]` ; post=[0,0,0,0,1]` 

This = [1,0,0,0,0]`; is=[0,1,0,0,0]` ; a=[0,0,1,0,0]` ; great=[0,0,0,1,0]` ; post=[0,0,0,0,1]` 

The above is called Word embeddings or word encodings.

Let us try to visualize these encodings, we can think of an n-dimensional space where n = no of words in Vocabulary V.

What is the Importance of Word Embeddings?

Word Embeddings importantly capture the context of a word in a sentence or in a corpus of text.

By placing the words Vector V ( as said above) in an X1, X2, X3 – Y1, Y2, Y3 (i.e. X-Y multi-dimensional axis) Word Embeddings solutions one of the objectives is to have words with similar context. This is determined from the X-Y axis diagram as words with similar contexts occupy close spatial positions.

Word Embeddings Pre-Trained Libraries

Word2Vec and Glove are two of the popular Pre-Trained Word Embeddings Libraries

The above kinds of embeddings are trained on large datasets, saved, and made available to the public community to be used for solving other tasks. That’s why pre-trained word embeddings are a form of Transfer Learning.

Pretrained word embeddings capture the semantic and syntactic meaning of a word as they are trained on large datasets. They are capable of boosting the performance of an NLP model

You developing a word embedding solution through NLP machine learning training, learning from scratch is a challenging problem due to reasons like the sparsity of training data and a Large number of trainable parameters

Related Topics:

1. Natural Language Processing Primer
2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
7. NLP Learnings 07 – Introducing NLP Language Models
8. NLP Learnings 08 – Language Models – Probability Types
9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
10. NLP Learnings 10 – Language Models – Defining Word Boundaries
11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
15. NLP Learnings 15 – Language Models – Classifiers Introduction
16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
21. NLP Learnings 21 – Language Models – Word Embeddings
22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider