NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation

AI & ML & Blockchain, Blog, Data & Analytics, Machine Learning

In the previous post, we discussed Word Boundaries. What do you have to consider while using NLP Language models for your business text data? In this post, we discuss “Words Representation” for your “Business Text Data”.

When building an NLP language model for your business data, it is important to select a method to represent the words. Several methods exist, including word embeddings, symbols, and integers.

Representing Words In Your Business Text Data

You can represent the words in your business text data in the following methods:

  1. Frequency of Words – take each word OR N-gram of words and find out how frequent each word is
  2. Word Boundaries – in the previous post we discussed defining word boundaries. Apply those concepts to your business text data. Word boundary concepts are should you consider punctuation, unknown words from your business text perspective, misspelled words from your business text perspective, and more
  3. One-Hot Representation of words – more on this understanding in upcoming blog posts
  4. Dense Arrays – more on this understanding in upcoming blog posts

In addition to the above, there are many pre-trained language models that use Wiki, Stackoverflow, popular books, and more. Evaluate the pre-trained models based on their training dataset and consider those pre-trained models that are best suited for your business text data.

What do you have to d now?

From the above section, having a basic understanding of how to represent words in your business text data, you can do this from your business context perspective:

 

Related Topics:

  1. Natural Language Processing Primer
  2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
  3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
  4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
  5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
  6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
  7. NLP Learnings 07 – Introducing NLP Language Models
  8. NLP Learnings 08 – Language Models – Probability Types
  9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
  10. NLP Learnings 10 – Language Models – Defining Word Boundaries
  11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
  12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
  13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
  14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
  15. NLP Learnings 15 – Language Models – Classifiers Introduction
  16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
  17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
  18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
  19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
  20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
  21. NLP Learnings 21 – Language Models – Word Embeddings
  22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
  23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider
Post Views: 507