NLP Learnings 10 – Language Models – Defining Word Boundaries

AI & ML & Blockchain, Blog, Machine Learning

In the previous post, we discussed the Probability of Words in NLP Language Models. Also discussed Uni-Gram and N-Gram models. Since all NLP Language models fundamentally start with a word (Unigram) and build on considering subsequent words (before the word i.e. back propagation OR after the word i.e. forward propagation), it is important for NLP engineers to define Word boundaries in NLP Language Models.

Defining Word Boundaries

  1. Punctuation
    • Should we consider punctuation in the word 
    • Should we separate punctuation from the word
    • We can predict the probability of the next word OR the previous word by considering or not-considering the punctuation of a word in a sentence
  2. Filler Words (“uh”, “ah”, “hmm” etc. are filler words) – just like above should we consider OR not-consider fillers
  3. Possessives of Words (circles vs circle) – should we consider or not consider possessive words
  4. Multi-Token words (South Africa, Port of Spain, etc) where more than one word makes one meaning word)
  5. Unknown Words (“MeToo”, and “BREXIT” are examples of unknown words)
  6. Misspelled Words – should we consider or not consider them in NLP Language Models
  7. Signal Words (STOP, TURN are examples) – should we consider or not consider them in NLP Language Models

Mathematics Or Statistical Analysis with Word Boundaries

Once we define the word boundaries, then we can apply mathematical or statistical analysis to words like frequency of word boundaries, distribution of such word boundaries, and more.

 

Related Topics:

  1. Natural Language Processing Primer
  2. NLP Learnings 02 – After Primer – NLP Models & Evaluating NLP Models
  3. NLP Learnings 03 – What is Linguistic in NLP and Linguistic Categories or Linguistic Levels
  4. NLP Learnings 04 – Sentence Linguistic Analysis – Words As First Step
  5. NLP Learnings 05 – Sentence Linguistic Analysis – Pragmatics Analysis
  6. NLP Learnings 06 – What can we do with A Sentence in NLP Tasks
  7. NLP Learnings 07 – Introducing NLP Language Models
  8. NLP Learnings 08 – Language Models – Probability Types
  9. NLP Learnings 09 – Language Models – Measuring Text Based On Probability
  10. NLP Learnings 10 – Language Models – Defining Word Boundaries
  11. NLP Learnings 11 – Language Models – Your Business Text Data and Their Words Representation
  12. NLP Learnings 12 – Language Models – Regularization Techniques And Your Business Text Data
  13. NLP Learnings 13 – Language Models – Smoothing Regularization Techniques And Your Business Text Data
  14. NLP Learnings 14 – Language Models – NLP Key Terms and Concepts
  15. NLP Learnings 15 – Language Models – Classifiers Introduction
  16. NLP Learnings 16 – Language Models – Classifiers – What are Probabilistic Classifiers
  17. NLP Learnings 17 – Language Models – Classifiers – Simple Classifiers Introduction
  18. NLP Learnings 18 – Language Models – Classifiers – Simple Classifiers – Linear Probabilistic Classifier Introduction
  19. NLP Learnings 19 – Language Models – Classifiers – Evaluating Classifiers – Precision Recall F-Score Confusion Matrix
  20. NLP Learnings 20 – Language Models – Classifiers – Represent Words In A Document – Choosing and Representing Features In The Right Way
  21. NLP Learnings 21 – Language Models – Word Embeddings
  22. NLP Learnings 22 – Language Models – Sentence and Document Embeddings
  23. NLP Learnings 23 – Language Models – Sentence and Document Embeddings – Which one to consider
Post Views: 567