Update working_with_text.ipynb

Author: bansalkanav

Module 4 - Machine Learning/02. Working with Text Data/1. Basics of NLP and Text Preprocessing/working_with_text.ipynb

@@ -879,7 +879,9 @@
  "\n",
  "### Disadvantages\n",
  "1. The size of vector increases with the size of the vocabulary. Thus, sparsity continues to be a problem. One way to control it is by limiting the vocabulary to n number of the most frequent words.\n",
- "2. It does not capture the similarity between different words that mean the same thing. Say we have three documents: “walk”, “walked”, and “walking”. BoW vectors of all three documents will be equally apart.\n",
+ "2. It does not capture the similarity between different words that mean the same thing. i.e. Semantic Meaning is not captured.\n",
+ "> a. \"walk\", \"walked\", and \"walking\". BoW vectors of all three tokens will be equally apart. \n",
+ "> b. \"search\" and \"explore\" are synonyms. BoW won't capture the semantic similarity of these words.\n",
  "3. This representation does not have any way to handle out of vocabulary (OOV) words (i.e., new words that were not seen in the corpus that was used to build the vectorizer).\n",
  "4. As the name indicates, it is a “bag” of words. Word order information is lost in this representation. One way to control it is by using n-grams.\n",
  "5. It suffers from **curse of high dimensionality.**"
@@ -1124,38 +1126,6 @@
  "print(dtm.toarray())"
  ]
  },
- {
- "cell_type": "code",
- "execution_count": 29,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'best': 1, 'time': 3, 'worst': 5, 'age': 0, 'wisdom': 4, 'foolishness': 2}\n",
- "dict_keys(['best', 'time', 'worst', 'age', 'wisdom', 'foolishness'])\n",
- "[('age', 0), ('best', 1), ('foolishness', 2), ('time', 3), ('wisdom', 4), ('worst', 5)]\n",
- "['age', 'best', 'foolishness', 'time', 'wisdom', 'worst']\n"
- ]
- }
- ],
- "source": [
- "vocabulary = vocab.vocabulary_\n",
- "\n",
- "print(vocabulary)\n",
- "\n",
- "print(vocabulary.keys())\n",
- "\n",
- "sort_vocab_tup = sorted(vocabulary.items(), key = lambda x : x[1])\n",
- "\n",
- "print(sort_vocab_tup)\n",
- "\n",
- "sort_vocab = [tup[0] for tup in sort_vocab_tup]\n",
- "\n",
- "print(sort_vocab)"
- ]
- },
  {
  "cell_type": "code",
  "execution_count": 30,
@@ -1178,7 +1148,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 31,
+ "execution_count": 32,
  "metadata": {},
  "outputs": [
  {
@@ -1259,7 +1229,7 @@
  "3 1 0 1 0 0 0"
  ]
  },
- "execution_count": 31,
+ "execution_count": 32,
  "metadata": {},
  "output_type": "execute_result"
  }
@@ -1270,7 +1240,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 32,
+ "execution_count": 33,
  "metadata": {},
  "outputs": [],
  "source": [
@@ -1283,7 +1253,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 33,
+ "execution_count": 34,
  "metadata": {},
  "outputs": [
  {
@@ -1300,7 +1270,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 34,
+ "execution_count": 35,
  "metadata": {
  "scrolled": true
  },
@@ -1323,7 +1293,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 35,
+ "execution_count": 36,
  "metadata": {
  "scrolled": true
  },
@@ -1432,7 +1402,7 @@
  "3 0 0 "
  ]
  },
- "execution_count": 35,
+ "execution_count": 36,
  "metadata": {},
  "output_type": "execute_result"
  }
@@ -1457,12 +1427,33 @@
  "source": [
  "## Term Frequency Inverse Document Frequency\n",
  "\n",
- "In BOW approach all the words in the text are treated as equally important i.e. there's no notion of some words in the document being more important than others. TF-IDF, or term frequency-inverse document frequency, addresses this issue. It aims to quantify the importance of a given word relative to other words in the document and in the corpus."
+ "In BOW approach all the words in the text are treated as equally important i.e. there's no notion of some words in the document being more important than others. TF-IDF, or term frequency-inverse document frequency, addresses this issue. It aims to quantify the importance of a given word relative to other words in the document and in the corpus.\n",
+ "\n",
+ "***\n",
+ "\n",
+ "Let's now try to understand:\n",
+ "1. Term Frequency \n",
+ "2. Inverse Document Frequency\n",
+ "\n",
+ "$$ TF \\ IDF = TF(word_i, doc_j) * IDF(word_i, corpus) $$\n",
+ "\n",
+ "$$ TF(word_i, doc_j) = \\frac{No \\ of \\ time \\ word_i \\ occurs \\ in \\ doc_j}{Total \\ no \\ of \\ words \\ in \\ doc_j} $$\n",
+ "\n",
+ "$$ IDF(word_i, corpus) = \\log_n(\\frac{No \\ of \\ docs \\ in \\ corpus}{No \\ of \\ docs \\ which \\ contains \\ word_i}) $$\n",
+ "\n",
+ "***\n",
+ "\n",
+ "### Advantages\n",
+ "1. If the word is rare in the corpus, it will be given more importance. (i.e. IDF)\n",
+ "2. If the word is more frequent in a document, it will be given more importance. (i.e. TF)\n",
+ "\n",
+ "### Disadvantages\n",
+ "> **Same as BOW**"
  ]
  },
  {
  "cell_type": "code",
- "execution_count": 36,
+ "execution_count": 37,
  "metadata": {},
  "outputs": [],
  "source": [
@@ -1477,7 +1468,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 37,
+ "execution_count": 38,
  "metadata": {},
  "outputs": [
  {
@@ -1494,7 +1485,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 38,
+ "execution_count": 39,
  "metadata": {
  "scrolled": true
  },
@@ -1518,7 +1509,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 39,
+ "execution_count": 40,
  "metadata": {},
  "outputs": [
  {
@@ -1599,7 +1590,7 @@
  "3 0.61913 0.000000 0.785288 0.00000 0.000000 0.000000"
  ]
  },
- "execution_count": 39,
+ "execution_count": 40,
  "metadata": {},
  "output_type": "execute_result"
  }
@@ -1612,6 +1603,16 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
+ "## Word Embeddings\n",
+ "\n",
+ "In natural language processing (NLP), [word embedding](https://en.wikipedia.org/wiki/Word_embedding) is a term used for the representation of words for text analysis, typically in the form of a real-valued vector that encodes the meaning of the word such that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using a set of [language modeling](https://en.wikipedia.org/wiki/Language_model) and [feature learning](https://en.wikipedia.org/wiki/Feature_learning) techniques where words or phrases from the vocabulary are mapped to vectors of real numbers.\n",
+ "\n",
+ "Methods to generate this mapping include **neural networks**, **dimensionality reduction on the word co-occurrence matrix**, **probabilistic models**, **explainable knowledge base method**, and **explicit representation in terms of the context in which words appear**.\n",
+ "\n",
+ "Traditionally, one of the main **limitations of word embeddings** (word vector space models in general) is that words with multiple meanings are conflated into a single representation (a single vector in the semantic space). In other words, [polysemy](https://en.wikipedia.org/wiki/Polysemy) and [homonymy](https://en.wikipedia.org/wiki/Homonym) are not handled properly. \n",
+ "\n",
+ "\n",
+ "\n",
  "## Word2Vec\n",
  "\n",
  "\"You shall know the word by the company it keeps.\" by JR Firth\n",