Update Text Representation (Embeddings).ipynb

Author: bansalkanav

Module 9 - GenAI (LLMs and Prompt Engineering)/1. Text Embeddings/Text Representation (Embeddings).ipynb

@@ -422,13 +422,16 @@
  "id": "1cabe4ef-4f68-4194-99b3-784d56e5cc09",
  "metadata": {},
  "source": [
- "At this point, we can notice that the classifier is doing poorly with identifying relevant articles, while it is doing well with non-relevant ones. Our large feature vector could be creating a lot of noise in the form of very rarely occurring features that are not useful for learning. Let us change the count vectorizer to take a certain number of features as maximum.\n",
+ "At this point, we can notice that the classifier is doing poorly with identifying relevant articles, while it is doing well with non-relevant ones. \n",
  "\n",
  "**Potential Reasons for poor classifier performance**\n",
  "1. Perhaps we need to balance the data - Clearly there is class imbalance\n",
  "2. Perhaps we need a better learning algorithm - Implement Logistic Regression, SVM, RF, etc...\n",
  "3. Perhaps we should look for tuning the classifier's parameter with the help of Hyperparameter Tuning\n",
- "4. Perhaps we need a better pre-processing and feature extraction mechanism - Right now we have a sparse and large feature vector"
+ "4. Perhaps we need a better pre-processing and feature extraction mechanism - Right now we have a sparse and large feature vector\n",
+ "\n",
+ "**Let's work with potential reason number 4.** \n",
+ "Our large feature vector could be creating a lot of noise in the form of very rarely occurring features that are not useful for learning. Let us change the count vectorizer to take a certain number of features as maximum."
  ]
  },
  {