SD
Uploaded bySagar Deogirkar
PPTX, PDF227 views

Comparative Study of Machine Learning Algorithms for Sentiment Analysis with TF-IDF Vector Creation

This document presents a comparative study of machine learning algorithms for sentiment analysis using tf-idf vector creation, focusing on the selection of the best-performing models. The research uses a 'twitter airline' dataset comprising 14 features and emphasizes the evaluation metrics like accuracy and classification error to compare models such as Support Vector Machine (SVM) and Gradient Boosting Trees. It concludes that SVM outperforms traditional models, while future work will explore advanced techniques like Recurrent Neural Networks for further improvements.

Download to read offline
“ Comparative Study of Machine Learning Algorithms for Sentiment Analysis with TF-IDF Vector Creation Sagar Vijay Deogirkar (10547321) MSc Data Analytics Ms. Terri Hoare Supervisor
Index • Introduction • Research Question and Objective • Methodology • Business and Data Understanding • Data Preparation • Modelling • Evaluation • Results • Conclusion and Future Work 2
Introduction Customers are expressing their thoughts about product and offered services more openly than never before. Considering this sentiment analysis is becoming an essential aspect to understand their sentiments. Sentiment analysis cites the use of Natural Language Processing technique to classify the type of sentiment. In other words, Sentiment Analysis is the process to determine if the given text is positive, negative or of neutral sentiment. It is often perform on textual data to help business entities, to monitor their brand’s product or services’ sentiments from client’s reviews. This helps to understand the customer’s requirement which may lead to necessary improvement in product or services if required. 3
Research Question Sentiment Analysis is trending, many programming and non programming platforms have arrived providing solution to this problem. But problem lies with platform selection and more over that, to the model or algorithm selection. The main obstacle is to know, which algorithm can be chosen with TF-IDF vector creation technique, which will lead to determine the class of the sentiment more correctly. 4 Objective The main objective of this research is to compare the State-of-the-Art Deep Learning with Machine Learning algorithms performance on TF-IDF vector creation for Sentiment Analysis.
Methodology Data Understanding 5 Data Preparation Modelling Evaluation Business Understanding Deployment • Determining Business Objective • Assess The Situation • Determine the Study Goal • Produce a Project Plan • Data Selection • Clean Data • Construct Data • Integrate Data • Format Data • Evaluate Result • Review Process • Determine Next Step • Data Collection • Describing the Data • Data Exploration • Verifying Data Quality • Select the Model • Generate Test Design • Build the Model • Assess the Model • Plan Development • Plan Monitory and Maintenance • Produce Final Report • Review Report This is research conducted following CRISP-DM (Cross-Industry Standard Process for Data Mining) methodology.
Business Understanding Sentiment classification is a term which comprises the method to determine the labelled sentiment from the available classes based on the aligned text data. It helps to identify the emotions i.e. sentiment behind the text of high volume data. Text data could be reviews from YouTube, or from any social media platform, or tweets on trending topic involving different hash tags and etc, or articles or news report or such which is in the form of text. 6 Data Understanding • For this research “Twitter Airline” dataset is used. • Dataset comprises of total 14 features and a label, having a total of 14640 rows. • Column Names - tweet id, airline sentiment, airline sentiment confidence, airline sentiment gold, negative reason, negative reason confidence, airline, name, negative reason gold, re tweet count, text, tweet coord, tweet created, tweet location, user time zone. •Sentiment Distribution: 9178 – Negative, 3099 – Neutral and 2363 - Positive. • From the above features only airline_sentiment and text is selected for the research.
Data Preparation • Text Pre-processing – Lowercasing is done. Unnecessary symbols and numbers are removed. • Sentiment Class Filtering - Neutral class’s sentiment is filtered and positive is turned to 1 and Negative to 0. • Data Balancing – Positive and Negative class sentiment is balanced to same number of samples. • Removing Stop Words – Common words in the language are removed. • Text Stemming – Porter Stemming is used to make word into its original form. • Tokenization – Every word is separated in the document. • TF-IDF vector – TF- IDF word vector is created having all the words in the data set with their weight. 7 Text Cleaning Text Processing Vector Creation Data Importation TF- IDF Vector Creation • Text Stemming • Tokenization • Data Balancing • Lowercasing • Removing Symbols and Numbers • Removing Stop Words. Data Importation to the Platform and Considering the Features and label.
Modelling • Selected Models are: Naive Bayes, Support Vector Machine (SVM), Generalised Linear Model (GLM), Logistic Regression, Decision Tree, Random Forest, Gradient Boosted Trees, and Deep Learning. • On Rapid Miner Auto Model is used with 3000 samples. •Deep learning (Neural Network) is observed on H2O AI platform with 3000 samples processed and saved from Rapid Miner. • On Python 4726 samples are used for modelling on above mentioned models. 8
Evaluation Performance of the model is evaluated by generating/calculating following parameters: • Classification Error – It is the total number of error made by the machine learning model to predict correct data from the total number of predicted samples. • Accuracy - Accuracy is the fraction of correct prediction made by the model to the total number of the samples. • AUC - It is the complete area under the 2-dimensional area under the ROC (Receiver Operating Characteristic) curve. • Precision - Precision is the measure of a model which represents the actual positive values predicted by the model from the total positive values. • Recall/Sensitivity - It is the measure of a model which represents the total number of actual positive values predicted by the model. • F1 Score - It is the harmonic mean between precision and recall. • Specificity - It is defined as the ration of the true negative prediction made by the model to the total number of negative values available in the set. 9
Results – Rapid Miner 10 Parameter/ Model NV GLM LR FLM DT RF GBT SVM DL Classification Error 49.6% (+/-) 0.7% 28.7% (+/-) 0.7% 28.7% (+/-) 0.7% 28.7% (+/-) 0.8% 29.2% (+/-) 0.9% 29.9% (+/-) 0.7% 24.5% (+/-) 1.1% 30.6% (+/-) 0.4% 25.9% (+/-) 1.8% Accuracy 50.3% (+/-) 0.7% 71.2% (+/-) 0.7% 71.2% (+/-) 0.7% 72% (+/-) 0.8% 70.7% (+/-) 0.9% 70% (+/-) 0.7% 75.4% (+/-) 1.1% 69.3% (+/-) 0.4% 74.1% (+/-) 1.8% AUC 7.6% (+/-) 2.3% 81.3% (+/-) 2.8% 81.3% (+/-) 2.8% 81.3% (+/-) 2.6% 71.07% (+/-) 1.6% 78.7% (+/-) 2.5% 81.6% (+/-) 1.1% 79.3% (+/-) 2.3% 82.2% (+/-) 2.5% Precision 49.6% (+/-) 0.8% 88.2% (+/-) 2.4% 88.2% (+/-) 2.4% 85.5% (+/-) 4.7% 87.3% (+/-) 4.2% 84% (+/-) 3.8% 77.4% (+/-) 2.6% 65.7% (+/-) 2.3% 83.4% (+/-) 5.8% Recall (Sensitivity) 99.5% (+/-) 0.6% 47.5% (+/-) 3.2% 47.5% (+/-) 3.2% 50.6% (+/-) 2.9% 45.8% (+/-) 3.7% 46.6% (+/-) 3.6% 72.7% (+/-) 4.3% 77.5% (+/-) 5.7% 58.2% (+/-) 3.1% F Measure 66.2% (+/-) 0.7% 61.7% (+/-) 2.4% 61.7% (+/-) 2.4% 63.4% (+/-) 2% 60% (+/-) 2.8% 59.8% (+/-) 2.5% 74.9% (+/-) 2.5% 71% (+/-) 1.6% 68.4% (+/-) 1.8% Specificity 3.1% (+/-) 3% 93.8% (+/-) 1.6% 93.8% (+/-) 1.6% 91.% (+/-) 3% 93.7% (+/-) 2.5% 91.7% (+/-) 2.5% 78% (+/-) 3.4% 61.7% (+/-) 4.7% 89.0% (+/-) 4.3% Result observed on Rapid Miner’s Auto Model are given below.
Results – Rapid Miner 11 Parameters Model Gradient Boosting Trees Deep Learning Classification Error 24.53% (+/-) 1.1% 25.9% (+/-) 1.8% Accuracy 75.46% (+/-) 1.1% 74.1% (+/-) 1.8% AUC 81.6% (+/-) 1.1% 82.2% (+/-) 2.5% Precision 77.4% (+/-) 2.6% 83.4% (+/-) 5.8% Recall (Sensitivity) 72.7% (+/-) 4.3% 58.2% (+/-) 3.1% F Measure 74.9% (+/-) 2.5% 68.4% (+/-) 1.8% Specificity 78% (+/-) 3.4% 89.0% (+/-) 4.3% GBT DL Two better performing models are compared below.
Results - Python 12 Parameters Model SVM GBC (M)NB DT RF LR Classification Error 10.71% 15.51% 13.04% 17.98% 13.32% 11.92% Accuracy 89.28% 84.86% 86.95% 82.72% 86.11% 88.08% AUC 89.35% 84.81% 87.03% 81.72% 85.69% 88.16% Precision (0/1) 91% / 87% 91% / 77% 89% / 85% 81% / 84% 87% / 85% 90% / 86% Recall (0/1) (Sensitivity) 88%/ 91% 80% / 90% 86% / 89% 84% / 82% 85% / 87% 87% / 90% F Measure (0/1) 90% / 89% 85% / 83% 87% / 87% 83% / 83% 86% / 86% 88% / 88% Specificity 90.80% 89.45% 88.52% 81.12% 86.38% 89.71% Result observed on Python are given below.
Results - Python 13 Parameters Model Support Vector Machine Logistic Regression Gradient Boosting Classifier Classification Error 10.71% 11.92% 15.51% Accuracy 89.28% 88.08% 84.86% AUC 89.35% 88.16% 84.81% Precision (0/1) 91% / 87% 90% / 86% 91% / 77% Recall (0/1) 88%/ 91% 87% / 90% 80% / 90% F Measure (0/1) 90% / 89% 88% / 88% 85% / 83% Specificity 90.80% 89.71% 89.45% SVM LR GBC Three better performing models are compared below.
Results – H2O AI 14 Predicted 0 Predicted 1 Error Rate Actual 0 169.0 276.0 0.6202 (276.0/445.0) Actual 1 42.0 413.0 0.0923 (42.0/455.0) Total 211.0 689.0 0.3533 (318.0/900.0) From the generated confusion matrix, following parameters are derived for Deep Learning Parameter Score Accuracy 64.6% Classification Error 35..4% AUC 74.44% Precision 59.96% Recall 90.76% F Measure 72.21% Specificity 37.97%
Results – Overall 15 •On Rapid Miner there is not so much difference in Classification Error, Accuracy, and AUC between Gradient Boosting Tree (GBT) and Deep Learning (DL) models, which are one of the most important evaluation criteria in machine learning classification algorithms. • Support Vector Machine is clearly outperforming every other traditional machine learning classification model on Python. • Either of the better performing model on Rapid Miner has not performed well with more number of samples on python platform or on H2O AI. • Support Vector Machine has got more score in all classification evaluating parameters. • Deep Learning model on H20 AI is giving unfavourable results if compared with the considered hypothesis. • The results from Deep Learning model on H20 platform do not outperform the Rapid Miner’s Auto model results.
Conclusion and Future Work 16 • From above results and discussion we can observe that Support Vector Machine model is performing better than other State-of-the-Art models with TF-IDF word vector creation. • Rapid Miner auto model’s score can be used as a bench mark for all the platforms and model. Different results can be observed depending on the number of samples. • The future work for this study will involve the use of Recurrent Neural Network with Keras for sentiment classification. • It also involves the use of different word vector creating technique such as Term Frequency (TF), Term Occurrence (TO), and Binary Term Occurrence (BTO).
Thank You

More Related Content

PPTX
Recommendation System
byAnamta Sayyed
 
PDF
Text classification & sentiment analysis
byM. Atif Qureshi
 
PPTX
Amazon Product Sentiment review
byLalit Jain
 
PPTX
social network analysis project twitter sentimental analysis
byAshish Mundra
 
PPTX
Apriori algorithm
byMainul Hassan
 
PDF
Sentiment Analysis
byData Science Society
 
PPTX
Movie lens movie recommendation system
byGaurav Sawant
 
PPTX
Sentiment analysis using ml
byPravin Katiyar
 
Recommendation System
byAnamta Sayyed
 
Text classification & sentiment analysis
byM. Atif Qureshi
 
Amazon Product Sentiment review
byLalit Jain
 
social network analysis project twitter sentimental analysis
byAshish Mundra
 
Apriori algorithm
byMainul Hassan
 
Sentiment Analysis
byData Science Society
 
Movie lens movie recommendation system
byGaurav Sawant
 
Sentiment analysis using ml
byPravin Katiyar
 

What's hot

PPTX
New sentiment analysis of tweets using python by Ravi kumar
byRavi Kumar
 
PPTX
Twitter sentiment analysis ppt
byAntaraBhattacharya12
 
PDF
Sentiment analysis - Our approach and use cases
byKarol Chlasta
 
PPTX
Sentiment Analysis
byAditya Nag
 
PPTX
Twitter sentiment analysis ppt
bySonuCreation
 
PPTX
Amazon seniment
bySubhadeep Chakraborty
 
PPT
Sentiment analysis in Twitter on Big Data
byIswarya M
 
PPTX
Sentiment analysis presentation
byGunjanSrivastava23
 
PPTX
Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)
byWeiwei Guo
 
PPTX
Recommender system
byNilotpal Pramanik
 
PPT
Opinion Mining Tutorial (Sentiment Analysis)
byKavita Ganesan
 
PPTX
Twitter Sentiment Analysis
byAyush Khandelwal
 
PPTX
Recommender systems using collaborative filtering
byD Yogendra Rao
 
PPTX
Intro to deep learning
byDavid Voyles
 
PDF
Sentiment Analysis of Twitter Data
bySumit Raj
 
PPTX
Sentiment Analaysis on Twitter
byNitish J Prabhu
 
PPTX
Clustering in Data Mining
byArchana Swaminathan
 
PPTX
Sentiment analysis using naive bayes classifier
byDev Sahu
 
PDF
Product Recommendations Enhanced with Reviews
bymaranlar
 
PPTX
Dbscan algorithom
byMahbubur Rahman Shimul
 
New sentiment analysis of tweets using python by Ravi kumar
byRavi Kumar
 
Twitter sentiment analysis ppt
byAntaraBhattacharya12
 
Sentiment analysis - Our approach and use cases
byKarol Chlasta
 
Sentiment Analysis
byAditya Nag
 
Twitter sentiment analysis ppt
bySonuCreation
 
Amazon seniment
bySubhadeep Chakraborty
 
Sentiment analysis in Twitter on Big Data
byIswarya M
 
Sentiment analysis presentation
byGunjanSrivastava23
 
Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)
byWeiwei Guo
 
Recommender system
byNilotpal Pramanik
 
Opinion Mining Tutorial (Sentiment Analysis)
byKavita Ganesan
 
Twitter Sentiment Analysis
byAyush Khandelwal
 
Recommender systems using collaborative filtering
byD Yogendra Rao
 
Intro to deep learning
byDavid Voyles
 
Sentiment Analysis of Twitter Data
bySumit Raj
 
Sentiment Analaysis on Twitter
byNitish J Prabhu
 
Clustering in Data Mining
byArchana Swaminathan
 
Sentiment analysis using naive bayes classifier
byDev Sahu
 
Product Recommendations Enhanced with Reviews
bymaranlar
 
Dbscan algorithom
byMahbubur Rahman Shimul
 

Similar to Comparative Study of Machine Learning Algorithms for Sentiment Analysis with TF-IDF Vector Creation

PDF
IRJET- Comparison of Classification Algorithms using Machine Learning
byIRJET Journal
 
PDF
IRJET- Sentimental Analysis for Online Reviews using Machine Learning Algorithms
byIRJET Journal
 
PDF
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
PDF
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
PDF
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
PPTX
Big Data Spain 2018: How to build Weighted XGBoost ML model for Imbalance dat...
byAlok Singh
 
PDF
Ankit presentation
byANKIT AGRAWAL
 
PDF
Sentiment Analysis: A comparative study of Deep Learning and Machine Learning
byIRJET Journal
 
PPTX
Business Analytics Final Capstone Project Presenation PPT.pptx
byKavitha860274
 
PDF
Choosing a Machine Learning technique to solve your need
byGibDevs
 
PDF
IRJET - Online Product Scoring based on Sentiment based Review Analysis
byIRJET Journal
 
PDF
Text Document Classification System
byIRJET Journal
 
PPTX
Build Deep Learning model to identify santader bank's dissatisfied customers
bysriram30691
 
PDF
IRJET- Comparative Study of Classification Algorithms for Sentiment Analy...
byIRJET Journal
 
PPTX
Machine-Learning-Overview a statistical approach
byAjit Ghodke
 
PPTX
Predicting Digital Marketing Success: Conversion Forecasting Strategies
byBoston Institute of Analytics
 
PPTX
ODSC18, London, How to build high performing weighted XGBoost ML Model for Re...
byAlok Singh
 
PDF
IRJET- Supervised Learning Classification Algorithms Comparison
byIRJET Journal
 
PDF
IRJET- Supervised Learning Classification Algorithms Comparison
byIRJET Journal
 
PPTX
Machine Learning: Transforming Data into Insights
bypemac73062
 
IRJET- Comparison of Classification Algorithms using Machine Learning
byIRJET Journal
 
IRJET- Sentimental Analysis for Online Reviews using Machine Learning Algorithms
byIRJET Journal
 
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
THE STUDY OF INCREMENTAL PROCEDURE WITH DEEP LSTM FOR MULTICLASS SENTIMENT AN...
byijait
 
Big Data Spain 2018: How to build Weighted XGBoost ML model for Imbalance dat...
byAlok Singh
 
Ankit presentation
byANKIT AGRAWAL
 
Sentiment Analysis: A comparative study of Deep Learning and Machine Learning
byIRJET Journal
 
Business Analytics Final Capstone Project Presenation PPT.pptx
byKavitha860274
 
Choosing a Machine Learning technique to solve your need
byGibDevs
 
IRJET - Online Product Scoring based on Sentiment based Review Analysis
byIRJET Journal
 
Text Document Classification System
byIRJET Journal
 
Build Deep Learning model to identify santader bank's dissatisfied customers
bysriram30691
 
IRJET- Comparative Study of Classification Algorithms for Sentiment Analy...
byIRJET Journal
 
Machine-Learning-Overview a statistical approach
byAjit Ghodke
 
Predicting Digital Marketing Success: Conversion Forecasting Strategies
byBoston Institute of Analytics
 
ODSC18, London, How to build high performing weighted XGBoost ML Model for Re...
byAlok Singh
 
IRJET- Supervised Learning Classification Algorithms Comparison
byIRJET Journal
 
IRJET- Supervised Learning Classification Algorithms Comparison
byIRJET Journal
 
Machine Learning: Transforming Data into Insights
bypemac73062
 

Recently uploaded

PDF
Fundamentals of Database System (CoSc2041) - Chapter 1.pdf
byssusere261f6
 
PPTX
Agentic Systems and Compliance - A brief intro [1.2]
byrobertowilliams
 
PPTX
MLUP_ppt_1_ for project it help students who are eligible
byanshumanbehera981
 
DOCX
Knowledge Management Information Sheet(58f671121fa4e5fd56cf1025ded949ee).docx
byjesusjeager
 
PDF
Google Search Console: The SEO power tool you're probably misusing
byMartin Hayman
 
PDF
Top 11 Best site Places to Buy LinkedIn Accounts.pdf
bymarketing
 
PPTX
Matrix and determinant URT [Autosaved].pptx
bymom91284
 
PPT
Reporting ov conducting cd analysiss.ppt
byDaffaFauziIrwanto
 
PDF
Top Reasons to Choose WordPress for Business Websites
byWebConnect Pvt Ltd
 
DOCX
How to twitter Ads Accounts in 2025 (1).docx
byBEST IT SMM
 
PDF
511167854-burjkhalifa-150516084413-lva1-app6892.pdf
byshokooh757
 
PPTX
Result Analysis - Mechanical engineering.pptx
bySolomon Raj
 
PPTX
Morgan (Kumpulan Tabel).pptx Morgan (Kumpulan Tabel).pptx
byDhika Adhi Pratama
 
DOCX
Why Buying, Old Gmail Accounts Still Matter.docx
byBEST IT SMM
 
PDF
gis based document for apparture radar analysis
byBedassaDessalegn3
 
PDF
MSc_Dissertation — Remote Work On Mental Health
byRobertSolomon23
 
PDF
Short note for sampling theory and research method
byTigistG2
 
PPTX
AoIR2025_LLM_assisted_frame_analysis.pptx
byLauraVodden
 
PPTX
Group project mgtkspgo gj a akf alkn mkla.pptx
bymadaminbekmahmudov02
 
DOCX
researchmethodologyreport-140226083638-phpapp02.docx
byomgorde230
 
Fundamentals of Database System (CoSc2041) - Chapter 1.pdf
byssusere261f6
 
Agentic Systems and Compliance - A brief intro [1.2]
byrobertowilliams
 
MLUP_ppt_1_ for project it help students who are eligible
byanshumanbehera981
 
Knowledge Management Information Sheet(58f671121fa4e5fd56cf1025ded949ee).docx
byjesusjeager
 
Google Search Console: The SEO power tool you're probably misusing
byMartin Hayman
 
Top 11 Best site Places to Buy LinkedIn Accounts.pdf
bymarketing
 
Matrix and determinant URT [Autosaved].pptx
bymom91284
 
Reporting ov conducting cd analysiss.ppt
byDaffaFauziIrwanto
 
Top Reasons to Choose WordPress for Business Websites
byWebConnect Pvt Ltd
 
How to twitter Ads Accounts in 2025 (1).docx
byBEST IT SMM
 
511167854-burjkhalifa-150516084413-lva1-app6892.pdf
byshokooh757
 
Result Analysis - Mechanical engineering.pptx
bySolomon Raj
 
Morgan (Kumpulan Tabel).pptx Morgan (Kumpulan Tabel).pptx
byDhika Adhi Pratama
 
Why Buying, Old Gmail Accounts Still Matter.docx
byBEST IT SMM
 
gis based document for apparture radar analysis
byBedassaDessalegn3
 
MSc_Dissertation — Remote Work On Mental Health
byRobertSolomon23
 
Short note for sampling theory and research method
byTigistG2
 
AoIR2025_LLM_assisted_frame_analysis.pptx
byLauraVodden
 
Group project mgtkspgo gj a akf alkn mkla.pptx
bymadaminbekmahmudov02
 
researchmethodologyreport-140226083638-phpapp02.docx
byomgorde230
 

Comparative Study of Machine Learning Algorithms for Sentiment Analysis with TF-IDF Vector Creation

  • 1.
    “ Comparative Study ofMachine Learning Algorithms for Sentiment Analysis with TF-IDF Vector Creation Sagar Vijay Deogirkar (10547321) MSc Data Analytics Ms. Terri Hoare Supervisor
  • 2.
    Index • Introduction • ResearchQuestion and Objective • Methodology • Business and Data Understanding • Data Preparation • Modelling • Evaluation • Results • Conclusion and Future Work 2
  • 3.
    Introduction Customers are expressingtheir thoughts about product and offered services more openly than never before. Considering this sentiment analysis is becoming an essential aspect to understand their sentiments. Sentiment analysis cites the use of Natural Language Processing technique to classify the type of sentiment. In other words, Sentiment Analysis is the process to determine if the given text is positive, negative or of neutral sentiment. It is often perform on textual data to help business entities, to monitor their brand’s product or services’ sentiments from client’s reviews. This helps to understand the customer’s requirement which may lead to necessary improvement in product or services if required. 3
  • 4.
    Research Question Sentiment Analysisis trending, many programming and non programming platforms have arrived providing solution to this problem. But problem lies with platform selection and more over that, to the model or algorithm selection. The main obstacle is to know, which algorithm can be chosen with TF-IDF vector creation technique, which will lead to determine the class of the sentiment more correctly. 4 Objective The main objective of this research is to compare the State-of-the-Art Deep Learning with Machine Learning algorithms performance on TF-IDF vector creation for Sentiment Analysis.
  • 5.
    Methodology Data Understanding 5 Data Preparation Modelling Evaluation Business Understanding Deployment • DeterminingBusiness Objective • Assess The Situation • Determine the Study Goal • Produce a Project Plan • Data Selection • Clean Data • Construct Data • Integrate Data • Format Data • Evaluate Result • Review Process • Determine Next Step • Data Collection • Describing the Data • Data Exploration • Verifying Data Quality • Select the Model • Generate Test Design • Build the Model • Assess the Model • Plan Development • Plan Monitory and Maintenance • Produce Final Report • Review Report This is research conducted following CRISP-DM (Cross-Industry Standard Process for Data Mining) methodology.
  • 6.
    Business Understanding Sentiment classificationis a term which comprises the method to determine the labelled sentiment from the available classes based on the aligned text data. It helps to identify the emotions i.e. sentiment behind the text of high volume data. Text data could be reviews from YouTube, or from any social media platform, or tweets on trending topic involving different hash tags and etc, or articles or news report or such which is in the form of text. 6 Data Understanding • For this research “Twitter Airline” dataset is used. • Dataset comprises of total 14 features and a label, having a total of 14640 rows. • Column Names - tweet id, airline sentiment, airline sentiment confidence, airline sentiment gold, negative reason, negative reason confidence, airline, name, negative reason gold, re tweet count, text, tweet coord, tweet created, tweet location, user time zone. •Sentiment Distribution: 9178 – Negative, 3099 – Neutral and 2363 - Positive. • From the above features only airline_sentiment and text is selected for the research.
  • 7.
    Data Preparation • TextPre-processing – Lowercasing is done. Unnecessary symbols and numbers are removed. • Sentiment Class Filtering - Neutral class’s sentiment is filtered and positive is turned to 1 and Negative to 0. • Data Balancing – Positive and Negative class sentiment is balanced to same number of samples. • Removing Stop Words – Common words in the language are removed. • Text Stemming – Porter Stemming is used to make word into its original form. • Tokenization – Every word is separated in the document. • TF-IDF vector – TF- IDF word vector is created having all the words in the data set with their weight. 7 Text Cleaning Text Processing Vector Creation Data Importation TF- IDF Vector Creation • Text Stemming • Tokenization • Data Balancing • Lowercasing • Removing Symbols and Numbers • Removing Stop Words. Data Importation to the Platform and Considering the Features and label.
  • 8.
    Modelling • Selected Modelsare: Naive Bayes, Support Vector Machine (SVM), Generalised Linear Model (GLM), Logistic Regression, Decision Tree, Random Forest, Gradient Boosted Trees, and Deep Learning. • On Rapid Miner Auto Model is used with 3000 samples. •Deep learning (Neural Network) is observed on H2O AI platform with 3000 samples processed and saved from Rapid Miner. • On Python 4726 samples are used for modelling on above mentioned models. 8
  • 9.
    Evaluation Performance of themodel is evaluated by generating/calculating following parameters: • Classification Error – It is the total number of error made by the machine learning model to predict correct data from the total number of predicted samples. • Accuracy - Accuracy is the fraction of correct prediction made by the model to the total number of the samples. • AUC - It is the complete area under the 2-dimensional area under the ROC (Receiver Operating Characteristic) curve. • Precision - Precision is the measure of a model which represents the actual positive values predicted by the model from the total positive values. • Recall/Sensitivity - It is the measure of a model which represents the total number of actual positive values predicted by the model. • F1 Score - It is the harmonic mean between precision and recall. • Specificity - It is defined as the ration of the true negative prediction made by the model to the total number of negative values available in the set. 9
  • 10.
    Results – RapidMiner 10 Parameter/ Model NV GLM LR FLM DT RF GBT SVM DL Classification Error 49.6% (+/-) 0.7% 28.7% (+/-) 0.7% 28.7% (+/-) 0.7% 28.7% (+/-) 0.8% 29.2% (+/-) 0.9% 29.9% (+/-) 0.7% 24.5% (+/-) 1.1% 30.6% (+/-) 0.4% 25.9% (+/-) 1.8% Accuracy 50.3% (+/-) 0.7% 71.2% (+/-) 0.7% 71.2% (+/-) 0.7% 72% (+/-) 0.8% 70.7% (+/-) 0.9% 70% (+/-) 0.7% 75.4% (+/-) 1.1% 69.3% (+/-) 0.4% 74.1% (+/-) 1.8% AUC 7.6% (+/-) 2.3% 81.3% (+/-) 2.8% 81.3% (+/-) 2.8% 81.3% (+/-) 2.6% 71.07% (+/-) 1.6% 78.7% (+/-) 2.5% 81.6% (+/-) 1.1% 79.3% (+/-) 2.3% 82.2% (+/-) 2.5% Precision 49.6% (+/-) 0.8% 88.2% (+/-) 2.4% 88.2% (+/-) 2.4% 85.5% (+/-) 4.7% 87.3% (+/-) 4.2% 84% (+/-) 3.8% 77.4% (+/-) 2.6% 65.7% (+/-) 2.3% 83.4% (+/-) 5.8% Recall (Sensitivity) 99.5% (+/-) 0.6% 47.5% (+/-) 3.2% 47.5% (+/-) 3.2% 50.6% (+/-) 2.9% 45.8% (+/-) 3.7% 46.6% (+/-) 3.6% 72.7% (+/-) 4.3% 77.5% (+/-) 5.7% 58.2% (+/-) 3.1% F Measure 66.2% (+/-) 0.7% 61.7% (+/-) 2.4% 61.7% (+/-) 2.4% 63.4% (+/-) 2% 60% (+/-) 2.8% 59.8% (+/-) 2.5% 74.9% (+/-) 2.5% 71% (+/-) 1.6% 68.4% (+/-) 1.8% Specificity 3.1% (+/-) 3% 93.8% (+/-) 1.6% 93.8% (+/-) 1.6% 91.% (+/-) 3% 93.7% (+/-) 2.5% 91.7% (+/-) 2.5% 78% (+/-) 3.4% 61.7% (+/-) 4.7% 89.0% (+/-) 4.3% Result observed on Rapid Miner’s Auto Model are given below.
  • 11.
    Results – RapidMiner 11 Parameters Model Gradient Boosting Trees Deep Learning Classification Error 24.53% (+/-) 1.1% 25.9% (+/-) 1.8% Accuracy 75.46% (+/-) 1.1% 74.1% (+/-) 1.8% AUC 81.6% (+/-) 1.1% 82.2% (+/-) 2.5% Precision 77.4% (+/-) 2.6% 83.4% (+/-) 5.8% Recall (Sensitivity) 72.7% (+/-) 4.3% 58.2% (+/-) 3.1% F Measure 74.9% (+/-) 2.5% 68.4% (+/-) 1.8% Specificity 78% (+/-) 3.4% 89.0% (+/-) 4.3% GBT DL Two better performing models are compared below.
  • 12.
    Results - Python 12 Parameters Model SVMGBC (M)NB DT RF LR Classification Error 10.71% 15.51% 13.04% 17.98% 13.32% 11.92% Accuracy 89.28% 84.86% 86.95% 82.72% 86.11% 88.08% AUC 89.35% 84.81% 87.03% 81.72% 85.69% 88.16% Precision (0/1) 91% / 87% 91% / 77% 89% / 85% 81% / 84% 87% / 85% 90% / 86% Recall (0/1) (Sensitivity) 88%/ 91% 80% / 90% 86% / 89% 84% / 82% 85% / 87% 87% / 90% F Measure (0/1) 90% / 89% 85% / 83% 87% / 87% 83% / 83% 86% / 86% 88% / 88% Specificity 90.80% 89.45% 88.52% 81.12% 86.38% 89.71% Result observed on Python are given below.
  • 13.
    Results - Python 13 Parameters Model Support Vector Machine Logistic Regression Gradient Boosting Classifier Classification Error 10.71%11.92% 15.51% Accuracy 89.28% 88.08% 84.86% AUC 89.35% 88.16% 84.81% Precision (0/1) 91% / 87% 90% / 86% 91% / 77% Recall (0/1) 88%/ 91% 87% / 90% 80% / 90% F Measure (0/1) 90% / 89% 88% / 88% 85% / 83% Specificity 90.80% 89.71% 89.45% SVM LR GBC Three better performing models are compared below.
  • 14.
    Results – H2OAI 14 Predicted 0 Predicted 1 Error Rate Actual 0 169.0 276.0 0.6202 (276.0/445.0) Actual 1 42.0 413.0 0.0923 (42.0/455.0) Total 211.0 689.0 0.3533 (318.0/900.0) From the generated confusion matrix, following parameters are derived for Deep Learning Parameter Score Accuracy 64.6% Classification Error 35..4% AUC 74.44% Precision 59.96% Recall 90.76% F Measure 72.21% Specificity 37.97%
  • 15.
    Results – Overall 15 •OnRapid Miner there is not so much difference in Classification Error, Accuracy, and AUC between Gradient Boosting Tree (GBT) and Deep Learning (DL) models, which are one of the most important evaluation criteria in machine learning classification algorithms. • Support Vector Machine is clearly outperforming every other traditional machine learning classification model on Python. • Either of the better performing model on Rapid Miner has not performed well with more number of samples on python platform or on H2O AI. • Support Vector Machine has got more score in all classification evaluating parameters. • Deep Learning model on H20 AI is giving unfavourable results if compared with the considered hypothesis. • The results from Deep Learning model on H20 platform do not outperform the Rapid Miner’s Auto model results.
  • 16.
    Conclusion and FutureWork 16 • From above results and discussion we can observe that Support Vector Machine model is performing better than other State-of-the-Art models with TF-IDF word vector creation. • Rapid Miner auto model’s score can be used as a bench mark for all the platforms and model. Different results can be observed depending on the number of samples. • The future work for this study will involve the use of Recurrent Neural Network with Keras for sentiment classification. • It also involves the use of different word vector creating technique such as Term Frequency (TF), Term Occurrence (TO), and Binary Term Occurrence (BTO).
  • 17.