An ensemble approach to stabilize the features for multi-domain sentiment analysis using supervised machine learning

Abstract

Sentiment classification or sentiment analysis has been acknowledged as an open research domain. In recent years, an enormous research work is being performed in these fields by applying numerous methodologies. Feature generation and selection are consequent for text mining as the high dimensional feature set can affect the performance of sentiment analysis. This paper investigates the inability of the widely used feature selection method (IG, Chi Square, Gini Index) individually as well as their combined approach on four machine learning classification algorithm. The proposed methods are evaluated on three standard datasets viz. IMDb movie review, electronics and kitchen product review dataset. Initially, select the feature subsets from three different feature selection methods. Thereafter, statistical method UNION, INTERSECTION and revised UNION method are applied to merge these different feature subsets to obtain all top ranked including common selected features. Finally, train the classifier SMO, MNB, RF, and LR (logistic regression) with this feature vector for classification of the review data set. The performance of the algorithm is measured by evaluation methods such as precision, recall, F-measure and ROC curve. Experimental results show that the combined method achieved best accuracy of 92.31 with classifier SMO, which is encouraging and comparable to the related research.

Introduction

An opinion is a viewpoint or judgment about a specific thing that acts as a key influence on an individual process of decision making. People’s belief and the choices they make are always dependent on how others see and evaluate the world. So opinion holds high values in many aspect of life. Sentiment analysis is the process of determining opinions or sentiments in textual documents as positive, or negative. In recent years, this field is widely appreciated by researchers due to its dynamic range of application in various numbers of fields. There are several areas such as marketing; politics; news analytics etc. which are benefited from the result of sentiment analysis. Due to the vast range of movies these days, it has become difficult for the audience to select their preferred genre of movie. Movie reviews turn out to be very useful reference. Despite of the willingness of people to share their thoughts and views about the movies, a problem persists due to the huge amount of total reviews. This develops a need for technology of data mining to uncover information. These solutions can be roughly categorized into machine learning approach and lexicon-based approach to solve the problem of sentiment classification. The former approach was applied to classify the sentiments based on trained as well as test data sets. The second category doesn’t require any prior training data set, it performs the task by identifying a list of words, phrases that consists of a semantic value. It mainly concentrates on patterns of unseen data. There are few researchers applied hybrid approaches [1, 2] by combining both approaches machine learning and lexical to improve the sentiment classification performance.

This field becomes more challenging due to the fact that many demanding and interesting research problems still exist in this field to solve. Sentiment based analysis of a document is quite tough to perform in comparison with topic based text classification. The opinion words and sentiments are always varied with situations. Therefore, an opinion word can be considered as positive in one circumstance but may be that becomes negative in some other circumstance. The opinionated word ‘unpredictable’ is used in different senses in a different domain. For example, “an unpredictable plot in the movie” gives a positive opinion about the movie, while “an unpredictable steering wheel” is a negative expression considering the product, car [3].

Sentiment classification process has been classified into three levels: document level, sentence level, and feature level. The entire document at the document level, based on the positive or negative opinion, is expressed by the authors. Sentiment classification at the sentence level, considers the individual sentence to identify whether the sentence is positive or negative. In feature level, we classify the sentiment with respect to the specific aspects of entities. Aspect level sentiment classification needs deeper analysis on features, mainly which are expressed implicitly and usually are hidden in a large text dataset. During this study, the focus has been made on feature level sentiment classification. We present the impact of supervised learning method on labelled data.

The main contribution of the paper can be stated in particular as:

• 1. We provide a novelty sentiment classification method based on feature selection and ML technique and the proposed method evaluate on three standard benchmark datasets such as: movie reviews of IMDb, Electronics and kitchen review datasets. We carried out experiments considering the 10-fold cross validation, as product review dataset consists of separate files for positive and negative reviews but training and testing data are not isolated.

  For movie review dataset, 25,000 samples are categorized as for training and another 25,000 for testing purpose. However, we noticed the distribution is sub-optimal since the training samples are not sufficient according to 25,000 testing reviews. Finally, to improve the performance of classifier we decided to use cross validation for movie as well as product review datasets.

• 2. We employed IG method as a single univariate method with low complexity, which ranks the features based on high information gain entropy in decreasing order. However, the IG method cannot handle redundant features. We addressed this problem by considering CHI and Gini Index as a multivariate and mutual information based method to find and filter the redundancy among relevant features. To achieve better accuracy we combine univariate and multivariate method by applying some statistical method UNION, INTERSECTION and revised UNION.

• 3. We trained the above feature representation on four different classification models namely, SMO, MNB, LR and RF to classify the sentiment polarity of review datasets.

• 4. Finally, the performance of the proposed approach has compared based on evaluation parameters like precision, recall, F-measure and AUC with the results in an existing work obtained by different researchers.

The rest of the paper is constructed as the following: “Related work” section consists of the existing literature that can connect to our approach. Then “Proposed approach” section describes the approaches used in this paper for polarity detection. “Methodology” section and “Combination of feature selection methods” section explains methodology includes features and proposed feature selection technique. The detail regarding implementation of proposed classification algorithm discussed in “Classification”. The particulars about experiments and results are expounded in “Experiments and results” section. Finally, “Conclusion” section concludes with a discussion of the proposed method and with ideas on future steps.

Related work

In current years, sentiment analysis of social media [4] content has become one of the most sought area among researchers because the number of product review sites, social networking sites, blogs, forums are developing extensively. This field mainly utilizes supervised, unsupervised and semi supervised technique for sentiment prediction and classification task. In this section we provide a brief overview of the previous studies regarding [5] supervised multiple machine learning (ML) algorithms [6].

Boiy et al. [7] employed three different ML algorithms such as SVM, NBM and ME. They considered N-gram features such as Unigram, Bigram and their combination. The performance of NBM algorithm was convincing according their analysis. Research work of Dave et al. [8] used some tools for analysis the reviews from Amazon and CNET for classification. They select bigram and trigram features using N-gram model and some scoring methods are applied finally to determine whether the review holds positive or negative opinion. SVM and NB classifier were implemented for sentence level classification with the accuracy of 87.0.

The movie reviews dataset IMDb was used to study by Annett and Kondrak [9]. They adopted lexical resource WordNet for sentiment extraction. Different classifier such as SVM, NB, alternating decision tree used for review classification and more than 75% accuracy was achieved.

Zhang et al. [10] proposed a classification approach of Chinese reviews on clothing product. They applied word2vec and SVM^pref technique while word2vec helps to capture the semantic features based on semantic relationship. SVM^pref is nothing but an alternative structural formulation of SVM optimization problem for binary classification. They achieved good outcomes of this combination for sentiment classification. Mouthami et al. [11] proposed new approach as sentiment fuzzy classification algorithm on the movie review dataset to improve the classification accuracy. Preprocessing method tokenization, stop word removal, TF-IDF, and POS tagging are used for initial pruning [12] they researched on travel blogs and applied various machine learning algorithm NB, SVM by considering the N-gram model to obtain the feature set. In this study, SVM worked best with 85.14% accuracy.

The author [13] approached an ensemble framework to perform sentiment analysis by combining different feature subsets and classification algorithms. The feature selection techniques, they applied are POS based feature sets and the word-relation based feature sets and thereafter these features are fed to three base classifiers such as, NB, ME and SVM. The aim of this paper is to perform the sentiment classification by employing three types of ensemble methods, including the fixed combination, weighted combination and meta-classifier combination. The highest accuracy they achieved 88.65 with kitchen dataset.

Whitehead et al. [14, 15] proposed to apply SVM as a base classifier with four different ensemble techniques such as boosting, bagging, random subspace, and bagging random subspaces. They achieved best performance through random subspace and bagging subspace method.

This research work [16] investigated on the behaviour of five feature selection method such as: Chi square, Correlation, GSS Coefficient Information Gain and Relief F. The final feature subset has been selected based on the average weight of the features assigned by combined feature selection method. SVM and NB classifier are employed to classify the sentiment of Arabic review corpus. The authors claimed that combined feature selection method outperformed the individual method with SVM classification algorithm.

The researchers [17] obtained the highest accuracy 86.9 after combining the feature selection method CHI, DFD and OCFS. They implemented a maximum entropy modelling (MEM) classifier to accomplish sentiment classification and the performance of classifier evaluated on movie review dataset with fivefold cross validation.

Agarwal et al. [1] have proposed a hybrid method merging rough set theory and Information Gain for sentiment classification. These methods are evaluated on four standard datasets such as: Movie review (IMDb) and product (book, DVD, and electronics) review dataset. SVM and NB classifier is used with tenfold cross validation for classifying sentiment polarity of review documents. F1-measure value is considered as a performance measure with maximum 87.7 and 80.9 for SVM and NB classifier.

Kolog et al. [18] implemented machine learning techniques to perceive sentiments in text form, regarding social influences on student’s life story. They applied k-means algorithm for clustering purpose and after that the main influences are identified and those are considered as class level for classification task. The supervised machine learning classifier MNB, SMO and J48 are employed with tenfold cross validation to detect the sentiment either as positive or negative.

The feature selection stage primarily helps in refining features, which are considered as input for classification task. Feature selection is definitely a beneficial task considered by Narayanan et al. [19] based on the experimental result. They have applied only Mutual Information feature selection method with Naïve Bayes (NB) classifier in the domain of movie review.

Amolik et al. [20] proposed a model for sentiment prediction of more than 21,000 tweets by applying the machine learning classifier SVM and NB. Feature vectors were also created to handle the problem of repeating characters in Twitter. They achieved the higher accuracy with SVM was 75% in comparison with NB (65%) by using evaluation matrices precision and recall. A huge number of research papers with different ML classifiers namely Naive Bayes (NB) [11, 21] Support Vector Machine (SVM) [10, 22,23,24], maximum entropy [17, 25, 26], decision trees [9, 21, 27] have been used mostly to build classification model in different domain (Table 1).

Table 1 Research work related to machine learning classifiers for sentiment analysis

Proposed approach

The proposed classification methods are summarized into several steps as described below:

• 1. Data collection In this work, movie review database (IMDB) and product review (Electronics, Kitchen) database are considered to solve the problem regarding sentiment classification.

• 2. Pre-processing This technique is required to remove noisy, inconsistent and incomplete information by considering tokenization, stop words removal, stemming method.

• 3. Feature extraction and selection Initially, to create a feature vector with numeric value we consider binary presence or absence of a feature in a document. The features score will be 1 if it presents in a document otherwise score will be 0. Next, feature selection method IG, CHI, and Gini Index applied to select different feature subsets. Then to generate a top ranked feature sub list we combined all three feature subsets.

• 4. Classification Finally, train the supervised machine learning classifiers SMO, MNB, RF and LR with the different feature sub list for different domain.

Methodology

Text classification as a research field was introduced long time ago [28], however, sentiment based categorization was initiated more recently [18, 25, 29]. The main purpose of this research work is to investigate the performance of various machine learning classifier (MLC) with three combined feature set. The whole process can be completed in four step including Data acquisition, pre-processing, feature selection and classification. A general overview of the proposed framework is introduced with Fig. 1, and the following subsections consist of a detailed description about each preliminary function.

Fig. 1

The architecture of a proposed framework for sentiment classification task

Dataset preparation

We conducted experiments on movie review data set [30], which were prepared by Pang and Lee 2004 [29]. This study uses movie review and product review dataset (Electronics and kitchen) to perform sentiment classification task. The movie review dataset is one of the popular benchmark dataset, which has been exploited by several researchers in order to analyze the experimental outcomes. The standard movie review dataset consists of overall 2000 reviews where 1000 reviews are tagged as positive and 1000 s are negative. The amazon products review dataset [31] provided by Blitzer et al. [32] are considered for investigation and we adopted the data set of Electronics and Kitchen domain from the corpus produced by Blitzer et al. Each domain of this corpus has 1000 pos+ and 1000 neg− labeled reviews. The pre-processing is approved to make this three dataset prepare for experiment. The statistics of this data set is given in Table 2.

Table 2 The detailed statistics of above mentioned datasets

Pre-processing

• Tokenization or segmentation It can be accomplished by splitting documents (crawled reviews) into a list of tokens such as word, numbers, special characters etc. and make the document ready to be used for further processing.

• Normalization This process converts all the word token of a document into either lower case or upper case because most of the reviews consist of both case i.e. lowercase and uppercase characters. The purpose of shifting all tokens into a single format can easily be used for prediction.

• Removal of stop words Stop words are very common and high-frequency words. This process carried out by removing frequently used stop words (prepositions, irrelevant words, special character, ASCII code), new line, extra white spaces etc. to enhance the performance of feature selection technique.

• Stemming It is the process of transforming all the tokens into their stem, or root form. Stemming is a swift and easy approach that makes the feature extraction process more effortless.

Feature selection

Feature selection method (FSM) is an essential task to enhance the accuracy of sentiment classification process. Generally, FSMs are statistically represented by the relationship between feature and class category. The performance of the classifier mostly depends on the feature set, if feature selection method [33] performs well then the simplest classifier may also give a good accuracy through training. These FSMs are often defined by some probabilities to realize the theoretical analysis of these probabilistic methods. We use a list of notation which is depicted in Table 3.

Table 3 Notation use for feature selection

Analytical information from the training data is required to determine these probabilities and notations about the training data are listed in Table 2 given as follows:

We denote $C_{i = 1}^{m} = \left\{ {{\text{c}}_{ 1,} {\text{c}}_{ 2} , \ldots ,{\text{c}}_{\text{m}} } \right\}$ is the set of classes (Table 4).

Table 4 Notation use for feature selection

Information Gain (IG)

This statistical property used as an effective solution for feature selection. IG method is used to select important features based on the class attribute rules of features classification. The IG value of each term can measures the number of bits of information acquired for class prediction by knowing the presence or absence of that term in the document [34]. The IG value of a certain term or feature is calculated by the following equation:

$$\begin{aligned} {\text{IG}}\left( {\text{f}} \right) = & \left\{ { - \mathop \sum \limits_{i = 1}^{m} P\left( {c_{i} } \right)\log P\left( {c_{i} } \right)} \right\} + \left\{ {P(f)\left[ {\mathop \sum \limits_{i = 1}^{m} P(c_{i} |f )logP(c_{i} |f)} \right]} \right\} \\ & + \left\{ {P(f)\left[ {\mathop \sum \limits_{i = 1}^{m} P\left( {c_{i} |f} \right)\log P\left( {c_{i} |f} \right)} \right]} \right\} \\ \end{aligned}$$

(1)

And it is defined as

$$\begin{aligned} \text{IG}(\text{f}) = & \left\{ { - \mathop \sum \limits_{i = 1}^{m} \frac{{N_{i} }}{{N_{all} }}\log \frac{{N_{i} }}{{N_{all} }}} \right\} + \left( {\mathop \sum \limits_{i = 1}^{m} W_{i} / N_{all} } \right)\left[ {\mathop \sum \limits_{i = 1}^{m} \frac{{W_{i} }}{{W_{i} + X_{i} }} log\frac{{W_{i} }}{{W_{i} + X_{i} }}} \right] \\ & + \left( {\mathop \sum \limits_{i = 1}^{m} Y_{i} /N_{all} } \right)\left[ {\mathop \sum \limits_{i = 1}^{m} \frac{{Y_{i} }}{{Y_{i} + Z_{i} }} log\frac{{Y_{i} }}{{Y_{i} + Z_{i} }}} \right] \\ \end{aligned}$$

(2)

IG offers a ranking of the features depending on their IG score, thus a certain number of features can be selected easily.

Chi square (χ²)

Chi square (χ²) is a very commonly applied statistical test, can quantify the association between the feature or term f and its related class c_i. It tests a null-hypothesis that, the two variables feature and class is completely independent of each other. The CHI value of feature f for class C_i is higher, the closer relationship exists between the variables feature f and class C_i. The features with the highest χ² values for a category should perform best for classifying the documents. The formulation of this method as follows:

$$\chi^{ 2} \left( {{\text{f}},{\text{c}}_{\text{i}} } \right) = \frac{{N_{all..} (W_{i} Z_{i} - Y_{i} X_{i} )^{2} }}{{\left( {W_{i} + Y_{i} } \right).\left( {X_{i} + Z_{i} } \right).\left( {W_{i} + X_{i} } \right).(Y_{i} + Z_{i} ) }}$$

(3)

It can also be defined by considering Y_i as (N_i − W_i) and Z_i as (N_all − N_i − X_i) and the above formula is rewritten as follows

$$\chi^{ 2} \left( {{\text{f}},{\text{c}}_{\text{i}} } \right) = \frac{{N_{all} .[W_{i} \left( {N_{all} - N_{i} - X_{i} } \right) - (N_{i} - W_{i} )X_{i} ]^{2} }}{{N_{i} .\left( {N_{all} - N_{i} } \right).\left( {W_{i} + X_{i} } \right).[N_{all} - (W_{i} + X_{i} )]}}$$

(4)

Gini Index (GI)

Gini Index measures the features ability to discriminate between classes. This method was mainly proposed to use for decision tree algorithm based on an impurity split method. The main principle of Gini Index is to consider S as a dataset of the sample having m number of different classes $C_{i = 1}^{m}$ = {c_1, c₂,…,c_m}. According to the class level, the sample set can be splitted into n subset $(Si$, i = 1, 2,…,n). The Gini index of the set S is

$${Gini\, Index} ( S) = 1 - \sum\limits_{{{\text{i}} = 1}}^{{\text{n}}} {{\text{P}}}_{\text{i}}^{2}$$

(5)

where probability P_i of any sample belongs to class C_i, can be computed by S_i/S [35]. Gini Index for a feature can be estimated independently for binary classification. We adopted Gini index Text (GIT) method for calculating the feature score, which was introduced by Park et al. [36]. This algorithm enhanced to overcome the limitations of Gini Index method.

According to previous notation defined in Table 3, we can compute the Gini Index for a feature f of document d belongs to class C_i.

$$GIT_{wi} \left( {f,C_{i} } \right) = P\left( {C_{i} \left| f \right.} \right)^{2}$$

(6)

$$GIT_{Xi} \left( {f,C_{i} } \right) = \left| {\frac{{P(Ci|f)^{2} }}{{\log_{2} P(f)}}} \right|$$

(7)

Combination of feature selection methods

As each feature selection method applied with different rules to extract a feature subset, it outcomes various feature subsets for same dataset. We merged these different feature sub lists by adopting either statistical method UNION to select all features or INTERSECTION to select only common features. In our paper, modified UNION method has been considered to obtain all top ranked including common selected features.

Let F {f1, f2,…,fn}be the primary feature set selected by preprocessing the review dataset D review dataset D. The feature subsets F_SUB1 {f₁₁, f₁₂,…,f_IG}, F_SUB2 {f₂₁, f₂₂,…,f_CHI} and F_SUB3 {f₃₁, f₃₂,…,f_CHI}, are selected with Info Gain, CHI and Gini Index, respectively. All the features exists in these subsets are must be sorted according to their score or weight.

To generate a feature sub list with UNION, we just combine all the features from above three feature selection method.

$$\text{F}_{{\text{SUB}}} 4 = \text{F}_{{\text{SUB}}} 1 \cup \text{F}_{{\text{SUB}}} 2 \cup \text{F}_{{\text{SUB}}} 3$$

(8)

To obtain a feature sub list with INTERSECTION, we only select common features in all three feature subsets.

$$\text{F}_{{\text{SUB}}} 5 = \text{F}_{{\text{SUB}}} 1 \cap \text{F}_{{\text{SUB}}} 2 \cap \text{F}_{{\text{SUB}}} 3$$

(9)

Next we applied revised UNION approach to catch all top ranked features along with common features. As the features are already sorted, the highest scored features and lowest scored features are got there accurate positions. Therefore, again we tested UNION and INTERSECTION method over top ranked (T1) and lowest rank (L1) subsequently.

According to revised UNION approach, we applied union on top T1% of features and intersection on remaining L2%.

$$\begin{aligned} {\text{F}}_{{{\text{SUB}}}} 6 = \ & \left\{ {{\text{T1}}\% {\mkern 1mu} \left\{ {{\text{ F}}_{{{\text{SUB}}}} 1} \right\}{\text{UT1}}\% \left\{ {{\text{F}}_{{{\text{SUB}}}} 2} \right\}{\text{UT1}}\% \left\{ {{\text{F}}_{{{\text{SUB}}}} 3} \right\}} \right\} \\ & {\text{U}}\{ {\text{L1}}\% \left\{ {{\text{F}}_{{{\text{SUB}}}} 1} \right\} \cap \left\{ {{\text{L1}}\% \left\{ {{\text{F}}_{{{\text{SUB}}}} 2} \right\} \cap \left\{ {{\text{L1}}\% \left\{ {{\text{F}}_{{{\text{SUB}}}} 3} \right\}} \right\}} \right\} \\ \end{aligned}$$

(10)

These merged feature sub list will be employed to learn to the supervised classifiers to compare the performance of classifiers with feature subsets obtained from individual feature selection method.

Classification

Machine learning techniques are widely used in artificial intelligence and document classification. Extracted feature sets are used to train the classifier to classify the review of the data set as positive or negative. We applied generative classification model (MNB) and discriminative model (SMO, LR and RF) as a prominent classification approach. Generative model captures p (d, c) and p (c), and then directly computes p (c, d) with conditionally independent assumption between features, on smaller dataset. The reasons to select these classifiers are attributed as follows:

• For this research work, we applied first multivariate Bernoulli naïve bayes (BNB) classifier which considers features vector only with binary or boolean values. The vector can focus on the presence or absence of the feature and not worried about how many times that feature occurs in the document.

• We considered multinomial naïve bayes (MNB) to be applied to overcome the aforementioned limitations of BNB. MNB classifier utilized to classify the document based on frequency counts of multiple features.

• MNB classifier is not comfortable with imbalanced (one class having more data samples than others) training dataset, while SMO can equally deal with imbalanced as well as balanced dataset.

• Based on the priority of the data status, we assign one more classifier suitable for binary classification that is linear regression (LR). We utilized the main advantage [16] of LR classifier i.e., to use the non linear function with a linear combination of features.

• According to previous studies, we designed Random forest classifier (RF) designed as consistently accurate to predict the sentiment’s class labels for large scale data set. This classifier always a good choice as it can tackle different types of features without scaling including, numerical, categorical, binary features.

Naive Bayes (NB)

Naive Bayes classification method is used for both purpose; classification as well as training. The fundamental theory of NB classifier [37] is based on the independence assumption; where the joint probabilities of features and categories are used to roughly calculate the probability score of categories of a given document. It is a simple probabilistic classifier, that helps in classifying a document d_r, out of classes c_i∈C ($C_{i = 1}^{m}$ = c_1, c₂,…..c_m). The best class returns in NB classification is the most probably or maximum posterior (MAP) class C_map.

$$\mathop {{\text{C}}_{\text{map}} }\limits_{{{\text{c}}_{\text{i}} \in {\text{C}}}} = argmaxP\left( {c_{i} } \right)P\left( {d_{r} |c_{i} } \right)$$

(11)

where the class P(c_i) can be estimated by dividing the number of documents of class c_i by the total number of documents. $P(d_{r} |c_{i} )$ indicated the number of occurrence of the feature in document d_r belongs to class c_i. The probability value P(c_i| $d_{r}$) will be computed for each possible class, but P($d_{r}$) doesn’t change for each class. Thus we can drop the denominator.

We thus select the highest probable classes’ c_map of given document d by calculating the posterior probability of each class.

There are several Naive Bayes variations. In this paper, we consider the Multi-nominal Naive Bayes classifier.

Multi-nominal Naïve Bayes (MNB)

The multi-nominal Naive Bayes model [38] is distinctly used for discrete counts. We consider MNB classifier for text classification task, where a document d is represented by a feature vector (f_1, f_2…, fn) with the integer value of word frequency in the given document. For multinomial NB model, The conditional distribution P (d|c_i) of document d given the class c is as follows:

$${\text{Multi-nominal}}\,\,\,P(|c_{i} ) = {\text{ P}}\left( {\left( {{\text{f}}_{ 1} ,{\text{f}}_{2} ,\ldots , {\text{f}}_{\text{n}} } \right) \, |{\text{c}}_{\text{i}} } \right) = \mathop \prod \limits_{1 \le j \le n} P(f_{j} |c_{i} )$$

(12)

The final equation with Bayes’ rules the highest probable classes by a Naive Bayes classifier as follows:

$$C_{\text{map}} = \mathop {argmax}\limits_{{{\text{c}}_{\text{i}} \in {\text{C}}}} \hat{P}\left( {c_{i} } \right)\mathop \prod \limits_{1 \le j \le n} \hat{P}(f_{j} |c_{i} )$$

(13)

Now, to estimate the probability $\hat{P}(f_{j} |c_{i} )$ we consider the feature as a word appears in the document’s bag of words. Thus we’ll compute $\hat{P}(wj|c_{i} )$ by considering N_jr as the number of occurrence of word wj in documents $d_{r}$ from class c_i among all words in all documents of class c_i. Then the estimated probability of a document given its class is given as follows.

$$P(d_{r} |c_{i} )= \left( {\sum\limits_{j}^{|v|} {N_{jr} } } \right)! \prod\limits_{j = 1}^{|v|} {\frac{{\hat{P}(w_{j} |c_{i} )^{{N_{jr} }} }}{{N_{jr} !}}}$$

(14)

where, v is the union of all the word types in all classes.

The probability of w_j in c_i is estimated from training dataset and it is defined as follows.

$$\hat{P} ( wj| {c_{i} } ) = \frac{{{\text{count(w}}_{\text{j}} {{\text{c}}}_{\text{i}} )}}{{\sum\nolimits_{{w \in {\text{V}}}} count(w,{\text{c}}_{\text{i}} )}}$$

(15)

Support Vector Machine (SVM)

Support Vector Machines (SVMs) are supervised learning model introduced [15] for binary classification in both linear and nonlinear versions. Generally, datasets are non-linearly inseparable, so the primary aim of the SVM classifier is to catch the best accessible surface to make a separation between positive and negative training samples based on empirical risk (training set and test set error) minimization principal. SVM method can try to define a decision boundary with the hyper-planes in a high dimensional feature space. This hyper-plane separates the vectorized document into two classes as well as determines a result to make a decision based on this support vector [5]. The optimization problem of SVM can be minimized as follows.

Given N linearly separable training set with feature vector x of d dimension. For dual optimization where α ϵ R^N and y ϵ {1, − 1}. Then the solution of SVMs (dual) can be minimized as follows:

$${}_{{\alpha ~~}}^{{ \to ~*}} = ~argmin\left\{ { - \mathop \sum \limits_{{i = 1}}^{n} \alpha _{i} + ~\mathop \sum \limits_{{i = 1}}^{n} \mathop \sum \limits_{{j = 1}}^{n} \alpha _{i} \alpha _{j} y_{i} y_{j} \left\langle {\overrightarrow {{x_{i} }} ,~~\overrightarrow {{x_{j} }} ~} \right\rangle } \right\}$$

(16)

where,

$$\mathop \sum \limits_{i = 1}^{n} \alpha_{i} y_{i} = 0; \quad 0 \le \alpha_{i} \le C$$

(17)

The classical SVM seems to be able to separate the linear dataset with a single hyperplane, which can separate two classes. For nonlinear dataset where more than two classes to be handled, kernel functions are used in that situation to lay out the data to a higher dimensional space in which it is linearly separable.

Sequential Minimal Optimization (SMO)

The algorithm Sequential Minimal Optimization (SMO) is employed as a learning algorithm to train Support Vector Machine with a linear kernel. It is highly efficient for solving the issue regarding quadratic programming (QP) problem which appears during the training of SVM. In this paper, we implemented SMO [18] by using data mining tools in python.

SMO is preferred because of its better scaling abilities for large and complicated SVM problems with less computational time than standard SVM. SMO solves the QP problem by decomposing the large problem into a sequence of small sub problems. Then SMO follows analytic QP steps to deal with smallest possible optimization problem.

The above QP problem in Eq. (17) will be solved by SMO as a smallest optimization problem, which consists of Lagrange multipliers $\alpha_{i}$. To get a definite solution of this QP problem, all the Lagrange multipliers should satisfy the Karush–Kuhn–Tucker (KKT) conditions.

Random forest (RF)

The Random forests are one of the most popular and widely used methods or framework for classification and regression problems. It has evolved as an ensemble learning approach based on multiple numbers of decision trees. According to the description of Wikipedia, Random Forest classifier operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. This classifier can solve the problem of over fitting by reducing the correlation between randomly selected trees and it helps in increasing the prediction power. The prediction for unseen samples can be done by averaging the predictions of the p number of trees. Random forest model performs well even when the feature size quite larger than number of samples. In case of high dimensional feature space RF classifier give poor accuracy, therefore to generate more accurate trees we applied some efficient feature selection method for dimensionality reduction. The basic procedure to build RF model with training dataset D is as follows.

Suppose, Training dataset ${\text{D = }}\left\{ {\left( {{\text{f}}_{\text{i}} ,{\text{ C}}_{\text{i}} } \right)_{{\text{i} = 1}}^{\text{N}} |{\text{ f}}_{\text{i}} \in {\text{R}}^{\text{F}} ,\;{\text{C}} \in \left\{ { 1, 2, \ldots ,{\text{c}}} \right\}} \right\}$ is given, where f_i are features, C_i is the set of classes and N denotes the number of training samples. Sample the training set D with replacement to create bagged samples D₁, D₂,…,D_p and each decision tree is grown from these bagged sample set. In each decision tree, for every node we consider a random and separate subset of predictive features as candidate feature for splitting the node. The class prediction of RF model with p number of trees can define as follows. Let assume $\hat{C}p$ be the prediction of tree T_p given input f.

$${\hat{C} = \text{majority}}\;\text{voting}\;{\{ \hat{C}}^{\text{p}} \}_{1}^{\text{p}}$$

(18)

In this research work, random forest (RF) classifier provides a striking precision in contrast to other classification model namely MNB, SVM, logistic regression, and GBM for medium length dataset. The F-score of RF classifier differ with size of different dataset in classification. The hyper parameters of RF classifier such as number of trees, number of features, and depth of trees plays a crucial role in maintaining the higher accuracy. The major characteristics of this algorithm are given below:

• This algorithm is easy to build and interpret.

• Classification model considered as robust and accurate.

• If some parameters are there they may be inserted easily, in such a way eliminating the requirement for pruning the trees.

Logistic regression

Logistic regression model consists of a set of classification rules extensively used for binary classification problem, to solve multiclass problem the model must be extended. This logistic function of this classifier utilizes to extract a set of weighted features from the input and estimates the correlation between the occurrence class, and extracted features. The researcher Allison [39] states that logistic regression becomes a suitable fit to the data by maximizing the log-likelihood function. Containing of all predictor into single model generally results poor predictions. The proper variable selection makes the model more accurate and generalized.

The probability of a feature vector i existing with positive class can be measured by logistic regression as given:

$$P\left( {{\text{c}} = 1|{\text{f}}} \right) = l(f) = \frac{1}{{1 + e^{{wT_{f} }} }}$$

(19)

where P(c = 1|f) refers to the probability of document ‘f’ of class ‘c’. ‘w’ indicates the feature-weight parameters to be estimated.

Experiments and results

Experimental settings

We applied Java SE (version 6) with Netbeans IDE (version 6.9) and Python with simple as well as efficient scikit-learn library to perform the experiments. We implemented standford POS tagger in java Netbeans IDE for POS tagging and remaining parts including tokenization, normalization, stop words removing are also performed by Java tools. Therefore, we can have a large feature space, from product review datasets [5], which are incorporated in python package. For feature selection and classification purpose, we used python tools. Specifically, we have utilized scikit-learn module with NumPy 1.8.1, SciPy 0.14.0 python library to enhance and extend the core python capabilities. We carried out experiments considering the tenfold (k = 10) cross validation, as the dataset we have considered for our experiment are comparatively smaller than other existing data sets. We separate the dataset into two portions, where (k − 1) 9-folds are used for training and 1-fold is used for testing.

Evaluation parameters

The performance of supervised ML algorithm can be evaluated based on the term or elements of confusion matrix on a set of test data. The confusion matrix consists of four terms are True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FP). According to the value of these elements, the evaluation matrices precision, recall, F-score and ROC are determined to estimate the performance score of any classifier.

$${\text{Precision }}\left( \pi \right):\frac{TP}{TP + FP}$$

(20)

$${\text{Recall }}\left( \rho \right):\frac{TP}{TP + FN}$$

(21)

$${\text{F-Score}}:\frac{2*Precision*Recall}{Precision + Recall}$$

(22)

ROC curve

In machine learning, AUC or ‘Area under the ROC Curve’ is most popular measurement metric to determine which of the model can predict the classes best. ROC curve (receiver operating characteristic curve) is a graphical plot showing the performance of a classification model at all thresholds by considering the parameters True positives (TF) on X-axis and True negatives on Y-axis. The AUC values are lies in between 0.5 and 1.0. For binary classification, the higher accuracy indicates the model performs best whereas the classifier with AUC value 1.0 is an excellent performer and AUC value 0.5 is consider as a bad performer.

Results and discussion

The following experimental results help in the study the effects of an individual as well as a combination of the different feature selection methods on the performance of the classifier. This result clearly exhibits how each the classifier behaves with different feature selection method. In this section, an in-depth investigation was carried out to measure the effectiveness of the proposed approach i.e., to compare the performance of four supervised classifiers SMO, MNB, RF and logistic regression based on the combination of the different feature selection method.

Firstly, we proposed the dataset with standard pre-processing method such as tokenizing, stop words removal, normalizing, stemming. Then we applied the feature selection method Information Gain (IG), Gini Index (GI), Chi square (CHI) to assign a score to each feature and three different feature subsets are generated based on the score. Next, we combined three different feature subsets by adopting statistical method UNION to select all features, INTERSECTION to select only common features and revised UNION to collect all top ranked features along with common features of three subsets. Thus a prominent feature vector by merging IG, CHI, GI feature subsets can be generated easily for classification. Finally, the classifiers SMO, MNB, RF and logistic regression machine learning classifier used individual feature subset as well as prominent feature vector for classifying the review document into either positive or negative.

Tables 5, 6, 7 displays the performance of machine learning methods SMO, MNB, RF and logistic regression with respect to different feature selection method and their combination. The method IG performed well in comparison with other FSMs.

Table 5 Results of the proposed model for movie review data set

Table 6 Results of the proposed model for electronics review data set

Table 7 Results of the proposed model for Kitchen ware review data set

The following table indicates that the combination of (IG, CHI, GI) produce better results than applying those method individually.

According to Table 5, the SMO classifier performs best for linear SVM (F-score 92.31) with combined (IG, CHI, GI) method. The result shows that feature selection method IG and GI also performed well with SMO classifier. The F-Score of SMO becomes 89.77 and 88.62 for IG and Gini Index, respectively. SMO achieves higher accuracy for large problems. With the classifier evaluation, the SMO performed better than MNB (86.38), RF (83.45) and logistic regression (86.88) classifiers for movie review dataset.

The NB performs surprisingly well for sentiment analysis in many previous studies. NB method is a simple and popular classification technique, although the conditional independence assumption is harsh. In our investigation, MNB is next best to SVM in performance. In all three datasets, MNB classifier equally provides a good result. In this paper, MNB performed great (90.53) especially with electronics product review data set.

As reported in Tables 5, 6, 7, the F score value obtained using combined methods (IG, CHI, GI) is comparatively better than that obtained using IG, CHI and GI method separately. If we consider all three domains, the combined method exhibits the better result in terms of F-score with more or less every classifier such as SVM, RF, LR and MNB. They chose the features based on their importance to the class level attribute.

The best performance of the logistic regression classifier is achieved with review data set of kitchen domain (88.47) when the combined (IG, CHI, GI) method is being used. Naïve Bayes classifier is quite convenient for small datasets as it is effortless to implement and very swift to train, but the classifier LR and SVM produce overall better performance for large dataset. In particular, unlike the NB classifier LR is capable of handling dependent features, while NB classifier is based on the independent assumption. In this case, LR underperformed SVM and MNB classifier for the datasets, such as: Movie, Electronics and outperformed RF classifier for most of the review datasets with combined feature selection method.

RF classifier got the maximum F score of 87.73 with Kitchen domain, when we consider the domain Movie and Electronics the F score for RF classifier reduced to 85.64 and 86.73, respectively.

In order to investigate the following figures, if we compare the classifier performance, SVM outperforms other classification methods MNB, RF and logistic regression for movie domain. MNB produce best result with Electronics dataset and the result obtained based on Kitchenware is the most favourable with RF classifier. According to the highest value of accuracy, recall and F-score value we estimate the results of three algorithms on testing dataset.

To see the impacts of different feature selection methods, we plot the ROC curve with highest AUC to represent the results of four different classification models for the aforementioned bench mark datasets of movie review and multi domain product review in Figs. 2, 3, 4. For readability, each graph presented four ROC curves for classifiers namely, SMO, MNB, LR and RF. According to the graph, we noticed all ROC curves incline to the upper left space in the graph. It specified that all the classifiers we have selected for this research work could reach both the maximum TPR as well as minimum FPR, as the point (0, 1) in the upper left corner of the roc space is also called a perfect classification which representing the true positive rate is 100% and the false positive rate is 0%.

Fig. 2

ROC curves obtained by classifiers for Movie review dataset

Fig. 3

ROC curves obtained by classifiers for Electronics review dataset

Fig. 4

ROC curves obtained by classifiers for Kitchen review dataset

We also calculated AUC and FI measure score in Figs. 5, 6, 7 where white column denoted to AUC and black for F1 measure. The highest AUC was 0.94 with SMO classifier for Movie review dataset. MNB and LR classifier obtained highest 0.93 with Electronics dataset and 0.92 with Kitchen review dataset, respectively. According to the results, we can predict that employing the selected machine learning classification models must have effectiveness for sentiment polarity detection.

Fig. 5

Comparing the resulting value of F1-score and AUC of classifiers for Movie review dataset

Fig. 6

Comparing the resulting value of F1-score and AUC of classifiers for Electronics review dataset

Fig. 7

Comparing the resulting value of F1-score and AUC of classifiers for Kitchen review dataset

Performance evaluation

This section compares between the accuracy of proposed approach with other existing approaches considered IMDb dataset. This comparison was carried out according to the accuracy value which these methods achieved. The adopted approach i.e., the combination of different feature selection methods produces a better result in comparison [40] with the result obtained by applying individual feature selection method in previous research approaches are shown in following tables.

The author [41] proposed to use classifier ensembles and lexicons for sentiment analysis of tweets automatically. They tried to compare between bag-of-words model and feature hashing technique regarding how they represent features. The result exhibits that classifier ensembles configured by SVM, MNB, RF, and logistic regression can enhance the classification accuracy in a huge amount. The highest accuracy they got is of 79.11 with ensemble classifier (Table 8).

Table 8 Comparisons of performance of proposed approach with different literature using different domain review Dataset

Kalaivani et al. [42] examined how classifier SVM, NB and k-NN works with different feature sizes of movie review dataset. Feature selection method Information Gain (IG) applied to select top p % ranked features to train the classifier. In this work, SVM approach outperformed the Navie Bayes and k-NN approaches with highest accuracy 81.71. The experimental result reported the precision and recall value for positive and negative corpus separately.

In [43], the investigation by Tripathy et al. employed machine ML classifiers namely NB, SVM, ME, SGD to perform sentiment classification of online movie reviews [36] with N-gram techniques. The performance evaluation can be done by the parameters such as precision, recall, F-measure, and accuracy. The results in comparing with our approach show that FSMs have a great impact on the classification performance. The feature ranking techniques (Information Gain, Chi Square, Gini Index method) improve classification performance over no feature selection.

Al-Moslmi et al. [44] studied on feature selection methods effects on machine learning approaches in Malay sentiment analysis. It was demonstrated that improved feature selections resulted in better performance in Malay sentiment-based classification. The author approached three feature selection methods (IG, Gini Index, and CHI) to enhance the performance of three machine learning classifiers (SVM, NB, and k-NN). A dataset of 2000 movie reviews are crawled from several web contents in Malay language. The results showed that the combination of SVM classifier and IG-base method established as the best classification algorithm, with an accuracy of 85.33% with feature sizes of 300. Authors have also reported that use of the FSMs yields improved results compared to those from the original classifier.

Conclusion

The aim of this paper is to explore the ability of a combination of three feature selection methods such as IG, Chi Square, Gini Index to enhance and refine the performance of four machine learning classifiers namely SMO, MNB, RF and LR on the multiple domains. The effectiveness of classification algorithm is evaluated in terms of F measure, precision, and recall. As discussed in before the combined feature sub list of IG, CHI and GI produce very convincing results. As we considered the dataset from multiple domains, thus the classifiers such as SMO, MNB and RF performed best for reviews of movie, electronics and kitchenware subsequently. These empirical experiments, exhibiting the proposed method, are highly effective and encouraging. The method we proposed in this work still has some drawbacks that are mentioned as follows:

• In some reviews or comments, user expressed their sentiment through some images or emoticons, but we have not considered these kinds of expressions for analysis.

• The comment in text format contains sarcasm, linguistic problems etc. To predict the sentiment of that comment we have to understand the nature and ambience of the comment thoroughly, as a single word can create a contradiction about the polarity of the comment. However, this aspect is also not considered in this paper.

All of above-mentioned downsides shall be considered for the future work to refine the quality of sentiment classification. We are also planning to merge the traditional machine learning method with deep learning techniques to tackle the challenge of sentiment prediction of massive amounts of unsupervised product review dataset in future.