Using deep learning for short text understanding

As we move towards the micro blogging era, and with the ever-growing presence of digital news, delivering the relevant content to the right users based on the limited context information present in short texts has become more important than ever. For example, let’s take twitter, where the character limit per tweet is a mere 140 characters. Displaying ads relevant to a particular user requires understanding what type of tweet she/he generally produces. If the user generally tweets about sports, it is more appropriate if sports related ads are shown to their timeline. Other areas where short texts are used are web queries, news recommendations based on news titles, etc. These all require understanding short texts and delivering the right content. However, understanding short texts involves confronting many challenges that make the task difficult. Short texts like web queries and tweets generally lack proper syntactic formation. Unlike in longer documents, there is a dearth of enough statistical information in those short texts. Also, short texts with similar meaning may not share any common words between them. For example, lets take two short texts, “new version of windows OS” and “upcoming Operating System from Microsoft”. The given short texts don’t share any keywords between them, but we know that these two are related. These factors make understanding short text a challenging task. To identify the relatedness of these texts, we need to find common concepts between them.

Finding concepts of words from the given short text and identifying the words that occur frequently with those words are collectively represented as semantic enrichment. Mapping of words into their concepts is an important task in understanding natural language. As we can see, concepts are the medium which glue terms that are not related syntactically to a common ground. For example, the terms “apple” and “orange” are not related syntactically but both of those terms can be glued by the concept “fruit”. Hence, finding the most appropriate concept of any given word is the main task in conceptualization. There is a challenge that is very hard to solve in conceptualization, which is the context sensitive nature of word to concept mapping. For instance, we can see the context of the word “apple” is “fruit” or “food” when it is used with words like “orchard” or “tree”. But the same word “apple” has the context of “company” or “firm” when it is used with “ipad”, “iphone” or some other technological terms. There are many approaches proposed to solve or mitigate this problem. One of them does this with a simple two-stage solution combining a topic model and a probabilistic knowledge base such that we can consider both semantic relationships among words for modeling the context, and conceptual relationships between words and concepts for mapping the words to concepts within the given context [1].

Use of universal probabilistic semantic networks like Probase to enrich the text adds far more context and meaning, enabling for better classification. However, there are many cases when using just the noun terms of a short text to generate concepts is not sufficient to clearly identify the context of those texts. This is especially true for ambiguous short texts. Let us take for example two short texts: “five people hospitalized by eating poisonous apple” and “two people imprisoned for hacking into apple”. Here, using just the noun terms generates the same concepts for both the texts, as both of them contain the noun terms “people and apple” but the context of these two texts are completely different. The context of those texts will be clear if we consider the verbs “eating” and “hacking” from the respective texts. Let us look at another example, “United defeated by City 2-1”. In this text, if we take the noun term only we cannot figure out its context properly. But if we take into consideration the verb “defeated”, we may get the sense that it is talking about a sporting events. It can also be showed that the adjectives used in the text also help to disambiguate it for higher accuracy in classification.

Our contribution through this paper will be generating richer concepts for a short text, considering the verbs in addition to the nouns present in the text, hence adding more precise features to the text. These added features generated from verbs help to remove ambiguity if any, present in the text that could not be removed through the concepts generated by the noun terms only. We then present a simple four layers deep neural network which takes the enriched text as input and classify it into one of the categories.

We have long known that computers need huge amounts of common-sense and domain specific world knowledge to understand natural language [2, 3]. But the prior works on semantic relatedness were purely based on a statistical approach that did not use background knowledge [4, 5] or on lexical resources that incorporate very limited knowledge about the world [6, 7].

Then various approaches came for enriching short texts that used domain-specific information. One of the approaches is to treat the short text as a web query and enrich the given text with the results from the search engine to that query (e.g titles of web page and snippets) [8–10]. This method takes advantage of variety of web documents to identify the appropriate context. Another effective approach used external knowledge base such as WordNet [11, 12], Wikipedia [13, 14], the Open Directory Project (ODP) [15] etc. to enrich the given short text. As Gabrilovich [14] has demonstrated, there is value in using Wikipedia as an additional source of features for text categorization and determining the semantic relatedness between texts. The open editing policy of Wikipedia yields amazing quality. A recent study [16] found Wikipedia accuracy to rival that of Britannica. Substantial works have been conducted in automatic web query classification. One of them is web query classification using labeled and unlabeled training data [17]. Another approach for web query classification uses machine learning techniques in an automatic classifier to categorize queries by geographical locality [18]. Query type classification for web document retrieval is another related work [19].

Apart from enriching the given short texts, work has been done to extract efficient representations of those texts so as to perform information retrieval or classification. Recently, Salakhutdinov and Hinton [20] have proposed semantic hashing which is a novel information retrieval mechanism. In this model, we stack RBMs [21] which learns the document representation through a compact binary code. The semantic of a document is supposedly captured by the code.

The approaches mentioned above can produce satisfactory result to some extent, but they each have their own limitations. Treating a short text as search query may work well for popular queries but for unpopular queries, the search engine will most likely return some irrelevant results which acts as a noise to the short query instead of enriching it. Enriching a short text with the information obtained from WordNet also has its problems. Although WordNet contains information about general terms, it does not contain information for proper nouns such as “USA” or “Microsoft”. Also, it does not weigh the senses based on the frequency of their usage. There is another approach to semantic enrichment which uses Probase for enriching the given short text [22]. Probase [1, 23, 24] is a large collection of worldly concepts and their likely instances presented in probabilistic form. It contains about millions of such concepts and instances. The basic idea is to extract the noun terms from the given short text and perform a lookup into Probase to generate multiple concepts that relate to those noun terms. We can also extract co-occurring terms from Probase for those noun terms. Then, these concept terms and co-occurring terms [22] along with those noun terms form the enriched short text. This approach produces by far the best classification result over other previous works.

We start our discussion with (i) Probase, a huge network connecting concepts and instances semantically, in a probabilistic manner, (ii) VerbNet, a manually constructed verb list which represents the categories of short texts and (iii) backpropagation, a widely used method to find gradient descent which in turns effectively helps to train the neural network and (iv) Deep neural network, a neural network with multiple hidden layers which captures the abstract relationships among different features of data hierarchically and an auto-encoder, one of the unsupervised deep learning algorithm.

Backpropagation

It is the most widely used method of training artificial neural networks. Using backpropagation [25], one can effectively approximate a real, discrete or vector valued target functions [26]. It generally takes place in two stages. First is the propagation stage. When training examples are presented to the network, it propagates forward, with each units in every layers taking the weighted sum of inputs and calculating output based on some activation function, until an output is generated from the output layer. We, then compare the generated output with the actual output and find the error using some error function. Now, the assumption is this error is due to the partial contribution from each neurons in previous layers. So, the error is propagated backwards with each neurons associated with some error value. This completes the propagation stage.

These back propagated error values are used to compute the gradient descent [27] of the loss function. We apply some weight update rule based on the gradient descent of loss function to optimize the weight of the network and hence, train the network to learn the function required to describe the input examples.

In Fig. 1, we show a simple neural network with three layers. First layer is \(L_1\) which consists of four nodes. Second is the hidden layers with two nodes represented by \(L_2\) and last one is an output layer with four nodes each representing a component of output. The nodes in hidden layer and output layer can be activated by same function or different ones. We have shown two functions \(\alpha\) and \(\beta\) acting on these layers. If i is an arbitrary node in a certain layer and j is a node that connects its output to it, then we can represent the weight from jth node in previous layer to ith node in next layer by \(w_{ij}\).

If we take squared error as the error function, which is given as:

$$\begin{aligned} E=\frac{1}{2}(h-y)^2 \end{aligned}$$

where E is the squared error, h is the estimated output from the network, and y is the actual output.

For each neuron j in the network, its output is:

$$\begin{aligned} a_j=\alpha (z_j)=\alpha \left (\sum _{i=1}^{n}w_{ij}a_i \right) \end{aligned}$$

where \(\alpha\) is the differentiable and non-linear activation function. Sigmoid function is the commonly used such function.

$$\begin{aligned} \alpha (z)=\frac{1}{1+e^{-z}} \end{aligned}$$

We can apply following chain rule to compute the descent of error with respect to each weight of the network.

$$\begin{aligned} \frac{\partial E}{\partial w_{ij}}=\frac{\partial E}{\partial a_j}\frac{\partial a_j}{\partial z_j}\frac{\partial z_j}{\partial w_{jk}} \end{aligned}$$

Applying some calculus, we can find

$$\begin{aligned} \frac{\partial E}{\partial w_{ij}}=\delta _{j}a_i \end{aligned}$$

where, \(\delta _j=\frac{\partial E}{\partial a_j}\alpha^{\prime}(z_j).\)

To get enriched text \(X^E\) from input text X, it first goes through pre-processing, conceptualization and co occurrence phase.

Our proposed solution to the problem defined above is sketched in algorithmic view below:

Algorithm 1 outlines the pre-processing of the input text. The first step in pre-processing is to remove the punctuation from the input text. Then it extracts the noun terms and stores them in a list. Similarly, it extracts Verb terms from input text and stores them in a separate list. For each noun in Noun list, if PROBASE contains it then add those terms to a pre-processed list. It does the same for each of the verb terms, but it looks into VERBNET instead of PROBASE. This algorithm has the run time complexity of O(n) as each operation of removing punctuation and extracting noun and verb terms depends linearly on the size of the input.

Algorithm 2 takes the output from algorithm 1 as input and performs conceptualization. For each term in its input, algorithm 2 performs the look-up into PROBASE to find n relevant concepts for the term and adds those concepts along with the terms in the conceptualization list. Algorithm 2 also prunes the irrelevant concepts gathered from PROBASE. We talk about this in detail in the upcoming section. The run time complexity of this algorithm is O(n) as the loop runs for n times which is the size of input.

This algorithm finds the co-occurring terms of the output terms from the conceptualization algorithm. For each term in its input, it finds top m relevant co-occurring terms. Again, PROBASE is used to find the terms. These generated co-occurring terms are stored in a list along with the terms from previous algorithms. Some filtering of irrelevant terms is done which is described in the next section. It runs on O(n) as well.

Algorithm 4 is concerned with predicting the correct category of input short text. After an input text goes through all of the previous algorithms, it is represented in vector form, normalized and then fed to our deep neural network model. The model then returns the prediction for the input text.

Enrichment and classification flow

Understanding short text boils down to two major phases, the enrichment phase and the classification phase. The enrichment phase adds context to the original short text while the classification phase classifies the enriched text to a certain category. Hence, our proposed system contains mainly two modules: the enrichment module and classification module. First we describe the enrichment module, and then the classification module in a later sub-section:

Let X be the input short text which first goes through the enrichment phase.

Pre-processing

We extract noun and verb terms from X. So new representation of X is: \(X_p={x_1, x_2,x_3 ,\ldots, x_k}\), where \(x_i , 1\le \,i\le k\) is the extracted noun, or verb terms of X. While pre-processing the given input text, we consider the longest phrase that is contained in PROBASE. For example, in short text “New York Times Bestseller”? although New York is a noun term in itself, we take “New York Times”? as a single term. We need to remove stopwords and perform word stemming, before parsing such terms.

Conceptualization

We enrich \(X_P\) obtained from pre-processing by finding the concepts of all the xi in \(X_P\) with the help of probase and our manually built verb network which further adds news features to \(X_P\). It is just a lookup operation in the networks which generates the ranked list of concepts which relate to the \(x_i\) in \(X_P\). We denote the result of this phase by \(X_C\): \(X_C={x_1,x_2,x_3, . . . , x_m}.\)

Where \(x_i, 1\le i \le m\) are the union of terms from XP and the terms obtained from conceptualization and m > k. There are cases when irrelevant concepts of given terms can appear in this phase. For instance, given a short text “apple and microsoft”? a concept like “fruit”? may appear, which is irrelevant in this context. So we apply following mechanism to remove such ambiguity.

Mechanism In case of ambiguity of two or more concepts given the terms, we can employ a simple Naive Bayesian mechanism proposed by Song et al. [10]. It estimates the posterior probabilities of each concepts given the instances. As a result, most common concepts associated with the instances get higher posterior probability. The probability of a concept is given formally c as:

$$\begin{aligned} p(c|E)=\frac{p(E|c)p(c)}{p(E)} \end{aligned}$$

(1)

which is proportional to: \(p(c)\prod _{i}^{M}p(e_i|c).\)

Using this, given a short text “apple and microsoft”?, we can get the higher posterior probabilities for concepts like “corporation”, “IT firm” etc. whereas irrelevant concept like “fruit” gets lower probability and hence pruned.

Co-occurring terms

We take the result from previous phase and enrich it using the co-occurring terms. The result of this is the finally enriched version of the original short text which is given by:

$$\begin{aligned} X_E = {x_1,x_2,x_3, . . . , x_n} \end{aligned}$$

(2)

where \(x_i, 1 \le i \le n\) are the union of terms from \(X_C\) and the terms obtained from this phase and n > m.

Co-occurrences give the important information to understand the meaning of words. These are the words that frequently co-occur with the original terms. We know from Distributional hypothesis [29] “words that occur in the same context tend to have similar meanings”.

To find the co-occurring words, we take the terms obtained from the pre-processing phase, and enrich with terms that frequently co-occur with these terms. For each original term, we look into probase to find the co-occurring terms.

DNN model

We designed a four layer deep neural network as the learning element for our input text. It was trained in semi-supervised fashion as we discussed earlier. It is as shown in Fig. 2.

Each layers except the input layer shown in the figure, are pre-trained with an auto-encoder. This is done in layer by layer fashion. To pre-train the first hidden layer, we train an auto-encoder with input layer and hidden layer same as in Fig. 2. The output layer tries to reconstruct the original input. Once this is done, we pre-train second hidden layer with separate auto-encoder. This auto-encoder has the previously trained hidden layer as its input and tries to generate code for its input. All subsequent layers are pre-trained in same manner. We used sigmoid as activation function and cross-entropy error as defined above for loss function. Through this pre-training, the parameters of our network achieve some nearly optimal value.

The pre-training phase alone can not optimize the network parameters completely although they can attain some good values. So we perform supervised fine-tuning to our pre-trained network. Fine-tuning process is as shown in Fig. 3.

We feed our DNN model with a feature vector that represents our short text through input layer. To correctly predict the category, we add softmax layer to our pre-trained network. Output \(o_i\) for any node i is computed as:

$$\begin{aligned} o_i=\frac{exp(a_i)}{\sum _{m=1}^{n}exp(a_m)}, \end{aligned}$$

where n is the total nodes on softmax layer. Output from this network is the prediction class for the given input vector. As shown above, our DNN model goes through two training stages. First is unsupervised pre-training and then the supervised fine-tuning [30].

For short text understanding, we experimented with the task of classifying news titles into categories. We used news titles as the short text and classified them into specific categories.

Result

Method	Enriched method	Input representation	Accuracy (%)
OS-SVM	Not enriched	3000D semantic feature vectors	29.00
CACT-SVM	CACT	3000D semantic feature vectors	47.52
WordNet-SVM	WordNet-based	3000D semantic feature vectors	36.15
Wiki-SVM	Wikipedia-based	3000D semantic feature vectors	39.06
EDNN-SVM	CACT	Learned 128D binary code vectors	51.35
WordNet-SH-SVM	WordNet-based	Learned 128D binary code vectors	40.21
Wiki-SH-SVM	Wikipedia-based	Learned 128D binary code vectors	42.76

Table 1 shows the classification accuracy of various methods applied to “Wikipedia Short Sentences” dataset as stated in [22]. We wanted to see the effect of adding verb terms in overall accuracy. So we re-implemented the EDNN-SVM done in [22] and tested with our dataset. In our work, we call EDNN-SVM method applied to our dataset as EDNN-N as it uses just the noun terms for enrichment. The accuracy of this experiment was 58.02%.

After that, we included the verb terms along with noun terms and noun phrases in feature vocabulary as well as feature vector and conducted the experiment to see if inclusion of verb terms can remove the ambiguity (if any) and improve the accuracy of the result. We call this EDNN-NV model. The accuracy obtained was 60.25%. So there was about 2% increase in accuracy after considering verb terms present in a given short text. Figure 4 shows the accuracy of each category predicted by our model EDNN-NV. We have a lowest accuracy for World News category which is 33.33%. Similarly, we have highest accuracy for Football News category which is 74.91%. The other five categories have accuracy in between these percentages. Figure 5 compares the results of EDNN-N and our EDNN-NV model

Our research tackled the problem of classifying short texts. The challenge of short texts are lack of enough context and the ambiguity that may arise due to it. So, we implemented the conceptualization and co-occurring terms enrichment technique in our work, for a given short text and presented a novel idea of including verb terms in text enrichment. First, we computed the sorted list of verbs occurring in the dataset and filtered the list with the verbs that somewhat represent the categories we had considered. Then, we inserted those filtered verbs into the previously computed feature vocabulary. This way, the representative verbs become the feature for incoming short text. We also presented a simple four layers deep neural network to classify the input short text.

Hence, our research shows that albeit low, the inclusion of verb terms in addition to noun terms helps to improve the classification of short text. Although the size of VerbNet is relatively smaller and just contains the representative verb in the dataset, it can be scaled up by mining the web. Through this work, we just wanted to show that verb terms can help improve the short text classification. This paves a way for considering other parts of speech like Adjectives to be included for classification.