Improving deep neural network design with new text data representations

Many current application domains of machine learning and artificial intelligence involve knowledge discovery from text, such as sentiment analysis, document ontology and spam detection. Humans have years of experience and training with language, enabling them to understand complicated, nuanced text passages with relative ease. Text classifiers attempt to emulate or replicate this knowledge so that computers can discriminate between concepts encountered in text. Learning high-level concepts from text, such as those found in many applications of text classification, is a difficult task due to the many challenges associated with text mining and classification. A common approach to this machine learning task is to find a way to represent text, such as human-engineered features extracted from the text, then using this representation with training data to train a classifier. In the current text classification paradigm, human researchers create an approach for extracting features from text by looking at words, phrases, parts of speech, other morphological features, employing a lexicon, and mapping relations between words to determine concept families and semantic similarities. These different types of features may be used separately or combined into a single feature space in an attempt to build a representation capturing the complexity of language. Unfortunately, these approaches are often language or even task dependant [1] and may contain less information than the original text. Thus, this approach may yield feature engineering that is only applicable to a narrow range of text classification tasks.

One of the most commonly used method of feature engineering in text is the bag-of-word approach. A word vector is constructed from the data by parsing the data (text documents), identifying the words used throughout all documents and making a vector with entries representing each word, then ascribing values to this vector for each instance or document using word presence of frequency. While this approach has been demonstrated to be effective, it is far from ideal. Word placement within a sentence is not preserved and contextual indicators that may change the meaning of a word are absent from the final feature space. Additionally, a very large number of words can be identified with many being infrequent or unique to a single document leading to a high dimensional, sparse feature space. This increases computational costs and can degrade performance due to overfitting. While high dimensionality may be addressed with feature selection techniques [2], this approach to text understanding eliminates the majority of information that humans use when reading and writing text.

More complicated approaches have been devised and combined (or used in place of) the bag-of-words approach, such as grouping words by semantic concepts [3], the addition of part of speech tags [4], detecting grammatical patterns [5], and word ontology driven natural language processing (NLP) [6]; however, these feature spaces still contain less information than the original text as the full structure and context of the text is not preserved. Additionally, as features may be designed with key domain knowledge, more complicated feature engineering methodologies result in a text understanding solution that is less versatile and may be restricted to a single text domain. While additional information from the text can be turned into features, the burden of engineering features from text still falls upon the human researcher. A researcher implements a feature engineering methodology to extract features they believe will be useful; however, this is ultimately an educated guess. The resulting feature space is an incomplete representation of the text and may be missing valuable information that may not be easily identifiable to the researcher.

Deep neural networks provide an alternative approach for text mining tasks and feature extraction. High-level features can be learned automatically, allowing for the removal of human bias in feature engineering and the preservation of more information as the original data can be used for training. As features are learned as part of the training process of a deep neural network, researchers are not required to provide any specialized domain knowledge to the network making this family of approaches language and task independent. Instead, large volumes (potentially petabytes) of data are leveraged to train through repeated example.

Convolutional neural networks (CNN) are a family of neural networks that have been shown to be among the best solutions for training computers in tasks of computer vision. Krizhevsky et al. [7] developed a deep CNN that outperformed all previously existing approaches for image object recognition on the ImageNet dataset, due to its ability to detect high-level, abstract features for image detection. Additionally, they have recently been demonstrated to be able to learn high-level text concepts from character-level representations of text [1] in a manner similar to how they learn features from and can classify images. CNNs do not need any prior knowledge of the data to train a classifier as complex features can be learned automatically when training the network. This makes them well suited to text classification, since they do not need any prior knowledge of language. Starting from raw, character-level text data, abstract language concepts, including words, syntax, grammar and semantic similarities, are learned automatically. Zhang et al. [1] demonstrated using deep CNNs with character-level data representations of text outperforms approaches using higher level human generated text representations such as bag-of-words. As the network is trained from raw text no information is lost in constructing features. Also, since the network can be trained from character-level text representations, data dimensionality is less of an issue as there are less commonly used characters compared to words.

In this paper, we investigate the use of our new embedding for character-level representation of text classification tasks for use with CNNs and network design considerations due to the adoption of this embedding approach. First, we explain our new character embedding approach and demonstrate that it greatly reduces memory use and network training time due to greatly reducing the size of the initial input received by the network. We show that it outperforms the previous character embedding for the task of binary tweet sentiment classification, i.e. determining if tweets convey positive or negative sentiment. We also show that our character embedding can employ a larger alphabet at little to no additional cost, further enhancing performance, as training time scales logarithmically with alphabet size instead of linearly. Furthermore, we also explore network design implications due to using our new embedding. Namely, our embedding results in matrix representations of text instances where one dimension is much greater than the other. In this scenario, convolutional layers with padding are required to allow deeper networks to be trained.

This paper demonstrates the effectiveness of a new character embedding designed for training CNNs and explores how neural network design may be impacted by its adoption. To the best of our knowledge, text classification using character-level representations and deep neural networks has been previously investigated by only one research group [1], and we are the first to propose a more compact character embedding and to investigate its implications on neural network design. We show that our character embedding greatly reduces computational costs and training time, and improves classification performance. Additionally, we also show that using padded convolutional layers allows our embedding to be used with networks of arbitrary depth and the use of padding does not negate the benefits of our character embedding. Thus, our proposed character embedding can be adapted to any big data domain where high-level understanding of text is required, such as sentiment analysis, webpage ontology and topic classification.

The remainder of this paper is organized as follows. “Related works” section provides an overview of related work in text mining and deep learning. “Character-level text representations” section describes our newly created embedding for character-level representation. “Convolutional neural network design” section describes how convolutional neural networks work and design considerations that must be made on account of our new embedding. “General methodology” section provides details on the experimental methodology used to train and evaluate networks. “Experimental results” section presents results for our embedding and the use of padded layers. Finally, conclusions and future work are contained in “Conclusion” section.

Deep learning has been used for many text mining tasks including various non-classification natural language processing tasks, aiding in feature extraction prior to applying a separate machine learning algorithm, and end-to-end classification on word-level text representations. More recently, it has been shown that learning from character-level text representations may allow better performing classifiers to be trained [1]. The remainder of this section provides details on applications and current state of deep learning in text mining.

One of deep learning’s most popular applications in text mining is to perform semantic indexing of text. In semantic indexing, relationships between words and/or phrases are found and a condensed feature space of abstracted data representations can be generated. As an example of this, Google’s Word2vec provides an automated means of extracting semantic representations from big data. Word2vec uses a large-scale text corpus as input and produces vector representations of words by constructing a vocabulary and learning how to describe words outside the vocabulary from those inside it [8]. Word2vec has been combined with big data tools to produce a publicy available dictionary from a 100 million word text corpus from Google News [9] and provides a 300-dimensional vector representation of each word. However, bag-of-words has been shown to outperform Word2vec, in some instances such as Amazon review polarity and Yahoo answer topic prediction, despite its simplicity and high dimensionality [1].

Long short term memory (LSTM) networks are a popular deep neural network design for learning tasks with sequential data. This makes them a logical choice for text classification tasks as the data can be represented as a sequence of of words of characters. Using a LSTM network has been demonstrated to be effective for a wide range of text classification tasks, including sentiment, and can outperform traditional learners such as SVM [10]. While LSTMs can produce models with high performance, they are also slow to train and prone to over-fitting [11]. Due to their sensitivity to network architecture changes and hyper-parameter tuning, training a LSTM network can be considerably more challenging than using a CNN. Additionally, LSTMs are slower to train than CNNs as there is no equivalent to NVIDIA’s CuDNN library [12] to greatly accelerate the training of LSTMs.

CNNs have been used to perform a variety of natural language processing tasks, including part-of-speech tagging, named entity extraction, identification of semantic roles, and linking semantically similar words [13]. More recently, CNNs have been used for end-to-end discriminative text classification tasks involving the identification of high-level concepts prevalent throughout an entire document; however, most implementations are built on top of existing feature engineering and extraction methodologies. Kim [14], demonstrated training a CNN on top of static, pre-made word vectors performs well for sentence classification tasks. Additionally, a network with a single convolutional layer performed better than any traditional learner, such as Multinomial Naïve Bayes and support vector machine (SVM), on their benchmarking datasets. CNNs have also been used to aid feature extraction from text. Poria et al. [15] built higher level features from textual data, represented by a 306 dimensional vector consisting of a word vector and part of speech values, by training a CNN. They used output of the penultimate fully connected layer to create features to use with other classifiers and found the features produced by the network allowed better classifiers to be trained and using a different learner for classification, such as SVM, performed better than relying on the CNN’s final output layer for classification.

Recently, Zhang and LeCun [1] proposed a novel approach for text learning tasks where they were able to use CNNs to train classifiers by representing text in an image like, character-level fashion. This enabled them to train a deep convolutional neural network for text classification tasks involving high-level concepts from scratch with no prior feature engineering or extraction. They accomplished this by employing 1-of-m embedding, where each character is represented by a vector of size m, where m is the number of characters in their alphabet. Each character in a text instance was represented as a character vector and the instance as a sequence of character vectors. When visualized, a braille like output is generated. Using this embedding, they fed their data into a network with six 1D temporal convolutional layers and three fully connected layers. Their results showed training a deep CNN from character-level data outperformed networks trained on data with features generated with bag-of-words, bag-of-centroids and the deep learning approach Word2Vec.

We build on the work of Zhang and LeCun [1] by providing a new character embedding methodology for text classification with CNNs, demonstrate that our new character embedding greatly reduces training time, memory use and improves classification performance. Additionally, we demonstrate any restrictions to network design imposed by this embedding approach can be alleviated through the use of padded convolutional layers.

While no feature extraction needs to be performed, as we are teaching our CNNs to read from scratch, character-level data must still be represented in a manner that is acceptable as input for our networks. For this purpose we propose a new character embedding offering greatly reduced memory consumption and training time. We benchmark our approach against the embedding used by Zhang and LeCun [1]. We also investigate using different sized character alphabets.

Zhang and LeCun use character-level text representations to train CNNs by employing 1-of-m embedding. With this embedding, each character is represented as an m sized vector where all values are zero except the entry corresponding to the position in the alphabet of the found character (which has a value of 1). Each instance is then represented by a sequence of m sized vectors with length l, where m is the alphabet size and l is the length of the character sequences. The sequence length, l, is a pre-defined constant which is chosen to suit the specific classification task as it is determined by the document character length. The algorithm for this embedding is presented in Algorithm 1 .Text encoded in this fashion generated a sparsely populated matrix, as seen by our visualizations of 1-of-m embedding in Fig. 1a, b. In these figures, each character is a column and the full image is created by binding the sequence of characters together to create an m × l image. Zhang and LeCun used an alphabet of 70 characters (m = 70). This alphabet includes all 26 lowercase English letters, 10 digits, the new line command and 33 other characters [1].

Characters not contained in this alphabet are represented as a vector of zeroes. This embedding turns each character into a vector of length 70 with one non-zero entry (one black pixel in the column). For example, “a” would be 1000...0 (a 1 followed by 69 0s). Zhang and LeCun note sequences of characters encoded in this manner are visually similar to braille.

We also chose to use the larger UTF-8 alphabet¹ which contains 256 distinct characters (m = 256). Using this alphabet preserves far more information from our raw text data than the 70 character alphabet of Zhang and LeCun. Using the UTF-8 alphabet allows us to distinguish between lowercase and uppercase letters, something that is not done with the 70 character alphabet. This is advantageous since capitalization has been found to be beneficial for the detection of sentiment in tweets [5].

Our proposed embedding is inspired by 1-of-m embedding, but achieves a much smaller, denser representation reducing memory use, network input layer size and training time. The premise behind our representation is that instead of having a single non-zero value in our character embedding vector, we can have multiple non-zero values. Our new approach functions as follows. Each character in our chosen alphabet is given an integer value. Characters in our instances are then replaced by the corresponding integer. We then find the equivalent binary representation of a character’s integer value, then turn this sequence into a vector of 0s and 1s. The exact embedding process for each instance is provided in Algorithm 2. Using this approach, an alphabet with n bits can be represented with n-dimensional character vectors. Thus, as the 256 unicode characters found in UTF-8 can be represented as an 8-bit integer, we can use a 8-dimensional vector to represent all characters in the UTF-8 alphabet. For example, within the UTF-8 alphabet, the letter “a” has an integer value of 97. This is equal to the binary representation “01100001” which we then turn into the 8-dimensional vector (0,1,1,0,0,0,0,1). Using 1-of-m embedding with the UTF-8 alphabet each character vector would be 256-dimensional.

Our new embedding can also be visualized and doing so creates braille-like results; however, it is far denser than Zhang and LeCun’s 1-of-m embedding and its vertical dimension is far smaller, as seen in Fig. 1. Applying our approach to any alphabet size, such as the 70 used by Zhang and Chen, reduces character vector size from m to \(log_2(m)\). Thus, using 256 characters requires a vector of size 8, while using 70 requires a vector of size 7. From this, we see that greatly increasing alphabet size has minimal impact on the size of character vectors using our embedding approach. As it reduces vector size from m to \(log_2(m)\) we choose to call it log-m embedding. In our experiments comparing 1-of-m embedding and log-m embedding, we chose alphabets of 70 and 256 characters and denote the four approaches as log-70, log-256, 1-of-256 and 1-of-70 for the remainder of the paper.

Notably, the resulting vector for each character with this embedding is 1/16th the size of the one produced by 1-of-m embedding for the UTF-8 alphabet since it only requires a vector of length \(log_2(m)\) instead of length m. Thus, the resulting matrix representations of each instance contain fewer values and requires less memory. Additionally, as the input layer of a CNN must match the dimension of instances from the input data a smaller input layer with less neurons can be employed. Thus, using log-m embedding results in a network containing far less total neurons, for the same depth and layer parameters, as 1-of-m embedding, greatly reducing training costs.

Convolutional neural networks consist of sparsely connected convolutional layers followed by fully connected dense layers (these dense layers are equivalent to a multilayer perceptron neural network). Convolutional layers are sparsely connected. Neurons in these layers are connected to a small region of the previous layer known as the receptive field, instead of the entire previous layer as is found in a dense layer. The most common receptor field sizes are small, such as \(3 \times 3\) or \(5 \times 5\) neurons. By being sparsely connected CNNs view and learn local correlations. The convolutional layers in the network are followed by several densely connected layers with the last layer containing one neuron for each possible classification outcome.

Several key hyper parameters control the behavior of the convolutional layers and number of neurons in the final network including number of filters, stride and padding. Neurons in each layer learn a weight vector to create a feature map. These weights are determined by applying numerous filters, which have a small receptive field size and are convolved across the dimensions of the data to create an activation map of the filter. Multiple filters can be applied so that more features can be extracted by connecting multiple neurons to each region of the previous layer. Stride is the distance between receptive fields on the input volume. Padding refers to adding a border of zeroes around the input volume. Typically, an nxn instance would be reduced in dimension by the application of a convolutional layer when output to the subsequent layer. For example, a \(3 \times 3\) convolution would reduce an \(n \times n\) instance to \((n-2) \times (n-2)\). This occurs because the convolution can not be moved to the extreme edges of the instance as it would have null inputs. Adding padding can eliminate this size reduction as it allows filters to be applied to the edges of the original input volume. Training iterations (epochs) employs gradient decent to learn network parameters. As each convolutional layer is a view of the neurons of the previous layer, stacking layers allows a larger region of the initial input to be viewable by neurons in deeper layers. For example two \(3 \times 3\) layers will result in a network that has \(5 \times 5\) views as each layer transforms \(3 \times 3\) views to a single output neuron.

For our study, we created two sets of CNNs. The first set consists of three networks designed for comparison of our different data representations. Three networks were required as we have three very different input dimensions, \(8 \times l\), \(70 \times l\), and \(256 \times l\), where l is the maximum instance length, and three networks were designed in an investigation of how our new embedding might impact network design. In our first set of networks, each network consists of three convolutional layers followed by three fully connected layers. Our convolutional layers used a \(3 \times 3\) receptive fields (filter size), with a stride of 1 and no padding, followed by three fully connected layers to match the networks used by Zhang and Lecun [1]. We used 32 filters based on preliminary experimentation.

When using 1-of-m embedding, a \(2 \times 2\) max pooling layer is placed after every convolutional layer (3 in total), while only a single max pooling layer is included in the network for log-m embedding. Due to having an input volume of \(8 \times l\), less max pooling layers could be used with for log-m embedding. Since each max pooling layer halves both dimensions of the input for the next layer and stacking three \(3 \times 3\) convolutional layers creates an effective receptive field of \(7 \times 7\), max pooling could not be performed on the \(8 \times l\) input volume generated by log-m embedding between each convolutional layer. Only one max pooling layer could be used in this network and it is placed between the convolutional and fully connected layers. When using log-70 embedding, we padded the character vectors with an extra zero to make them 8 dimensional instead of 7 so that the same network architecture could be used for both log-m approaches.

The first two fully-connected layers consist of 512 neurons. Each is followed by a dropout layers, with \(p = 0.5\), that helps avoid overfitting [18]. Since we are performing binary sentiment classification, the final fully-connected layer contains two neurons using the softmax logistic function [19]. We use binary cross entropy [20] to create our loss function to evaluate the network using gradient descent after each epoch. Parameters and architecture for our networks are provided in Table 1.

As noted above, when using log-m embedding for our two alphabet sizes, we have an input volume of \(8 \times l\). Without padding, each convolutional layer reduces the dimension by 2. Thus, after three layers we are looking at an \((8-3*2) \times (l-3*2)\) or \(2 \times (l-6)\) volume. Since we have a dimension of 2 in one direction, no further convolutional layers with \(3 \times 3\) filters can be applied. This sets a limit of three for the maximum number of \(3 \times 3\) convolutional layers that can be used with this embedding. By using padding, this limit can be removed. Since padding adds a border of zeroes around our input volume to prevent its dimension from being reduced by each convolutional layer, we no longer have a limit to the number of layers employed. Thus, we can design a network similar to AlexNet [7], one of the highest performing network architectures for image classification. This architecture consists of stacking three \(3 \times 3\) convolutional layers, followed by a \(2 \times 2\) max pooling layer. Again, our small dimension (8) imposes a limit on the number of layer stacks we can have due to max pooling. After our first max pooling layer, we have a \(4 \times (l/2)\) volume, after the second we have a \(2 \times (l/4)\) volume and no further \(3 \times 3\) convolutions can be performed. Thus, for the deepest network, we can construct an architecture consisting of three convolutional layers followed by a max pooling layer, three more convolutional layers, a second max pooling layer, then the fully connected layers.

Using additional convolutional layers should improve classifier performance as higher-level features will be extracted. As we are curious about the benefits of using padded layers to build deeper networks on data where one dimension of input volume is small, comparing a six layer network against a three layer network is unfair. Thus, we designed two networks, one using padding and one with no padding, that are otherwise equal. Each network consists of three \(3 \times 3\) convolutonal layers, 128 filters on each layer, followed by a \(2 \times 2\) max pooling layer, then two fully connected layers with 1024 neurons and a final output layer of two neurons. Parameters for the convolutional layers are provided in Table 1.

This section provides details on our data, system environment and experimental design for evaluating our character embedding and the impact of padding in convolutional layers.

Convolutional Neural Networks have been shown to be effective for text mining tasks including feature extraction and classification, and recently have been used to enable a classifier to be trained from character-level text data due to their ability to automatically identify and extract high-level concepts and features from text. Training from character-level data has the advantage of requiring no feature engineering or pre-training. Additionally, this approach has been shown to outperform training networks from data represented by previously extracted features [1]. In this study, we devised a new method of character embedding to facilitate faster training of deep convolutional neural networks from character-level representations of text data, discuss how it changes network architecture and investigate using padding to building deep networks when using this type of character embedding.

We present and demonstrate our new character embedding and show that it performs better than previous approaches. In our first set of experiments, we train networks using our new embedding approach and a previously defined method of embedding, 1-of-m, and measured the accuracy and training time of each approach. The results demonstrate our new character embedding approach, denoted as log-m, greatly reduces training time by 4.85 to 5.75× and reduces memory use by up to a factor of 16. Furthermore, using log-m embedding resulted in significantly better classification performance compared to 1-of-m embedding.

When comparing padded vs. non-padded convolutional layers, we found little difference in classification performance; however, using padding to enable creation of deeper networks increased training time by 2.72×, and using padding while doubling the number of convolutional layers increased training time by 3.26×.

Though padding increases training time, it allowed us to build a network with six convolutional layers resulting in a 2.2–2.4% increase in classification accuracy. We observed using six convolutional layers required less than 20% more time to train than the padded CNN with three layers. Additionally, the reduction in training time when using log-m embedding more than offsets the cost of padding. Thus, the combination of log-m embedding with padded convolutional layers allows networks using popular deep CNN architectures, such as AlexNet, to be trained with higher classification accuracy in less time than 1-of-m embedding. Additionally, these results should apply to more recent networks architectures, such as the inception model [27], as the use of padding enables arbitrarily deep networks to be constructed from any input layer size.

While our experiments were conducted on tweet sentiment data, our approach can be applied to all text classification tasks. 1-of-m encoding has already been demonstrated to work in multiple text classification domains [1]. Future work should continue to investigate network design when using log-m embedding and performance of on longer text documents should be investigated. Also, while padding allows deep networks to be constructed with a variety of architectures, it does increase training time. Mixing padded and non-padded layers may help improve training time. Alternatively, non-square convolutional filters and max pooling layers could be implemented to allow deeper networks to be trained without relying on padding and should be investigated. Experiments should be extended to include text in other languages, since our proposed embedding should be far more efficient than 1-of-m embedding or word embeddings when using text with thousands of unique characters, such as Chinese.