HCudaBLAST: an implementation of BLAST on Hadoop and Cuda

The DNA or protein sequence searching is the most obvious operation in the analysis of any new sequence and the reason for the same is pretty simple—finding similar regions of nucleotides or proteins between two or more nucleotide or protein sequences. The similarity can be used to determine many things including similarity of two or more species, identifying a completely new species, locating domains within the sequence of interest, etc. However, the difficulty in finding the similar regions between two or more sequences is very hard due to the size of the existing sequence involved. To overcome this difficulty, various tools or algorithms have been proposed. Let us look at some of these in the following paragraphs.

In computational biology and bioinformatics, aligning sequences to determine similarity between them is an essential and widely used computational procedure for biological sequences. There have been wide range of computational algorithms applied to the sequence alignment challenge. Methods like Smith–Waterman algorithm [1], which is quite slow but accurate and is based on dynamic programming, and, basic local alignment search tool (BLAST) [2] or FASTA [3] algorithm which is faster but less accurate and is based on heuristic or probabilistic programming. The very first algorithm was given by Smith and Waterman in the form of Smith–Waterman algorithm in 1981. This is a global sequential alignment algorithm which involves high time complexity but at the same time, it gives optimal results. To overcome the time consumption of Smith–Waterman algorithm, Lipman and Pearson proposed FASTA tool in 1985, which takes a given nucleotide or amino acid sequence and searches a corresponding sequence database by using local sequence alignment. It is based on heuristic method which contributes to the high speed of its execution. It is also a heuristic algorithm like FASTA but it is more time-efficient as it searches only for the significant patterns in the sequences with comparative sensitivity.

Due to the heuristic approach, the execution speed of BLAST is significantly increased but the amount of data being processed is very large and is increasing at an exponential rate; such as UniMES for metagenomic data sets [4], which continues to expand exponentially as next generation sequencing (NGS). However, the solution for the scalability problem of the BLAST has been solved by combining Hadoop and BLAST together, which is called HBLAST [5]. It is a hybrid “virtual partitioning” approach that automatically adjusts the database partition size depending on the Hadoop cluster size as well as the number of input query sequences. Yet another improvement in the speed of BLAST algorithm has been proposed in the form of GPU-BLAST [6] or other CUDA based implementations [7]. GPU-BLAST can perform protein alignments up to four times faster than the single-threaded NCBI-BLAST. When GPU-BLAST is compared with six-threaded NCBI-BLAST then it performs nearly two times faster.

Hadoop is an open source framework developed by Apache and it is a platform that can store and manage large volumes of data and it also provides a relatively easy way to write and run applications for massive data processing. Hadoop basically works in two folds, the first fold is Hadoop distributed file system (HDFS) [8], which is a good fault-tolerant system for storing data using low-cost hardwares. The second fold is MapReduce [9], which is a programming model to process the stored data by running tasks with efficient scheduling. Hadoop works in the fashion of Master–Slave architecture and it comprises many daemon processes like NameNode, DataNode, NodeManager, ResourceManager, etcetera. HDFS comprises of a NameNode and numerous DataNodes, wherein the NameNode is in charge of administration of metadata, file blocks and namespace of HDFS-stored files and the DataNodes are in charge of physically storing and managing the data on the block level.

Hadoop provides scalability and gives efficient handling of existing database of sequences and GPU provides time efficiency for the whole process. It is intuitive to merge these two prominent technologies to gain the full strength of existing technologies and provide better results in terms of time and storage. So this article presents HCudaBLAST, which is a combination of Hadoop and CUDA processing that runs the existing NCBI BLAST algorithm.

The rest of the article is organized as follows. In "Background and related work" section, we introduce related work available in this topic. In "Proposed work" section, we give the points of interest of our proposed work and algorithm of HCudaBLAST, and in "Experimental setup" section, we give description of our experimental setup, followed by conclusion in "Conclusion" section.

As discussed, there are many variants of BLAST algorithm such as blastn, blastp, blastx, tblastx, etcetera. Here, the focus is only on implementation of blastp, which is used for searching protein sequences in an existing protein database. The blastp is already implemented with GPU in the aforementioned article and it reduces the searching time complexity. It is also already implemented with MapReduce in HBLAST and it also reduces the time consumption as well as it gives scalability to the storage of database. Here, presented an integration of these two solutions into one—an implementation of blastp algorithm with Hadoop and CUDA to take advantage of both the solutions.

The working of HCudaBLAST is exceptionally straightforward. There is a mapping file as an input which consists of name of the query file and the name of the partition files of the database in each line with comma separation. This mapping file will be given to running job and it will be divided according to the number of mappers available. Each such mapper takes the part of the file as an input and processes each line at a time. In each iteration of the mapper, it will process the query file with given database partition files and generate all the seeds for the query with the word length of 3. Then the mapper will pass it to the process running on the GPU to calculate final alignments and collect the result generated by GPU process and then pass the query Id as a key & score with subject Id as a value to the reducer process.

The GPU process will basically take all the subject sequences, the query sequence, the generated seeds and the score matching matrix Blosum62 as inputs. It will then give each subject to a different thread and each thread will align the query sequence with subject sequence for each seed available. It will also extend the alignment of the seed till the score cut-off and expected cut-off are not met. This pair of the seed of query sequence and the part of the subject sequence with their score is called high scoring segment pairs (HSPs). By distributing tasks on all the processor available on GPU, it will parallelize the calculation of the HSPs and after calculating all the HSPs, it will return these HSPs to the mapper. After running the mapper process, Hadoop will do sorting and shuffling on the output generated by the mapper and all results will be merged according to the query Id, so the reducer will get the query Id as a key and lists of subject Ids and their scores as values. The reducer will collect all the score belonging to the same subject & the same query Id and calculate the sum of all those scores. It then gives query Id and subject Id as a key and their sum of score as a value, which is finally stored in an output file.

The tool used to partition the database is “makeblastdb”. The benefit of partitioning the database is that it will properly distribute on a cluster and it will easily fit in an available memory. The block size available on Hadoop is 128 MB, that is why the maximum partition size of the database should be 128 MB, so that the time consumed by disk I/O could be reduced. The searching will be done for all the partitions, which are on a single node in a single mapper iteration. The formula to calculate expected score for limiting the alignment is \({\text{eScore}} = K*m*n*\mathbf {{e}^{-\text{s}*\uplambda }},\) where n is the length of the query sequence, m is the length of the subject sequence, K and λ are statistical parameters estimated by fitting the distribution of local alignment scores and s is the current score. The algorithm for HCudaBLAST is appeared in Fig. 1.

The evaluation of the HCudaBLAST algorithm is done on a cluster of machines, setup with Hadoop framework and each such machine has Nvidia CUDA capability in it. Two machines in this cluster have Intel Xeon E5-2630 processor and 6 cores with 8 GB RAM and these machines have two CUDA devices, first one is Tesla C2075 with 14 multiprocessors and computing capability of 2.0 and second one is Quadro 2000 with four multiprocessor and computing capability of 2.1. Another three machines have Intel Xeon E31245 processor and 4 cores with 8 GB RARM and these machines have one CUDA device Quadro 600 with computing capability of 2.1. All the physical machines in a cluster was connected with a single switch in a local LAN. The cluster created here running Hadoop version 2.7 and CUDA toolkit 7.5 on each machine. Here, one machine act as a master node which runs NameNode and ResourceManager on it and all other four machines run DataNode and NodeManager on it. All the code run on Hadoop is written in Java programming language and the native code running on GPUs written in C, which called by Java using Native Interface (JNI).

The database comprises of approximate six million sequences of proteins and it has been taken from the NCBI web site [13]. The different size of query sequences have also been taken from same web site to perform the searches. This experiment has been performed in twofolds—one with two million sequences and the other with 6 million sequences with different size of queries and different number of partitions of the database to properly analyze the time taken by the algorithm.

Tables 1 and 2 show the results given by this experiment and the time taken by both Hadoop implementation of BLAST and HCudaBLAST. Obviously, HCudaBLAST shows a significant improvement in terms of time taken with different number of subject sequences and different size of query sequence.