Table 1 Major feature of big data analysis frameworks
From: A novel approach for big data processing using message passing interface based on memory mapping
Framework | Abstraction | Supported languages | Underlying engine | Availability |
|---|---|---|---|---|
Hadoop | MapReduce | Java, Python | Hadoop | Open source |
Spark | RDD | Scala, Python, Java, R | Spark, Yarn | Open source |
H2O | Procedural + Library | REST, R, Python | H2O | Open source |
SciDB | Declarative | SQL | SciDB | Open source |
AzureML | Visual User Interface | REST | Microsoft Azure | Microsoft |
R | Procedural | R | R | Open source |
SparkR | Procedural | R | Spark | Open source |
Mahout | Toolkit | Java, Scala | Hadoop, Spark | Open source |
Spark MLib | Library | Scala, Python | Spark | Open source |
Samsara | Declarative | Java, Scala | Spark, Flink, H2O | Open source |
SystemML | Declarative | R, Python | Spark | Open source |
Google ML | Visual User Interface | Python | Google Cloud Dataflow | |
Amazon ML | Visual User Interface | N/A | Apache MXNet, TensorFlow, PyTorch | Amazon |
BigML | Visual User Interface | Python | BigML Predict Server | BigML |
Tensorflow | Visual User Interface | Python, Haskell, Java, Go, Julia, R | CUDA, TPU | Open source |
KeystoneML | Procedural + Library | Scala | Spark | Open source |