awesome-spark/awesome-spark
Big Data 5 months ago 1.3k
A curated list of awesome Apache Spark packages and resources.
Dec 27th - Jan 2nd, 2021
Packages
GIS
- Cluster computing system for processing large-scale spatial data.
- Machine learning orchestration platform.Dec 6th - Dec 12th, 2021
Resources
Docker Images
Packages
Storage
- Integration with the lakeFS atomic versioned storage layer.Packages
General Purpose Libraries
- joblib backend for running tasks on Spark clusters.Nov 29th - Dec 5th, 2021
Packages
Notebooks and IDEs
- Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from Netflix.Packages
SQL Data Sources
- CSV reader and writer (obsolete since Spark 2.0 [SPARK-12833]).Packages
Machine Learning Extension
Scala and Spark API to benchmark and analyse clustering algorithms on any vectorization you can generate.
- A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment.Packages
Natural Language Processing
- Natural language processing library built on top of Apache Spark ML.Resources
Papers
Nov 1st - Nov 7th, 2021
Packages
Middleware
- A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark.
- A library of general purpose functions and UDF's.Aug 9th - Aug 15th, 2021
Resources
Books
Packages
General Purpose Libraries
- A Scala library with essential Spark functions and extensions to make you more productive.
- A native PySpark implementation of spark-daria.Mar 15th - Mar 21st, 2021
Packages
Monitoring
- Cross-platform monitoring tool (Spark UI / Spark History Server replacement).
- A library that brings useful functions from modern database management systems to Apache Spark.
- Storage layer with ACID transactions.
- Pandas DataFrame API on top of Apache Spark.Sep 28th - Oct 4th, 2020
Packages
Utilities
- Static type annotations for PySpark (obsolete since Spark 3.1. See SPARK-32681).Packages
Language Bindings
- C# bindings (Deprecated in favor of .NET for Apache Spark).
- .NET bindings.Sep 21st - Sep 27th, 2020
Packages
Utilities
- Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.Resources
Papers
- REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.
- Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- A scala kernel for Aug 19th - Aug 25th, 2019
Packages
SQL Data Sources
- Apache Avro reader and writer (obselete since Spark 2.4 [SPARK-24768]).Jan 21st - Jan 27th, 2019
Packages
Web Archives
- Open-source toolkit for analyzing web archives.Aug 13th - Aug 19th, 2018
Packages
Language Bindings
- Clojure DSL.
- An alternative R backend, using dplyr.
- Haskell on Apache Spark.Packages
Notebooks and IDEs
- Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
- Scalable and stable Scala and Spark focused notebook bridging the gap between JVM and Data Scientists (incl. extendable, typesafe and reactive charts).
- Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livystars980, in Jupyter notebooks.Packages
General Purpose Libraries
- Support for efficient queries on compressed data.Packages
SQL Data Sources
- XML parser and writer.
- Cassandra support including data source and API and support for arbitrary queries.
- Riak TS & Riak KV connector.
- Official MongoDB connector.
- Official OrientDB connector.Packages
Bioinformatics
- Set of tools designed to analyse genomics data.
- Genetic analysis framework.Packages
GIS
- Geospatial analytics using Spark.Packages
Time Series Analytics
- Scala / Java / Python library for interacting with time series data on Apache Spark.
- A time series library for Apache Spark.Packages
Graph Processing
- Graph analytics platform on top of Neo4j and GraphX.
- Data frame based graph API.
- Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support.
- Library extending GraphX features with multiple functionalities useful in graph analytics (measures, generators, link prediction etc.).Packages
Machine Learning Extension
- An Implementation of the DBSCAN clustering algorithm on top of Apache Spark by irvingc and based on the paper from He, Yaobin, et al. MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data.
- Declarative machine learning framework on top of Spark.
- Scikit-learn integration with distributed model training.
- PMML transformer library for Spark ML.
- Distributed deep learning framework with PySpark and Keras.
- A system to manage machine learning models for spark.ml and scikit-learn 
.
- H2O interoperability layer.
- Distributed Deep Learning library.
- Execution engine and serialization format which supports deployment of o.a.s.ml models without dependency on SparkSession.Packages
Middleware
- Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.
- Service for exposing Spark analytical jobs and machine learning models as realtime, batch or reactive web services.
- IPython protocol based middleware for interactive applications.Packages
Utilities
- Collection of tools varying from ML extensions to additional RDD methods.
- Helpers & syntactic sugar for PySpark.
- A command-line tool for launching Spark clusters on EC2.Packages
Natural Language Processing
- DataFrame wrapper for Stanford CoreNLP.Packages
Streaming
- Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).Packages
Interfaces
- Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.
- Interface for querying larger than memory datasets using Pandas-like syntax. It supports both Spark DataFrames and RDDs.Packages
Testing
- Collection of base test classes.
- A lightweight and fast testing framework.Packages
Workflow Management
- Workflow management system with Spark backendstars801.Apr 10th - Apr 16th, 2017
Resources
Papers
Apr 3rd - Apr 9th, 2017
Resources
Papers
Feb 27th - Mar 5th, 2017
Resources
Books
Resources
MOOCS
Resources
Workshops
Resources
Projects Using Spark
Resources
Miscellaneous
Oct 17th - Oct 23rd, 2016
Resources
Miscellaneous
Jun 20th - Jun 26th, 2016
Resources
Projects Using Spark
Resources
Docker Images
Jun 6th - Jun 12th, 2016
Resources
MOOCS
Packages
Machine Learning Extension
Feb 1st - Feb 7th, 2016
Resources
Projects Using Spark
Last Checked At: 2022-05-28T15:53:51.215Z
