awesome-spark/awesome-spark

Packages

General Purpose Libraries

Apache DataFu - A library of general purpose functions and UDF's.

Resources

Books

Learning Spark, 2nd Edition - Introduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts.

Packages

General Purpose Libraries

spark-dariastars618 - A Scala library with essential Spark functions and extensions to make you more productive.

quinnstars286 - A native PySpark implementation of spark-daria.

Packages

Middleware

Kyuubistars783 - Improved implementation of Thrift JDBC/ODBC Server.

Packages

Monitoring

Data Mechanics Delightstars183 - Cross-platform monitoring tool (Spark UI / Spark History Server replacement).

Packages

General Purpose Libraries

itachistars29 - A library that brings useful functions from modern database management systems to Apache Spark.

Packages

Storage

Delta Lakestars3.7k - Storage layer with ACID transactions.

Packages

Interfaces

Koalasstars3k - Pandas DataFrame API on top of Apache Spark.

Packages

Utilities

pyspark-stubsstars105 - Static type annotations for PySpark (obsolete since Spark 3.1. See SPARK-32681).

Packages

Language Bindings

Mobiusstars931 - C# bindings (Deprecated in favor of .NET for Apache Spark).

.NET for Apache Sparkstars1.7k - .NET bindings.

Packages

Utilities

Optimusstars1.1k - Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.

Resources

Papers

Large-Scale Intelligent Microservices - Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives.

Resources

Papers

[Structured Streaming: A Declarative API for Real-Time

Packages

Middleware

Livystars653 - REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.

Packages

Testing

deequstars1.9k - Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Packages

Notebooks and IDEs

almond - A scala kernel for Jupyter.

Polynote - Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Orginating from Netflix.

Packages

SQL Data Sources

Spark CSVstars1k - CSV reader and writer (obsolete since Spark 2.0 [SPARK-12833]).

Spark Avrostars540 - Apache Avro reader and writer (obselete since Spark 2.4 [SPARK-24768]).

Packages

Machine Learning Extension

Clustering4Everstars117 Scala and Spark API to benchmark and analyse clustering algorithms on any vectorization you can generate

Packages

Web Archives

Archives Unleashed Toolkitstars109 - Open-source toolkit for analyzing web archives.

Packages

Language Bindings

Flambostars609 - Clojure DSL.

sparklyrstars815 - An alternative R backend, using dplyr.

sparklestars428 - Haskell on Apache Spark.

Packages

Notebooks and IDEs

Apache Zeppelin - Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.

Spark Notebookstars3.1k - Scalable and stable Scala and Spark focused notebook bridging the gap between JVM and Data Scientists (incl. extendable, typesafe and reactive charts).

sparkmagicstars1.1k - Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livystars968, in Jupyter notebooks.

Packages

General Purpose Libraries

Succinct - Support for efficient queries on compressed data.

Packages

SQL Data Sources

Spark XMLstars386 - XML parser and writer.

Spark-Mongodbstars306 - MongoDB reader and writer.

Spark Cassandra Connectorstars1.8k - Cassandra support including data source and API and support for arbitrary queries.

Spark Riak Connectorstars54 - Riak TS & Riak KV connector.

Mongo-Sparkstars617 - Official MongoDB connector.

OrientDB-Sparkstars19 - Official OrientDB connector.

Packages

Bioinformatics

ADAMstars902 - Set of tools designed to analyse genomics data.

Hailstars752 - Genetic analysis framework.

Packages

GIS

Magellanstars511 - Geospatial analytics using Spark.

GeoSparkstars982 - Cluster computing system for processing large-scale spatial data.

Packages

Time Series Analytics

Spark-Timeseriesstars1.2k - Scala / Java / Python library for interacting with time series data on Apache Spark.

flintstars914 - A time series library for Apache Spark.

Packages

Graph Processing

Mazerunnerstars377 - Graph analytics platform on top of Neo4j and GraphX.

GraphFramesstars811 - Data frame based graph API.

neo4j-spark-connectorstars266 - Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support.

SparklingGraph - Library extending GraphX features with multiple functionalities useful in graph analytics (measures, generators, link prediction etc.).

Packages

Machine Learning Extension

dbscan-on-sparkstars163 - An Implementation of the DBSCAN clustering algorithm on top of Apache Spark by irvingc and based on the paper from He, Yaobin, et al. MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data.

Apache SystemML - Declarative machine learning framework on top of Spark.

Mahout Spark Bindings [status unknown] - linear algebra DSL and optimizer with R-like syntax.

spark-sklearnstars1.1k - Scikit-learn integration with distributed model training.

JPMML-Sparkstars91 - PMML transformer library for Spark ML.

Distributed Kerasstars618 - Distributed deep learning framework with PySpark and Keras.

ModelDB - A system to manage machine learning models for spark.ml and scikit-learn .

Sparkling Waterstars909 - H2O interoperability layer.

BigDLstars3.8k - Distributed Deep Learning library.

MLeapstars1.3k - Execution engine and serialization format which supports deployment of o.a.s.ml models without dependency on SparkSession.

Packages

Middleware

spark-jobserverstars2.7k - Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.

Miststars316 - Service for exposing Spark analytical jobs and machine learning models as realtime, batch or reactive web services.

Apache Toreestars684 - IPython protocol based middleware for interactive applications.

Packages

Utilities

silexstars17 - Collection of tools varying from ML extensions to additional RDD methods.

sparklystars47 - Helpers & syntactic sugar for PySpark.

Flintrockstars583 - A command-line tool for launching Spark clusters on EC2.

Packages

Natural Language Processing

spark-corenlpstars424 - DataFrame wrapper for Stanford CoreNLP.

spark-nlpstars2.4k - Natural language processing library built on top of Apache Spark ML.

Packages

Streaming

Apache Bahir - Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).

Packages

Interfaces

Apache Beam - Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.

Blazestars3k - Interface for querying larger than memory datasets using Pandas-like syntax. It supports both Spark DataFrames and RDDs.

Packages

Testing

spark-testing-basestars1.3k - Collection of base test classes.

spark-fast-testsstars297 - A lightweight and fast testing framework.

Packages

Workflow Management

Cromwellstars740 - Workflow management system with Spark backendstars740.

Resources

Papers

Spark SQL: Relational Data Processing in Spark - Paper introducing relational underpinnings, code generation and Catalyst optimizer.

Resources

Papers

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing - Paper introducing a core distributed memory abstraction.

Resources

Books

Advanced Analytics with Spark - Useful collection of Spark processing patterns. Accompanying GitHub repository: sryza/aasstars1.5k.

Mastering Apache Spark - Interesting compilation of notes by Jacek Laskowski. Focused on different aspects of Spark internals.

Spark Gotchasstars319 - Subjective compilation of tips, tricks and common programming mistakes.

Spark in Action - New book in the Manning's "in action" family with +400 pages. Starts gently, step-by-step and covers large number of topics. Free excerpt on how to setup Eclipse for Spark application development and how to bootstrap a new application using the provided Maven Archetype. You can find the accompanying GitHub repo herestars266.

Resources

MOOCS

Data Science and Engineering with Apache Spark (edX XSeries) - Series of five courses (Introduction to Apache Spark, Distributed Machine Learning with Apache Spark, Big Data Analysis with Apache Spark, Advanced Apache Spark for Data Science and Data Engineering, Advanced Distributed Machine Learning with Apache Spark) covering different aspects of software engineering and data science. Python oriented.

Resources

Workshops

AMP Camp - Periodical training event organized by the UC Berkeley AMPLab. A source of useful exercise and recorded workshops covering different tools from the Berkeley Data Analytics Stack.

Resources

Projects Using Spark

Oryx 2stars1.8k - Lambda architecture platform built on Apache Spark and Apache Kafka with specialization for real-time large scale machine learning.

Resources

Blogs

Spark Technology Center - Great source of highly diverse posts related to Spark ecosystem. From practical advices to Spark commiter profiles.

Resources

Miscellaneous

Spark with Scala Gitter channel - "A place to discuss and ask questions about using Scala for Spark programming" started by @deanwampler.

Resources

Miscellaneous

Apache Spark User List and Apache Spark Developers List - Mailing lists dedicated to usage questions and development topics respectively.

Resources

Projects Using Spark

Crossdatastars170 - Data integration platform with extended DataSource API and multi-user environment.

Resources

Docker Images

sequenceiq/docker-sparkstars769 - Yarn images from SequenceIQ.

Resources

Projects Using Spark

Photon MLstars778 - A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model.

Resources

Docker Images

jupyter/docker-stacks/pyspark-notebook - PySpark with Jupyter Notebook and Mesos client.

Resources

MOOCS

Big Data Analysis with Scala and Spark (Coursera) - Scala oriented introductory course. Part of Functional Programming in Scala Specialization.

Packages

Machine Learning Extension

KeystoneML - Type safe machine learning pipelines with RDDs.

Resources

Projects Using Spark

PredictionIO - Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time.