Packages / Language Bindings
- spark-connect-csharp (⭐1)
- C# bindings.
Packages / General Purpose Libraries
- spark-daria (⭐751)
- A Scala library with essential Spark functions and extensions to make you more productive.
- quinn (⭐632)
- A native PySpark implementation of spark-daria.
Packages / Storage
- Apache Hudi (⭐5.4k)
- Upserts, Deletes And Incremental Processing on Big Data..
- Apache Iceberg (⭐6.4k)
- Upserts, Deletes And Incremental Processing on Big Data..
Packages / Data quality
- python-deequ (⭐717)
- Python API for Deequ.
Packages / Testing
- spark-fast-tests (⭐432)
- A lightweight and fast testing framework.
- chispa (⭐606)
- PySpark test helpers with beautiful error messages.
- Rust bindings.
- Golang bindings.
- A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark.
- Kotlin API bindings and extensions.
- Cluster computing system for processing large-scale spatial data.
- Machine learning orchestration platform.
- 
- Integration with the lakeFS atomic versioned storage layer.
- Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from 
- A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment.
- Natural language processing library built on top of Apache Spark ML.
- A library of general purpose functions and UDF's.
- Cross-platform monitoring tool (Spark UI / Spark History Server replacement).
- A library that brings useful functions from modern database management systems to Apache Spark.
- Storage layer with ACID transactions.
- Pandas DataFrame API on top of Apache Spark.
- .NET bindings.
- Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.
- REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.
- Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- A scala kernel for 
- Open-source toolkit for analyzing web archives.
- An alternative R backend, using 
- Haskell on Apache Spark.
- Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
- 
- XML parser and writer.
- Cassandra support including data source and API and support for arbitrary queries.
- Official MongoDB connector.
- Set of tools designed to analyse genomics data.
- Genetic analysis framework.
- Data frame based graph API.
- Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support.
- Declarative machine learning framework on top of Spark.
- PMML transformer library for Spark ML.
- A system to manage machine learning models for 
.
- 
- Distributed Deep Learning library.
- Execution engine and serialization format which supports deployment of 
- Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.
- IPython protocol based middleware for interactive applications.
- Helpers & syntactic sugar for PySpark.
- A command-line tool for launching Spark clusters on EC2.
- Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).
- Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.
- Collection of base test classes.
- Workflow management system with