manuzhang/awesome-streaming

Big Data 3 months ago 2k

a curated list of awesome streaming frameworks, applications, etc

View byDAY/WEEK/README

View on Github

Mar 16th

Website

Data Pipeline

Redpanda stars4.4k [C++] - Redpanda is Kafka compatible, ZooKeeper-free, JVM-free and source available.

Feb 9th

Website

Toolkit

Nussknacker stars296 [Scala] - A visual tool to define and run real-time decision algorithms.

Feb 7th

Website

Data Pipeline

Apache RocketMQ stars17.5k [Java] - distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Feb 1st

Website

Streaming Library

Akka Streams stars12.3k [Scala] - stream processing library on Akka Actors.

Daggy stars130 [C++] - real-time streams aggregation and catching.

Oct 30th, 2021

Website

Readings

Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter

Oct 6th, 2021

Website

Streaming Library

YoMo stars1.1k [Go] - An open source Streaming Serverless Framework for building Low-latency Geo-distributed system. YoMo Built atop QUIC Transport Protocol and Functional Reactive Programming interface.

Oct 4th, 2021

Website

Data Pipeline

fluvio stars1.1k [Rust/WASM] - Real-time programmable data streaming platform with in-line computation capabilities.

Oct 2nd, 2021

Website

Readings

Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Reuven Lax, Slava Chernyak, and Tyler Akidau

Sep 22nd, 2021

Website

Streaming Engine

Scramjet Transform Hub stars40 [JavaScript/Node.js] - data processing engine for running multiple data processing apps (sequences) written in JavaScript or TypeScript

Website

Streaming Library

Scramjet Framework - functional reactive stream programming framework written on top of Node.js object streams.

May 7th, 2021

Website

Streaming Engine

Apache Ballista [Rust] - distributed compute platform powered by Apache Arrow.

Website

Data Pipeline

StreamSets Data Collector stars43 [Java] - continuous big data ingestion infrastructure that reads from and writes to a large number of end-points, including S3, JDBC, Hadoop, Kafka, Cassandra and many others.

Website

Streaming SQL

Siddhi stars1.3k [Java] - A cloud native Streaming and Complex Event Processing engine that understands Streaming SQL queries in order to capture events from diverse data sources, process them, detect complex conditions, and publish output to various endpoints in real time.

Apr 1st, 2021

Website

Streaming SQL

Materialize [Rust] - A source-available streaming SQL engine for maintaining materialized views on data from message brokers and databases.

ksqlDB stars5k [Java] - A cloud-native, source-available database purpose-built for stream processing applications

Feb 21st, 2021

Website

Streaming Engine

Maki Nage stars26 [Python] - A stream processing framework for data scientists, based on Kafka and ReactiveX.

Aug 12th, 2020

Website

Streaming Library

Tributary stars380 [Python] - A python library for constructing dataflow graphs. Supports synchronous, reactive data streams built using python generators that mimic complex event processors, as well as lazily-evaluated acyclic graphs and functional currying streams.

Jun 12th, 2020

Website

Readings

Grokking Streaming Systems by Josh Fischer & Ning Wang

May 7th, 2020

Website

Data Pipeline

RudderStack stars3.1k [Go] - an open source customer data infrastructure (segment, mparticle alternative).

Apr 30th, 2020

Website

Data Pipeline

Gazette stars324 [golang] - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.

Dec 30th, 2019

Website

IoT

Apache StreamPipes stars318 [Java] - a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

Dec 10th, 2019

Website

Streaming Engine

Apache Heron (incubating)stars3.6k [Java] - a realtime, distributed, fault-tolerant stream processing engine from Twitter.

Oct 26th, 2019

Website

Streaming Engine

mantis stars1.2k [Java] - Netflix's platform to build an ecosystem of realtime stream processing applications

Oct 25th, 2019

Website

Data Pipeline

LogDevice [C++] - a high-performant distributed system by Facebook for streaming and storing sequential data, using a log structure.

Oct 10th, 2019

Website

DSL

Apache Beam stars5.6k [Java, Python, SQL, Scala, Go] - unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs), open sourced by Google.

Website

Closed Source

Cloud Dataflow[Java, Python, SQL, Scala] - Google's managed stream and batch data processing engine. Supports running Beam pipelines.

Sep 8th, 2019

Website

Streaming Library

Stream Ops stars44 [Java] - A fully embeddable data streaming engine and stream processing API for Java.

Aug 27th, 2019

Website

Streaming Engine

Gearpump stars759 [Scala] - lightweight real-time distributed streaming engine built on Akka.

Aug 11th, 2019

Website

Streaming Library

Streamz stars1.1k [Python] - A lightweight library for building pipelines to manage continuous streams of data; supports complex pipelines that involve branching, joining, flow control, feedback, back pressure, and so on.

Jul 23rd, 2019

Website

Online Machine Learning

streamDM stars474 [Scala] - mining Big Data streams using Spark Streaming from Huawei.

StormCV stars165 [Java] - enables the use of Apache Storm for video processing by adding computer vision (CV) specific operations and data model.

trident-ml stars385 [Java] - realtime online machine learning library based on Trident.

yurita stars104 [Scala] - Anomaly detection framework built on Spark Structured Streaming from Paypal.

Jul 22nd, 2019

Website

Data Pipeline

brooklin stars776 [Java] - a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale from Linkedin (replaced databus).

Jul 14th, 2019

Website

Streaming SQL

StreamCQL stars0 [Java] - Continuous Query Language on RealTime Computation System.

Website

Closed Source

Amazon Kinesis Streams [Java] - real-time, fully managed and scalable data stream engine provided by AWS.

Azure Stream Analytics [.NET] a massively scalable, fully managed, real-time, data stream engine provided by Microsoft Azure.

concord [C++] - a distributed stream processing framework built in C++ on top of Apache.

IBM Streams [Python/Java/Scala] - platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box.

jubatus [C++] - distributed processing framework and streaming machine learning library.

millwheel - framework for building low-latency data-processing applications that is widely used at Google.

Apr 10th, 2019

Website

Streaming Engine

Apache Apex stars347 [Java] - unified platform for big data stream and batch processing.

Apache Flink stars19.2k [Java] - system for high-throughput, low-latency data stream processing that supports stateful computation, data-driven windowing semantics and iterative stream processing.

Apache Samza stars742 [Scala/Java] - distributed stream processing framework that build on Kafka(messaging, storage) and YARN(fault tolerance, processor isolation, security and resource management).

Apache Spark Streaming stars33.2k [Scala] - makes it easy to build scalable fault-tolerant streaming applications.

Apache Storm stars6.4k [Clojure/Java] - distributed real-time computation system. Storm is to stream processing what Hadoop is to batch processing.

AthenaX stars1.2k [Java] - Uber's Stream Analytics Framework used in production

Faust stars6.2k [Python] - stream processing library, porting the ideas from Kafka Streams to Python

Hazelcast Jet stars995 [Java] - A general purpose distributed data processing engine, built on top of Hazelcast.

hailstorm stars88 [Haskell] - distributed stream processing with exactly-once semantics based on Storm.

mupd8(muppet)stars127 [Scala/Java] - mapReduce-style framework for processing fast/streaming data.

Onyx stars2k [Clojure] - Distributed, masterless, high performance, fault tolerant data processing.

s4 stars41 [Java] - general-purpose, distributed, scalable, fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

SABER stars39 [Java/C] - Window-Based Hybrid CPU/GPU Stream Processing Engine.

SPQR stars29 [Java] - dynamic framework for processing high volumn data streams through pipelines.

tigon stars278 [C++/Java] - high throughput real-time streaming processing framework built on Hadoop and HBase.

Teknek stars8 [Java] - Simple elegant stream processing with interactive prototying shell SOL (Stream Operator Language)

Website

Streaming Library

Apache Kafka Streams stars22.3k [Java] - lightweight stream processing library included in Apache Kafka (since 0.10 version).

Benthos stars4.5k [Go] - Benthos is a high performance and resilient message streaming service, able to connect various sources and sinks and perform arbitrary actions, transformations and filters on payloads

FS2(prev. 'Scalaz-Stream')stars2.1k [Scala] - Compositional, streaming I/O library for Scala.

monix stars1.8k [Scala] - high-performance Scala / Scala.js library for composing asynchronous and event-based programs.

Streamline stars155 [Java] - Stream Analytics Framework by Hortonworks, designed as a wrapper around existing streaming solutions like Storm. Aimed to allow users to drag-and-drop streaming components to focus on business logic.

StreamAlert stars2.7k [Python] - Airbnb's Real-time Data Analysis and Alerting.

Swave stars174 [Scala] - A lightweight Reactive Streams Infrastructure Toolkit for Scala.

Website

Streaming Application

straw stars100 [Python/Java] - A platform for real-time streaming search.

storm-crawler stars750 [Java] - Web crawler SDK based on Apache Storm.

Website

IoT

sensorbee stars212 [Go] - lightweight stream processing engine for IoT.

Apache Edgent stars205 [Java] - a programming model and runtime that enables continuous streaming analytics on gateways and edge devices which can work with centralized systems to provide efficient and timely analytics across the whole IoT ecosystem: from the center to the edge, opens sourced by IBM.

Website

DSL

coast stars59 [Scala] - a DSL that builds DAGs on top of Samza and provides exactly-once semantics.

Esper stars741 [Java] - component for complex event processing (CEP) and event series analysis.

Streamparse stars1.5k [Python] - lets you run Python code against real-time streams of data via Apache Storm.

summingbird stars2.1k [Scala] - library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including Storm and Scalding.

Website

Data Pipeline

Apache Kafka stars22.3k [Scala/Java] - distributed, partitioned, replicated commit log service, which provides the functionality of a messaging system, but with a unique design.

Apache Pulsar stars11.1k [Java] - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

camus stars883 [Java] - Linkedin's Kafka -> HDFS pipeline.

databus stars3.4k [Java] - Linkedin's source-agnostic distributed change data capture system.

flume stars2.3k [Java] - distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

metaq stars1.3k [Java] - Taobao's high available, high performance distributed messaging system

NATS streaming stars2.4k [Go] - fast disk-backed messaging solution

nsq stars21.5k [Go] - realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.

suro stars770 [Java] - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data.

Website

Online Machine Learning

Apache Samoa stars240 [Java] - distributed streaming machine learning (ML) framework that contains a programing abstraction for distributed streaming ML algorithms.

DataSketches stars778 [Java] - sketches library from Yahoo!.

StreamingBandit stars77 [Python] - Provides a webserver to quickly setup and evaluate possible solutions to contextual multi-armed bandit (cMAB) problems.

Website

Streaming SQL

pipelinedb stars2.5k [C] - An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.

squall stars267 [Java] - Squall executes SQL queries on top of Storm for doing online processing.

Website

Benchmark

storm-perf-test stars77 [Java] - a simple storm performance/stress test.

streaming-benchmarks stars579 [Java] - Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, etc.

flotilla stars233 [Go] - Automated message queue orchestration for scaled-up benchmarking.

Website

Toolkit

akka stars12.3k [Scala] - toolkit and runtime for building highly concurrent, distributed, and resilient message-driven application on the JVM.

pulsar stars1.9k [Python] - Actor based event driven concurrent framework for Python.

aeron stars6.1k [Java/C++] - efficient reliable unicast and multicast message transport.

StreamFlow stars242 [Java] - stream processing tool designed to help build and monitor processing workflows.

samza-luwak stars100 [Java] - uses Luwak, a stored-query engine built on Lucene, to implement full-text search on streams.

Turbine stars824 [Java] - tool for aggregating streams of Server-Sent Event (SSE) JSON data into a single stream.

Jun 13th, 2016

Website

Readings

Real Time Analytics: Algorithms and Systems (VLDB 2015)

Feb 13th, 2016

Website

Readings

In-Stream Big Data Processing

The world beyond batch: Streaming 101 by Tyler Akidau.

Last Checked At: 2022-06-24T17:52:23.539Z

igorbarinov/awesome-data-engineering

awesome-spark/awesome-spark

Track Awesome List

// Track it, not just star it.

Top 50 Awesome List

Recently Updated

Platforms

Programming Languages

Front-End Development

Back-End Development

Computer Science

Big Data

Theory

Books

Editors

Gaming

Development Environment

Entertainment

Databases

Media

Learn

Security

Content Management Systems

Hardware

Business

Work

Networking

Decentralized Systems

Higher Education

Events

Testing

Miscellaneous

Unmaintained

Top 50 Awesome List

Recently Updated

Platforms

Programming Languages

Front-End Development

Back-End Development

Computer Science

Big Data

Theory

Books

Editors

Gaming

Development Environment

Entertainment

Databases

Media

Learn

Security

Content Management Systems

Hardware

Business

Work

Networking

Decentralized Systems

Higher Education

Events

Testing

Miscellaneous

Unmaintained

manuzhang/awesome-streaming

Website

Data Pipeline

Feb 9th

Website

Toolkit

Feb 7th

Website

Data Pipeline

Feb 1st

Website

Streaming Library

Oct 30th, 2021

Website

Readings

Oct 6th, 2021

Website

Streaming Library

Oct 4th, 2021

Website