Table of Contents / Engines and Platforms

• Clink (⭐5) - Embedded-first, Arrow-native stream processing engine with streaming SQL and exactly-once checkpoints.

Table of Contents / Data Integration and Pipelines

• OLake (⭐1.5k) - Replicates Postgres, MySQL, MongoDB, Oracle, SQL Server and Kafka into Apache Iceberg with CDC, full refresh and incremental sync.

Table of Contents / Libraries, SDKs, and Programming Models

• PySAD (⭐296) - Python library for anomaly detection on streaming data, with online detectors, stream simulators, and evaluators.

Table of Contents / Data Integration and Pipelines

• StreamPulse (⭐0) - Real-time Kafka telemetry ingestion and stream processing pipeline with Redis sliding-window aggregations and live dashboard.

Table of Contents / Engines and Platforms

• SQLFlow (⭐802) - Stream processing engine that runs DuckDB SQL over Kafka, WebSocket, and webhook streams.

Table of Contents / Data Integration and Pipelines

• yasdb (⭐4) - Durable Streams protocol server backed by SlateDB object storage, with offset-based catch-up reads and SSE or long-poll tailing.

Table of Contents / Engines and Platforms

• Aeron (⭐8.9k) - Reliable UDP unicast, multicast, and IPC message transport.

• Apache Heron (⭐3.6k) - Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.

• Apache Pulsar (⭐15k) - Distributed pub-sub messaging and event streaming platform.

• Apache StreamPipes (⭐751) - Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.

• AthenaX (⭐1.2k) - Uber's retired SQL-based streaming analytics platform.

• eKuiper (⭐1.7k) - Lightweight data stream processing engine for resource-constrained IoT edge devices.

• Fluvio (⭐5.3k) - Composable, stateful data streaming system with programmable in-line computation.

• LaminarDB (⭐49) - Embeddable streaming SQL engine built on Apache Arrow and DataFusion.

• LogDevice (⭐1.9k) - Facebook's archived distributed storage system for sequential data.

• Mantis (⭐1.5k) - Netflix platform for building real-time, operations-focused stream processing applications.

• Metaq (⭐1.3k) - Taobao distributed messaging system.

• mupd8 (Muppet) (⭐128) - MapReduce-style framework for processing fast-moving data streams.

• NATS Streaming (⭐2.5k) - Deprecated disk-backed messaging system superseded by NATS JetStream.

• NSQ (⭐26k) - Real-time distributed messaging platform designed to operate at scale.

• PipelineDB (⭐2.7k) - Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.

• S4 (⭐43) - Retired distributed platform for processing continuous unbounded data streams.

• SensorBee (⭐231) - Discontinued lightweight stream processing engine for IoT.

• Squall (⭐274) - Online SQL query processing engine built on Apache Storm.

• Tigon (⭐284) - High-throughput real-time stream processing framework built on Hadoop and HBase.

• Trill (⭐1.3k) - Single-node query processor for temporal and streaming data from Microsoft Research.

• Wally (⭐1.5k) - Distributed stream processing framework formerly named Wallaroo.

Table of Contents / Libraries, SDKs, and Programming Models

• Akka (⭐13k) - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.

• Akka Streams (⭐13k) - Reactive Streams implementation built on Akka actors.

• Apache Edgent (⭐222) - Retired programming model and runtime for streaming analytics on gateways and edge devices.

• Apache Pekko (⭐1.6k) - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.

• Apache SAMOA (⭐251) - Retired distributed streaming machine learning framework.

• Apache StormCrawler (⭐1k) - Scalable web crawler SDK based on Apache Storm.

• DataSketches (⭐957) - Apache library of stochastic streaming algorithms known as sketches.

• FastStream (⭐5.4k) - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.

• FS2 (⭐2.5k) - Compositional streaming I/O library for Scala.

• Jubatus (⭐706) - Distributed processing framework and library for online machine learning.

• MediaPipe (⭐37k) - Cross-platform, customizable ML solutions for live and streaming media.

• Monix (⭐1.9k) - High-performance Scala and Scala.js library for asynchronous and event-based programs.

• Numalogic (⭐172) - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.

• Pulsar (⭐1.8k) - Actor-based event-driven concurrency framework for Python.

• Scramjet TypeScript (⭐41) - Reactive stream programming framework for asynchronous JavaScript and TypeScript applications.

• Streamiz.Kafka.Net (⭐544) - .NET stream processing library for Apache Kafka.

• Streamparse (⭐1.5k) - Python API, command-line tools, and topology DSL for Apache Storm.

• Summingbird (⭐2.1k) - Retired library for expressing streaming MapReduce computations over Storm and Scalding.

• Tributary (⭐466) - Python library for constructing synchronous, reactive, and lazy dataflow graphs.

• WindFlow (⭐89) - Data stream processing parallel library for multi-core CPUs and GPUs.

• Yurita (⭐109) - PayPal anomaly detection framework built on Spark Structured Streaming.

Table of Contents / Data Integration and Pipelines

• Apache Flume (⭐2.6k) - Distributed service for collecting, aggregating, and moving large amounts of log-like data.

• Brooklin (⭐970) - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.

• Camus (⭐877) - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.

• Databus (⭐3.7k) - LinkedIn source-agnostic distributed change data capture system.

• faucet-stream (⭐16) - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.

• inGestr (⭐4k) - Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.

• Redpanda Connect (⭐8.8k) - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.

• Suro (⭐798) - Netflix data pipeline for collecting, aggregating, and dispatching application events.

Table of Contents / Applications and Tools

• Streamdal (⭐615) - Code-native data privacy controls for detecting PII in application data flows.

Table of Contents / Managed and Closed Source

• Amazon Kinesis Data Streams - Fully managed service for ingesting and processing real-time data streams on AWS.

• Azure Stream Analytics - Fully managed service for serverless real-time analytics in the cloud and at the edge.

• Concord - Historical distributed stream processing framework built on Apache Mesos.

• Google Cloud Dataflow - Fully managed service for running Apache Beam batch and streaming pipelines.

• IBM Streams - Discontinued proprietary platform for distributed stream processing and real-time analytics.

• NVIDIA DeepStream SDK - GStreamer-based toolkit with open-source components and proprietary NVIDIA libraries for real-time AI streaming analytics and multi-sensor processing.

Table of Contents / Benchmarks

• Flotilla (⭐238) - Automated message queue orchestration for scaled-up benchmarking.

Table of Contents / Readings

• MillWheel: Fault-Tolerant Stream Processing at Internet Scale

Table of Contents / Engines and Platforms

• Wingfoil (⭐225) - Graph-based stream processing engine for latency-critical systems.

Table of Contents / Libraries, SDKs, and Programming Models

• CapyMOA (⭐151) - Machine learning library for data streams with a Python API and MOA backend.

Table of Contents / Applications and Tools

• beava (⭐138) - Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.

• Eventum (⭐69) - Data generation platform for producing synthetic event streams.

Table of Contents / Libraries, SDKs, and Programming Models

• Kzmlabs StateFun Actors (⭐13) - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.

Table of Contents / Data Integration and Pipelines

• Bruin (⭐1.8k) - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.

Table of Contents / Readings

• Real Time Analytics: Algorithms and Systems (VLDB 2015)

Table of Contents / Engines and Platforms

• NebulaStream (⭐100) - Data management system for cloud-edge-sensor environments.

Table of Contents / Engines and Platforms

• ArkFlow (⭐1.3k) - High-performance stream processing engine with pluggable sources, processors, and sinks.

Table of Contents / Data Integration and Pipelines

• CocoIndex (⭐12k) - Incremental data transformation engine for continuously updated AI and agent workloads.

Table of Contents / Engines and Platforms

• Arroyo (⭐5k) - Distributed stream processing engine for stateful SQL computations over unbounded data.

Table of Contents / Engines and Platforms

• RisingWave (⭐9.4k) - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.

Table of Contents / Engines and Platforms

• AutoMQ (⭐11k) - Diskless Kafka-compatible streaming platform that stores durable data in object storage.

Table of Contents / Engines and Platforms

• Numaflow (⭐2.8k) - Kubernetes-native, language-agnostic platform for scalable event-driven applications.

Table of Contents / Engines and Platforms

• Proton (⭐2.3k) - Unified streaming and historical data analytics database powered by ClickHouse.

Table of Contents / Data Integration and Pipelines

• Pathway (⭐62k) - Python ETL framework for stream processing, real-time analytics, and AI data pipelines.

Table of Contents / Applications and Tools

• javactrl-kafka (⭐18) - Code-first distributed workflow engine for microservice orchestration and business process automation.

Table of Contents / Applications and Tools

• Substation (⭐407) - Toolkit for routing, normalizing, and enriching security event and audit logs.

Table of Contents / Libraries, SDKs, and Programming Models

• Quix Streams (⭐1.6k) - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.

Table of Contents / Engines and Platforms

• Bytewax (⭐2.1k) - Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.

Table of Contents / Libraries, SDKs, and Programming Models

• River (⭐6.1k) - Online machine learning library for Python.

Table of Contents / Engines and Platforms

• Scramjet Cloud Platform (⭐71) - Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.

Table of Contents / Libraries, SDKs, and Programming Models

• Scramjet C++ (⭐4) - Reactive stream programming framework for C++.

• Scramjet Python (⭐35) - Reactive stream programming framework for Python.

Table of Contents / Engines and Platforms

• Redpanda (⭐13k) - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.

Table of Contents / Applications and Tools

• Nussknacker (⭐743) - Visual tool for defining and running real-time decision algorithms.

Table of Contents / Engines and Platforms

• Apache RocketMQ (⭐23k) - Cloud-native messaging and streaming platform for event-driven applications.

Table of Contents / Libraries, SDKs, and Programming Models

• Daggy (⭐166) - Utility and developer library for aggregating and capturing data streams.

Table of Contents / Readings

• Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter

Table of Contents / Readings

• Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Reuven Lax, Slava Chernyak, and Tyler Akidau

Table of Contents / Engines and Platforms

• Siddhi (⭐1.6k) - Streaming SQL and complex event processing engine.

Table of Contents / Engines and Platforms

• ksqlDB (⭐315) - Source-available database purpose-built for stream processing applications.

• Materialize - Incremental SQL engine for maintaining continuously updated views over changing data.

Table of Contents / Engines and Platforms

• HStreamDB (⭐722) - Cloud-native streaming database for IoT data storage and real-time processing.

Table of Contents / Engines and Platforms

• Maki Nage (⭐42) - Stream processing framework for data scientists based on Kafka and ReactiveX.

Table of Contents / Readings

• Grokking Streaming Systems by Josh Fischer & Ning Wang

Table of Contents / Data Integration and Pipelines

• RudderStack (⭐4.5k) - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.

Table of Contents / Engines and Platforms

• Gazette (⭐798) - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.

Table of Contents / Engines and Platforms

• LightSaber (⭐73) - Multi-core stream processing engine using code generation for window aggregation.

Table of Contents / Libraries, SDKs, and Programming Models

• Apache Beam (⭐8.7k) - Unified programming model and language-specific SDKs for batch and streaming data processing.

Table of Contents / Libraries, SDKs, and Programming Models

• Stream Ops (⭐50) - Embeddable data streaming engine and stream processing API for Java.

Table of Contents / Engines and Platforms

• Gearpump (⭐755) - Lightweight real-time distributed streaming engine built on Akka.

Table of Contents / Libraries, SDKs, and Programming Models

• Streamz (⭐1.3k) - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.

Table of Contents / Libraries, SDKs, and Programming Models

• StormCV (⭐174) - Computer vision operations and data models for Apache Storm.

• streamDM (⭐497) - Huawei library for mining data streams with Spark Streaming.

• trident-ml (⭐384) - Online machine learning library based on Apache Storm Trident.

Table of Contents / Engines and Platforms

• StreamCQL (⭐0) - Storm-based continuous query language for real-time computation.

Table of Contents / Engines and Platforms

• Apache Apex (⭐350) - Unified platform for big data stream and batch processing.

• Apache Flink (⭐26k) - Distributed engine for stateful computation over bounded and unbounded data streams.

• Apache Kafka (⭐34k) - Distributed event streaming platform for high-performance data pipelines and applications.

• Apache Samza (⭐845) - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.

• Apache Spark Streaming (⭐44k) - Scalable, fault-tolerant stream processing on Apache Spark.

• Apache Storm (⭐6.7k) - Distributed real-time computation system for reliably processing unbounded streams of data.

• Esper (⭐876) - Complex event processing, Streaming SQL, and event series analysis engine.

• hailstorm (⭐93) - Distributed stream processing with exactly-once semantics based on Storm.

• Hazelcast Jet (⭐1.1k) - Stream and batch processing engine whose development moved into Hazelcast Platform.

• Onyx (⭐2k) - Distributed, masterless, fault-tolerant data processing platform.

• SABER (⭐45) - Window-based hybrid CPU/GPU stream processing engine.

• SPQR (⭐30) - Dynamic framework for processing high-volume data streams through pipelines.

• Teknek (⭐10) - Stream processing platform with the interactive Stream Operator Language shell.

Table of Contents / Libraries, SDKs, and Programming Models

• Apache Kafka Streams (⭐34k) - Stream processing library included with Apache Kafka.

• coast (⭐60) - DSL that builds exactly-once dataflow graphs on top of Apache Samza.

• Faust (⭐6.8k) - Deprecated Python stream processing library inspired by Kafka Streams.

• samza-luwak (⭐100) - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.

• Swave (⭐173) - Lightweight Reactive Streams infrastructure toolkit for Scala.

Table of Contents / Applications and Tools

• straw (⭐104) - Platform for real-time streaming search.

• StreamAlert (⭐2.9k) - Airbnb serverless framework for real-time security log analysis and alerting.

• StreamFlow (⭐257) - Tool for building and monitoring stream processing workflows.

• StreamingBandit (⭐86) - Web application for setting up and evaluating contextual multi-armed bandit experiments.

• Streamline (⭐167) - Hortonworks visual stream analytics framework built around engines such as Apache Storm.

• Turbine (⭐833) - Netflix tool for aggregating Server-Sent Event JSON streams.

Table of Contents / Benchmarks

• storm-perf-test (⭐75) - Apache Storm performance and stress test.

• streaming-benchmarks (⭐648) - Benchmarks for low-latency stream processing systems including Storm, Spark, and Flink.

Table of Contents / Readings

• In-Stream Big Data Processing

• The world beyond batch: Streaming 101 by Tyler Akidau.