an web application for alert management resulting from scheduled searches into Elasticsearch.
Next-generation web analytics processing with Scala, Spark, and Parquet.
a platform that integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis.
open source web crawler.
capturing, processing and sharing of data for NASA's scientific archives.
content analysis toolkit.
Time series monitoring and alerting platform.
a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
a backend for managing dimensional time series data.
Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring.
open source mobile and web analytics platform, based on Node.js & MongoDB.
Run, scale, share, and deploy models — without any infrastructure.
Eclipse-based reporting system.
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.
open source event analytics platform.
open source simulation and visualization platform.
asynchronous message broker built on top of Kafka.
Splunk analytics for Hadoop.
Large scale analytics platform by indeed.
Web & mobile analytics tool, with data warehouse (AWS, BigQuery) integration.
Notebook and project application for interactive data science and scientific computing across all programming languages.
an open source framework for processing, monitoring, and alerting on time series data.
open source Distributed Analytics Engine from eBay.
data-processing library of an RDBMS to analyze data.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
R on Pivotal HD / HAWQ and PostgreSQL.
auto-scaling Hadoop cluster, built-in data connectors.
open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.
a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster.
enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.
R frontend for Spark.
analyzer for machine-generated data.
Substation is a cloud native data pipeline and transformation toolkit written in Go.
cloud based analyzer for machine-generated data.
unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig.
micro-benchmarks for testing Hadoop performances.
real-world big data workload benchmark.
a Hadoop benchmark suite.
benchmark suite for MapReduce applications.
extended Yahoo Cloud Serving Benchmark for NoSQL databases.
Hadoop cluster benchmarking from Yahoo engineer team.
Even more lists .
Analytics .
WTF! .
Other awesome lists .
Community Detection .
Decision Tree Papers .
Fraud Detection Papers .
Gradient Boosting Papers .
Graph Classification .
Kafka .
Monte Carlo Tree Search Papers .
Network Embedding .
Public Datasets .
A book about data engineering in general and the Azure platform specifically
Big Data teaches you to build big data systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data.
Data Science at Scale with Python and Dask teaches you how to build distributed data projects that can handle huge amounts of data.
– Theory of distributed systems. Include parts about time and ordering, replication and impossibility results.
This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field.
Fusion in Action teaches you to build a full-featured data analytics pipeline, including document and data search and distributed data clustering.
.
Alessandro Negro. Combine graph theory and models to improve machine learning projects
Grokking Streaming Systems helps you unravel what streaming systems are, how they work, and whether they’re right for your business. Written to be tool-agnostic, you’ll be able to apply what you learn no matter which framework you choose.
Kafka in Action is a fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.
Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort.
Another list? .
Reactive Data Handling is a collection of five hand-picked chapters, selected by Manuel Bernhardt, that introduce you to building reactive applications capable of handling real-time processing with large data loads--free eBook!
& Spark in Action 2nd Ed. - Spark in Action teaches you the theory and skills you need to effectively handle batch and streaming data using Spark. Fully updated for Spark 2.0.
Storm Applied is a practical guide to using Apache Storm for the real-world tasks associated with processing and analyzing real-time data streams.
Presents a new paradigm suitable for stream and complex event processing.
Streaming Data introduces the concepts and requirements of streaming and real-time data systems.
Unified Log Processing is a practical guide to implementing a unified log of event streams (Kafka or Kinesis) in your business
business intelligence platform in the cloud.
business intelligence made simple.
lean business intelligence platform to visualize and explore your data.
notebook-based anlytics and visualisation platform using SQL or drag-and-drop.
self-service business intelligence tool in the cloud.
Large scale geospatial analytics for Google BigQuery based on Kepler.gl.
platform for data products and embedded analytics.
Performance Monitoring for Amazon Redshift
powerful business intelligence suite.
customisable Business Intelligence platform.
Interactive Big Data Analytics.
open source business intelligence platform. (former SpagoBi)
The open source Looker alternative built on dbt
The simplest, fastest way to get business intelligence and analytics to everyone in your company.
business intelligence software and platform.
software platforms for business intelligence, mobile intelligence, and network applications.
Fast, clean SQL client and business intelligence.
business intelligence platform.
business intelligence and analytics platform.
Open source business intelligence platform, supporting multiple data sources and planned queries.
Open source analytics platform.
modern B.I platform powered by Apache Spark.
business intelligence platform.
Big Data Analytics.
column-oriented analytic database.
Amazon's cloud offering, also based on a columnar datastore backend.
an open-source column-oriented database management system that allows generating analytical data reports in real time.
an explanation of what columnar storage is and when you might want it.
a distributed, column-oriented database built for large-scale event collection and analytics.
Google's cloud offering backed by their pioneering work on Dremel.
an open-source columnar storage format for fast & realtime analytic with big data.
an experimental analytics database aiming to set a new standard for query performance on commodity hardware.
column store database.
columnar storage format for Hadoop.
purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one.
A GPU powered big data database, designed for analytics and data warehousing, with ANSI-92 compliant SQL, suitable for data sets from 10TB to 1PB.
is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.
data pipeline as a service enabling moving data sources such as MySQL into data warehouses.
real-time processing of streaming data at massive scale.
serverless fully managed extract, transform, and load (ETL) service
data collection system.
service to manage large amount of log data.
distributed publish-subscribe messaging system.
Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
tool to transfer data between Hadoop and a structured datastore.
A reverse ETL product that let you sync data from your data warehouse to SaaS Applications. No engineering favors required—just SQL.
open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.
streamed log data aggregator.
tool to collect events and logs.
Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
open source stream processing software system.
framework for connecting disparate data sources with Hadoop.
distributed message queue system.
stream of change capture events for a database.
linkedin's universal data ingestion framework.
utility package for compressing sorted integer arrays.
log aggregator and dashboard.
a tool for managing events and logs.
log agregattor like Storm and Samza based on Chukwa.
is a service implementing Kafka log persistance.
A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
an open source customer data infrastructure (segment, mParticle alternative) written in go.
sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.
continuous big data ingest infrastructure with a simple to use IDE.
An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol.
Web UI for PrestoDB.
fast, simple and flexible JavaScript (HTML5) charting library featuring pure JS API.
graph visualization library using web workers and jQuery.
visualize logs and time-stamped data stored in Solr. Port of Kibana.
Web UI for Impala.
A powerful Python interactive visualization library that targets modern web browsers for presentation, with the goal of providing elegant, concise construction of novel graphics in the style of D3.js, but also delivering this capability with high-performance interactivity over very large or streaming datasets.
D3-based reusable chart library
open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.
open source HTML5 Charts visualizations.
responsive, retina-compatible charts with just an img tag.
another open source HTML5 Charts visualization.
JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
JavaScript library for time series visualization.
JavaScript library for visualizing complex networks.
javaScript library for manipulating documents.
Compose complex, data-driven visualizations from reusable charts and components.
A fairly robust set of reusable charts and styles for d3.js.
Analytical Web Apps for Python, R, Julia, and Jupyter. Built on top of plotly, no JS required
one-stop data application development management portal.
Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.
Large scale geospatial analytics for Google BigQuery based on Kepler.gl.
High-performance plugin-based React chart for Bootstrap and Material Design.
Baidus enterprise charts.
dynamic HTML5 visualization.
write SQL queries that return SVG charts rather than tables
GitHub-inspired simple and modern SVG charts for the web with zero dependencies.
pen source real-time dashboard builder for IOT and other web mashups.
An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X.
simple charting API.
graphite dashboard frontend, editor and graph composer.
scalable Realtime Graphing.
simple and flexible charting API.
provides a rich architecture for interactive computing.
visualize logs and time-stamped data
open source big data analysis and visualization platform
plotting with Python.
a library built on top of D3 that is optimized for time-series data
chart components for d3.js.
Progressive SVG bar, line and pie charts.
Easy-to-use web service that allows for rapid creation of complex charts, from heatmaps to histograms. Upload data to create and style charts with Plotly's online spreadsheet. Fork others' plots.
The open source javascript graphing library that powers plotly.
A composable charting library built on React components
simple but powerful library for building data applications in pure Javascript and HTML.
open-source platform to query and visualize data.
a web application framework for R.
JavaScript library dedicated to graph drawing.
a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought.
a visualization grammar.
a notebook-style collaborative data analysis.
JavaScript charting library for big data.
reliable file sharing at memory speed across cluster frameworks.
a distributed object store that supports storage of trillion of small immutable objects as well as billions of large objects.
a way to store large files across multiple machines.
Hadoop's storage layer to enable fast analytics on fast data.
distributed filesystem.
formerly FhGFS, parallel distributed file system.
software storage platform designed.
distributed filesystem.
object storage system.
distributed filesystem.
scalable, highly available storage.
GGFS, Hadoop compliant in-memory file system.
high-performance distributed filesystem.
HDFS-compatible storage in Azure cloud
open-source distributed file system.
scale-out network-attached storage file system.
simple and highly scalable distributed file system.
decentralized cloud storage system.
distributed data processing and storage system originally developed at AddThis.
run Spark on Hadoop MapReduce v1.
a unified, enterprise platform for big data stream and batch processing.
an unified model and set of language-specific SDKs for defining and executing data processing workflows.
a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce.
collection of user-defined functions for Hadoop and Pig developed by LinkedIn.
high-performance runtime, and automatic program optimization.
real-time big data streaming engine based on Akka.
framework for in-memory data model and persistence.
BSP (Bulk Synchronous Parallel) computing framework.
programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
high level language to express data analysis programs for Hadoop.
retainable evaluator execution framework to simplify and unify the lower layers of big data systems.
framework for stream processing, implementation of S4.
stream processing framework, based on Kafka and YARN.
framework for in-memory cluster computing.
framework for stream processing, part of Spark.
framework for stream processing by Twitter also on YARN.
application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN.
abstraction over YARN that reduces the complexity of developing distributed applications.
an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale.
data processing and querying library.
High Performance, Custom Data Warehouse on Top of MapReduce.
framework for data management/analytics on Hadoop.
MapReduce library for Clojure.
alternative MapReduce paradigm.
real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
Hadoop enhancement which removes single point of failure.
Map Reduce framework.
distributed in-memory datastore.
create data pipelines to help themæingest, transform and analyze data.
map reduce framework.
fault tolerant stream processing framework.
platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box.
declarative programming language for working with structured, semi-structured and unstructured data.
is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
framework for real-time analysis of large datasets.
map-reduce for Clojure which compiles to Apache Pig.
MapReduce framework developed by Nokia.
Distributed computation for the cloud.
asynchronous job execution system.
Python MapReduce and HDFS API for Hadoop.
multi-tenant distributed metric processing system
A fast and simple framework for building and running distributed applications.
High performance distributed data processing in NodeJS.
general purpose cluster computing framework.
useful for counting activities of event streams over different time windows and finding the most active one.
Libraries to enable building IBM Streams application in Java, Python or Scala.
Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!
Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm.
Scala library for Map Reduce jobs, built on Cascading.
Streaming MapReduce with Scalding and Storm, by Twitter.
TimeSeries AggregatoR by Twitter.
The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
commercial object-oriented database management systems .
is an open source massively scalable data store. It requires zero administration.
Facebook’s Paxos-like NoSQL database.
document oriented datastore over Hadoop.
horizontally scalable document-oriented NoSQL data store.
Schema-agnostic Enterprise NoSQL database technology.
NoSQL cloud database service with protocol support for MongoDB
Document-oriented database system.
A transactional, open-source Document Database.
document database that supports queries like table joins and group by.
ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.
a software library that provides a high-performance embedded database for key/value data.
Erlang LSM BTree Storage.
a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
ultra-fast, ultra-compact key-value embedded data store developed by Symas.
embeddable persistent key-value store for fast storage based on LevelDB.
framework for distributed processing. Integrates MapReduce (parallel processing), YARN (job scheduling) and HDFS (distributed file system).
general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via *functions* and processes data via *column operations* as opposed to having only set operations in conventional approaches like MapReduce or SQL.
platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis.
A platform for reproducible and scalable machine learning and deep learning.
An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc...) streaming applications.
High Throughput Real-time Stream Processing Framework.
a new generation multi-model graph database for the modern complex data environment.
implementation of Pregel, based on Hadoop.
implementation of Pregel, part of Spark.
multi model distributed database.
A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.
a lightweight graph based database that does not require any third-party libraries.
TAO is the distributed data store that is widely used at facebook to store and serve the social graph.
Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.
open-source graph database.
graph processing framework.
a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API.
resilient Distributed Graph System on Spark.
graph traversal Language.
RDF-centric Map/Reduce framework.
tools to construct large-scale graphs on top of Hadoop.
open-source, distributed graph database
Massively Parallel Graph processing on GPUs.
a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine.
graph database written entirely in Java.
A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.
document and graph database.
framework for large scale graph processing.
distributed graph database, built over Cassandra.
distributed graph database.
Google** - The Google File System.
Google** - MapReduce: Simplied Data Processing on Large Clusters.
Google** - The Chubby lock service for loosely-coupled distributed systems.
Google** - Bigtable: A Distributed Storage System for Structured Data.
Amazon** - Dynamo: Amazon’s Highly Available Key-value Store.
AMPLab** - Chukwa: A large-scale monitoring system.
HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
Facebook** - Finding a needle in Haystack: Facebook’s photo storage.
AMPLab** - Spark: Cluster Computing with Working Sets.
Google** - Pregel: A System for Large-Scale Graph Processing.
Google** - Large-scale Incremental Processing Using Distributed Transactions and notifications base of Percolator and Caffeine.
Google** - Dremel: Interactive Analysis of Web-Scale Datasets.
Yahoo** - S4: Distributed Stream Computing Platform.
AMPLab** - Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters.
AMPLab** - Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.
Google** - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
Twitter** - The Unified Logging Infrastructure
AMPLab** - Blink and It’s Done: Interactive Queries on Very Large Data.
AMPLab** - Fast and Interactive Analytics over Hadoop Data with Spark.
AMPLab** - Shark: Fast Data Analysis Using Coarse-grained Distributed Memory.
Microsoft** - Paxos Replicated State Machines as the Basis of a High-Performance Data Store.
Microsoft** - Paxos Made Parallel.
AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
Google** - Processing a trillion cells per mouse click.
Google** - Spanner: Google’s Globally-Distributed Database.
AMPLab** - Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices.
AMPLab** - MLbase: A Distributed Machine-learning System.
AMPLab** - Shark: SQL and Rich Analytics at Scale.
AMPLab** - GraphX: A Resilient Distributed Graph System on Spark.
Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
Microsoft** - Scalable Progressive Analytics on Big Data in the Cloud.
Metamarkets** - Druid: A Real-time Analytical Data Store.
Google** - Online, Asynchronous Schema Change in F1.
Google** - F1: A Distributed SQL Database That Scales.
Google** - MillWheel: Fault-Tolerant Stream Processing at Internet Scale.
Facebook** - Scuba: Diving into Data at Facebook.
Facebook** - Unicorn: A System for Searching the Social Graph.
Facebook** - Scaling Memcache at Facebook.
Stanford** - Mining of Massive Datasets.
Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.
Benchmark of Redshift, Hive, Shark, Impala and Stiger/Tez.
Guide to monitoring Cassandra, including native methods for metrics collection.
Guide to monitoring Hadoop, with an overview of Hadoop architecture, and native methods for metrics collection.
Guide to monitoring Apache Kafka, including native methods for metrics collection.
Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison.
Platform for Internet of things.
Pub/sub messaging platform for IoT
a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
Cloud-based bi-directional monitoring and messaging hub
Making products smart
If this then that
Analytics platform to process network data on Spark.
Data stream network
Cloud-based sensor analytics.
Rapid development and connection of intelligent systems
distributed key/value store, built on Hadoop.
column-oriented distributed datastore, inspired by BigTable.
column-oriented distributed datastore, inspired by BigTable.
an Internet-scale database, inspired by BigTable.
evolution of HBase made by Facebook.
column-oriented distributed datastore.
is a fully managed, schemaless database for storing non-relational data over BigTable.
column-oriented distributed datastore, inspired by BigTable.
is accessed through a MySQL interface and use massive parallel processing to parallelize queries.
column-oriented distributed datastore written in C++, totally compatible with Apache Cassandra.
Transactions for HBase.
real-time, multi-tenant distributed database for Twitter scale.
NoSQL flash-optimized, in-memory. Open source and "Server code in 'C' (not Java or Erlang) precisely tuned to avoid context switching and memory copies."
distributed key/value store, implementation of Dynamo paper.
a fast, simple, efficient, and persistent key-value store written natively in Go.
an embedded key-value database for Go.
Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more
a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support.
is a protocol-compatible Server replacement for Redis.
Distributed database specialized in exporting data from Hadoop.
distributed time series database.
a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
a simple, fast, versioned, authenticated, embeddable key-value store database in pure Go(lang).
suitable for sensor data stored in a timeseries.
a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance.
is an in-memory key-value data store providing full SQL-compliant data access that can optionally be backed by disk storage.
is a simple persistent data store with very low latency and high throughput.
distributed key/value storage system.
distributed key-value database by Oracle Corporation.
in memory key value datastore.
a decentralized datastore.
library to work with asynchronous key value stores, by Twitter.
an in-memory, NoSQL key/value database, with disk persistence and using the Raft consensus algorithm.
an efficient NoSQL database and a Lua application server.
a distributed key-value database powered by Rust and inspired by Google Spanner and HBase.
a geolocation data store, spatial index, and realtime geofence, supporting a variety of object types including latitude/longitude points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON
key-value store that's replicated and sharded and provides atomic multirow writes.
Cloud-based AzureML, R, Python Machine Learning platform
CPU and GPU-accelerated Machine Learning Library.
Neural networks in JavaScript.
machine learning library for Cascading.
Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem.
Flexible and Extensible Machine Learning in Ruby.
Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.
machine learning framework that supports a variety of advanced algorithms, as well as support classes to normalize and process data.
text classification with machine learning.
scalable Machine Learning in Scalding.
A feature store for the management, discovery, and access of machine learning features. Feast provides a consistent view of feature data for both model training and model serving.
A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.
statistical, machine learning and math runtime with Hadoop. R and Python.
An unsupervised machine learning library for graph structured data. Python
An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow.
Lambdo is a workflow engine which significantly simplifies the analysis process by unifying feature engineering and machine learning operations.
A subsampling library for graph structured data. Python
An Apache-backed machine learning library for Hadoop.
All-in-one web-based IDE specialized for machine learning and data science.
distributed machine learning libraries for the BDAS stack.
Fast multilayer perceptron neural network library for iOS and Mac OS X.
MOA performs big data stream mining in real time, and large scale machine learning.
Text mining made easy. Extract and classify data from text.
A matrix library for the JVM. Numpy for Java.
Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.
Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning.
machine learning server built on Hadoop, Mahout and Cascading.
a temporal extension library for PyTorch Geometric .
Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem.
distributed streaming machine learning framework.
scikit-learn: machine learning in Python.
A data-driven framework to quantify the value of classifiers in a machine learning ensemble.
System for Large Scale Machine Learning at Google.
a Spark implementation of some common machine learning (ML) functionality.
Library from Google for machine learning using data flow graphs.
A Python-focused machine learning library supported by the University of Montreal.
A deep learning library with a Lua API, supported by NYU and Facebook.
System for serving machine learning predictions.
learning system sponsored by Microsoft and Yahoo!.
suite of machine learning software.
key/value cache for flash storage.
fork of Memcache.
A fast, light-weight proxy for memcached and redis.
key/value cache for flash storage.
fork of Memcache.
MySQL databases in Amazon's cloud.
evolution of MySQL 6.0.
MySQL databases in Google's cloud.
enhanced, drop-in replacement for MySQL.
MySQL implementation using NDB Cluster storage engine.
enhanced, drop-in replacement for MySQL.
High Performance Proxy for MySQL.
TokuDB is a storage engine for MySQL and MariaDB.
is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
commercially supported, open-source SQL relational database management system.
a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database.
data warehouse service, based on PostgreSQL.
statistic oriented SQL database.
a simple, modular, networked and distributed transaction layer built atop SQLite.
scales out PostgreSQL through sharding and replication.
Scalable, Geo-Replicated, Transactional Datastore.
a clustered RDBMS built on optimistic concurrency control techniques.
distributed database designed to enable scalable, flexible and intelligent applications.
distributed database, inspired by F1.
distributed SQL database built on Spanner.
globally distributed semi-relational database.
is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications.
linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.
NoSQL plugin for MySQL/MariaDB.
infinity scalable RDBMS.
a relational database backed by Apache Kafka.
GPU in-memory database, big data analysis and visualization platform.
in memory SQL database witho optimized columnar storage on flash.
SQL/ACID compliant distributed database.
in-memory, relational database management system with persistence and recoverability.
Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
is an in-memory, column-oriented, relational database management system.
distributed, realtime, semi-structured database.
database used for flexible, high performance analysis of behavioral data.
open source software for both file and database synchronization.
TiDB is a distributed SQL database. Inspired by the design of Google F1.
claims to be fastest in-memory database.
open source, high-performance, distributed SQL database compatible with PostgreSQL.
hybrid of MapReduce and DBMS.
high-performance data warehouse appliances.
The Streaming SQL Database. An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables
Scalable Open Source PostgreSQL-based Database Cluster.
Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
open source MPP database system solely targeted at data warehousing and data mart applications.
An open-source time-series database optimized for fast ingest and complex queries
multi-peta-byte database / MPP derived by PostgreSQL.
high performance interactive SQL access to all Hadoop data.
framework that allows efficient translation of queries involving heterogeneous and federated data.
framework for interactive analysis, inspired by Dremel.
table and storage management layer for Hadoop.
SQL-like data warehouse system for Hadoop.
SQL skin over HBase.
SQL-like analytic processing for MapReduce.
framework for interactive analysis, Inspired by Dremel.
SQL-like query language for Cascading.
full SQL query engine for big datasets.
an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow.
distributed SQL query engine.
framework for interactive analysis, implementation of Dremel.
an open table format for huge analytic datasets. Iceberg adds tables to Trino and Spark that use a high-performance format that works just like a SQL table.
SQL engine for online and on-premise use with integrated local data replication and 70+ connectors.
is a streaming database for real-time applications using SQL for queries and supporting a large fraction of PostgreSQL.
an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
SQL-like data warehouse system for Hadoop.
database for storing petabyte-scale volumes of structured and semi-structured data.
is a Query Optimization Framework for Spark and Shark.
Manipulating Structured Data Using Spark.
a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
interactive query for Hive.
distributed data warehouse system on Hadoop.
enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
a platform to programmatically author, schedule and monitor workflows.
is a service scheduler that runs on top of Apache Mesos.
data management framework.
workflow job scheduler.
cloud-based pipeline orchestration for on-prem, cloud and HDInsight
distributed and fault-tolerant scheduler.
Distributed, easy to install, NodeJS based, task scheduler
a data orchestrator for machine learning, analytics, and ETL.
batch workflow job scheduler.
Scala DSL for agile scheduling of Hadoop jobs.
scheduling platform.
is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.
Search engine library.
Search platform for Apache Lucene.
is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture.
Search and analytics engine based on Apache Lucene.
– Freemium robust web application for exploring, filtering, analyzing, searching and exporting massive datasets scraped from across the Web.
is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy.
continuous indexing system.
continuous indexing system.
implementation of Percolator, part of HBase.
quickly and easily search for any content stored in HBase.
is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
search architecture at LinkedIn.
is a realtime search/indexing system written in Java.
MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms.
fulltext search engine.
is an engine for low-latency computation over large data sets. It stores and indexes your data such that queries, selection and processing over the data can be performed at serving time.
Weaviate is a GraphQL-based semantic search engine with build-in (word) embeddings.
real time monitoring solution
single point of secure access for Hadoop clusters.
Central security admin & fine-grained authorization for Hadoop
security module for data stored in Hadoop.
The vulnerability detector for Hadoop and Spark
runtime for distributed, and fault tolerant event-driven applications on the JVM.
data serialization system.
Java libraries for Apache ZooKeeper.
OSGi runtime that runs on top of any OSGi framework.
framework to build binary protocols.
centralized service for process management.
a lock service for loosely-coupled distributed systems.
a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
cluster manager.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
message passing framework.
decentralized solution for service discovery and orchestration.
a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.
libraries for working with LZOP-compressed data.
asynchronous network stack for the JVM.
operational framework for Hadoop management.
system deployment framework for the Hadoop ecosystem.
cluster management framework.
cluster manager.
is a YARN application to deploy existing distributed applications on YARN.
set of libraries for running cloud services.
Cluster manager.
library that simplifies application deployment and management.
Similar to Apache BigTop based on Groovy language.
web application for interacting with Hadoop.
multi datacenters replication system.
job scheduling and monitoring system.
job scheduling and monitoring system.
application that can deploy HBase cluster on YARN.
a system for automating deployment, scaling, and management of containerized applications.
Linkis helps easily connect to various back-end computation/storage engines.
Mesos framework for long-running services.
Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support.
Facebook's in-memory time-series database.
A distributed system designed to ingest and process time series data
a time series storage built to store time series highly compressed and for fast access times.
uses MongoDB to store time series data.
Fast distributed metrics database
Column oriented distributed data store ideal for powering interactive applications
is a scalable time series database based on Cassandra and Elasticsearch.
a time series database with optimised IO and queries, supports pgsql and influx wire protocols.
scalable, general-purpose time series database.
similar to OpenTSDB but allows for Cassandra.
a distributed time series database that can be used for storing realtime metrics at long retention.
a time series database based on Apache Cassandra.
distributed time series database on top of HBase.
a time series database and service monitoring system.
high-performance, open-source SQL database for applications in financial services, IoT, machine learning, DevOps and observability.
A time-series object store for Cassandra that handles all the complexity of building wide row indexes.
Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data.
Highly-scalable, robust and fast, open source time series database with cluster functionality.
a time series database in C utilizing unique features of IoT to improve read/write throughput and reduce space needed to store data
Thanos is a set of components to create a highly available metric system with unlimited storage capacity using multiple (existing) Prometheus deployments.
Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana.
an efficient tool for storing and querying series of events.
fast, scalable and resource-effective open-source TSDB compatible with Prometheus. Single-node and cluster versions included
Introduction to schema design for data warehouse using the star schema method.
LiveVideo tutorial that covers searching, analyzing, and visualizing big data on a cluster with Elasticsearch, Logstash, Beats, Kibana, and more.
LiveVideo tutorial that covers machine learning, Tensorflow, artificial intelligence, and neural networks.
Spark in Motion teaches you how to use Spark for batch and streaming data analytics.