Apache Spark

.NET bindings.

Set of tools designed to analyse genomics data.

A scala kernel for Jupyter.

Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).

Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.

A library of general purpose functions and UDF's.

Upserts, Deletes And Incremental Processing on Big Data..

Upserts, Deletes And Incremental Processing on Big Data..

A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark.

Cluster computing system for processing large-scale spatial data.

Declarative machine learning framework on top of Spark.

IPython protocol based middleware for interactive applications.

Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.

Open-source toolkit for analyzing web archives.

Distributed Deep Learning library.

PySpark test helpers with beautiful error messages.

Workflow management system with Spark backend.

Cross-platform monitoring tool (Spark UI / Spark History Server replacement).

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Storage layer with ACID transactions.

A command-line tool for launching Spark clusters on EC2.

Data frame based graph API.

Genetic analysis framework.

A library that brings useful functions from modern database management systems to Apache Spark.

`joblib` backend for running tasks on Spark clusters.

PMML transformer library for Spark ML.

Type safe machine learning pipelines with RDDs.

Pandas DataFrame API on top of Apache Spark.

Kotlin API bindings and extensions.

Integration with the lakeFS atomic versioned storage layer.

REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.

\[status unknown\] - linear algebra DSL and optimizer with R-like syntax.

A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment.

Execution engine and serialization format which supports deployment of `o.a.s.ml` models without dependency on `SparkSession`.

Machine learning orchestration platform.

A system to manage machine learning models for `spark.ml` and `scikit-learn` .

Official MongoDB connector.

Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support.

Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.

Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from Netflix.

Python API for Deequ.

A native PySpark implementation of spark-daria.

Cassandra support including data source and API and support for arbitrary queries.

XML parser and writer.

C# bindings.

Golang bindings.

Rust bindings.

A Scala library with essential Spark functions and extensions to make you more productive.

A lightweight and fast testing framework.

Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.

Natural language processing library built on top of Apache Spark ML.

Collection of base test classes.

Haskell on Apache Spark.

H2O interoperability layer.

Helpers & syntactic sugar for PySpark.

An alternative R backend, using `dplyr`.

Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livy, in Jupyter notebooks.