v1.0.0 — stable release

Replace your data stack.

Build production data pipelines in Python — batch or streaming, scheduled or event-driven. ematix-flow combines a high-performance query engine, orchestration, data-quality checks, and an operator web UI into a single runtime. One pip install. No cluster required.

One runtime, not a stack

Instead of stitching together stream processors, workflow schedulers, data-quality frameworks, and operational dashboards, ematix-flow provides one integrated runtime that does it all. It runs efficiently on a single machine while automatically optimizing execution, so you spend less time managing infrastructure and more time building data products.

Instead of custom Kafka consumers

Streaming pipelines with manual-commit at-least-once delivery, DLQs, Schema Registry, and exactly-once for Kafka → Kafka.

Instead of a separate orchestrator

Workflows, DAGs, and composite cron / event / message triggers — no Airflow deployment to babysit.

Instead of a bolt-on quality tool

Expectations and freshness SLOs declared on the pipeline — failures alert, stalled pipelines page.

Instead of a bespoke dashboard

An operator Web UI plus Prometheus / OpenTelemetry, Slack, email, and PagerDuty — built in.

9.6× faster than a 4-node cluster — on a single node
38× the price-performance of the clusters: ¼ the hardware, 9.6× the speed
22/22 TPC-H queries pass, value-validated, on shipped defaults — zero flags

Every engine we measured — TPC-H, same hardware class

Sum of 22 query medians · single nodes are 1× r7i.8xlarge (32 vCPU / 256 GB), clusters are 4× r7i.4xlarge — twice the box's cores and RAM

ematix auto · 1 node 384 s
ematix-flow · 1 node 417 s
DuckDB · 1 node 468 s
Trino · 4 nodes · 2× the hardware 4121 s
PySpark · 4 nodes · 2× the hardware DNF — 4/22 queries, application aborted from Q05 on

AUTO runs the whole suite on ONE box with a loopback peer, probing twin / mesh / mesh+broadcast per join query and running the fastest — 384.2 s, ahead of the single-node engine on the same box (16/22 queries) and DuckDB's 467.9 s. The clusters get a hardware handicap in their favor — twice the cores and RAM of the single box — and Trino still finishes 10.7× behind AUTO. PySpark's suite covers only Q01–Q04; from Q05 on the Spark master repeatedly removed the application (executor loss on the shuffle-heavy joins). Raw per-query JSONs with full provenance ship with every run stamp.

Sum of 22 query medians · single nodes are 1× c7i.4xlarge, clusters are 4× c7i.4xlarge

ematix auto · 4 nodes 46.0 s
ematix-flow · 1 node 51.8 s
DuckDB · 1 node 59.0 s
PySpark · 4 nodes 375 s
Trino · 4 nodes 497 s
ClickHouse MergeTree · 1 node · 19/22 · +24 min ingest 58.2 s*
Polars · 1 node · 16/22 251 s*
ClickHouse parquet · 1 node · 20/22 741 s*

*partial suites — totals cover completed queries only (the missing ones would only add time). Polars 16/22: five queries exceed the 32 GB box in-memory, one lacks a SQL variant. ClickHouse MergeTree 19/22 (its own DDL + queries, newest stable server): Q02/Q05/Q08 each blew a 600 s-per-execution bound, and the 58.2 s comes after a 23.9-minute ingest plus 41 GiB of duplicated storage. ClickHouse on parquet 20/22: Q09 exceeds the memory cap a 32 GB box survives; Q12 hits a ClickHouse parquet-reader bug. PySpark completes all 22 after we gave it EBS shuffle scratch. Every number: same box class, same Parquet files, query-in-place unless labeled as ingest-first.

Faster

Distributed, TPC-H SF=100

8.1–10.8× faster than the cluster engines — identical 4 nodes.

ematix auto 46.0 s
PySpark 375 s
Trino 497 s

4× c7i.4xlarge · SF=10: 5.6× vs Trino, 6.5× vs PySpark · SF=1: 7.2× / 19×

Cheaper

Every engine, SF=10

One ematix node beats the single-node engines and the 4-node clusters.

ematix-flow · 1 node 4.8 s
DuckDB · 1 node 5.8 s
Polars · 1 node · 19/22 9.1 s
Trino · 4 nodes 56.4 s
PySpark · 4 nodes 65.0 s

Same queries, same S3 parquet — one node outruns the 4-node clusters ~12–14×

Simpler

Single node, SF=1, zero tuning

Fastest single-node engine at every scale we measured.

ematix-flow 0.7 s
DuckDB 1.4 s
Polars · 21/22 2.7 s

Same lead holds up-scale: SF=10 4.8 s vs 5.8 s · SF=100 51.8 s vs 59.0 s

Measured on AWS, not a laptop — TPC-H on c7i.4xlarge (16 vCPU / 32 GB) at SF=1/10/100 and r7i.8xlarge (32 vCPU / 256 GB) at SF=1000, July 2026, shipped pip install ematix-flow defaults, no flags, results value-validated. Full benchmarks →

Documentation

Start the tutorial →

Featured concepts

All concepts →

Quick peek

Full tutorial →

A workflow with a composite trigger (event + cron) plus within-DAG ordering.

from ematix_flow import ematix, ManagedTable, Annotated, BigInt, Text, pk

@ematix.connection
class warehouse:
    kind = "postgres"
    url = "${WAREHOUSE_URL}"

class OrdersExtracted(ManagedTable):
    __schema__ = "analytics"; __tablename__ = "orders_extracted"
    order_id: Annotated[BigInt, pk()]
    customer_id: BigInt
    amount_cents: BigInt

class OrdersEnriched(ManagedTable):
    __schema__ = "analytics"; __tablename__ = "orders_enriched"
    order_id: Annotated[BigInt, pk()]
    amount_bucket: Text

@ematix.job(name="extract_orders",
            target=OrdersExtracted, target_connection="warehouse",
            mode="merge", keys=("order_id",))
def extract_orders(conn):
    return "SELECT order_id, customer_id, amount_cents FROM raw.orders"

@ematix.job(name="enrich_orders",
            target=OrdersEnriched, target_connection="warehouse",
            mode="merge", keys=("order_id",),
            depends_on=["extract_orders"])
def enrich_orders(conn):
    return "SELECT order_id, CASE WHEN amount_cents < 10000 THEN 'small' ELSE 'large' END AS amount_bucket FROM analytics.orders_extracted"

# Workflow declares the trigger; member jobs declare their DAG position.
ematix.workflow(
    name="orders_etl",
    triggered_by=["upstream_workflow"],
    schedule="0 21 * * *",
    timezone="America/New_York",
    jobs=["extract_orders", "enrich_orders"],
)
Stable — v1.0.0

On PyPI as ematix-flow. The public API is stable under semantic versioning — no breaking changes without a major bump. Bug reports and design pushback are still exactly what we want — file issues on GitHub.