Build production data pipelines in Python — batch or streaming,
scheduled or event-driven. ematix-flow combines a
high-performance query engine, orchestration, data-quality
checks, and an operator web UI into a single runtime.
One pip install. No cluster required.
Instead of stitching together stream processors, workflow schedulers, data-quality frameworks, and operational dashboards, ematix-flow provides one integrated runtime that does it all. It runs efficiently on a single machine while automatically optimizing execution, so you spend less time managing infrastructure and more time building data products.
Streaming pipelines with manual-commit at-least-once delivery, DLQs, Schema Registry, and exactly-once for Kafka → Kafka.
Workflows, DAGs, and composite cron / event / message triggers — no Airflow deployment to babysit.
Expectations and freshness SLOs declared on the pipeline — failures alert, stalled pipelines page.
An operator Web UI plus Prometheus / OpenTelemetry, Slack, email, and PagerDuty — built in.
Sum of 22 query medians · single nodes are 1× r7i.8xlarge (32 vCPU / 256 GB), clusters are 4× r7i.4xlarge — twice the box's cores and RAM
AUTO runs the whole suite on ONE box with a loopback peer, probing twin / mesh / mesh+broadcast per join query and running the fastest — 384.2 s, ahead of the single-node engine on the same box (16/22 queries) and DuckDB's 467.9 s. The clusters get a hardware handicap in their favor — twice the cores and RAM of the single box — and Trino still finishes 10.7× behind AUTO. PySpark's suite covers only Q01–Q04; from Q05 on the Spark master repeatedly removed the application (executor loss on the shuffle-heavy joins). Raw per-query JSONs with full provenance ship with every run stamp.
Sum of 22 query medians · single nodes are 1× c7i.4xlarge, clusters are 4× c7i.4xlarge
*partial suites — totals cover completed queries only (the missing ones would only add time). Polars 16/22: five queries exceed the 32 GB box in-memory, one lacks a SQL variant. ClickHouse MergeTree 19/22 (its own DDL + queries, newest stable server): Q02/Q05/Q08 each blew a 600 s-per-execution bound, and the 58.2 s comes after a 23.9-minute ingest plus 41 GiB of duplicated storage. ClickHouse on parquet 20/22: Q09 exceeds the memory cap a 32 GB box survives; Q12 hits a ClickHouse parquet-reader bug. PySpark completes all 22 after we gave it EBS shuffle scratch. Every number: same box class, same Parquet files, query-in-place unless labeled as ingest-first.
8.1–10.8× faster than the cluster engines — identical 4 nodes.
4× c7i.4xlarge · SF=10: 5.6× vs Trino, 6.5× vs PySpark · SF=1: 7.2× / 19×
One ematix node beats the single-node engines and the 4-node clusters.
Same queries, same S3 parquet — one node outruns the 4-node clusters ~12–14×
Fastest single-node engine at every scale we measured.
Same lead holds up-scale: SF=10 4.8 s vs 5.8 s · SF=100 51.8 s vs 59.0 s
Measured on AWS, not a laptop — TPC-H on c7i.4xlarge (16 vCPU /
32 GB) at SF=1/10/100 and r7i.8xlarge (32 vCPU / 256 GB) at
SF=1000, July 2026, shipped pip install ematix-flow
defaults, no flags, results value-validated.
Full benchmarks →
Install ematix-flow, declare a connection, and ship your first job + workflow. About 10 minutes end-to-end.
How ematix-flow is put together. Why it exists, what's inside, and the mental model behind jobs, workflows, modes, and streaming.
Recipes for common tasks — schedule with composite triggers, run streaming pipelines, drive the Web UI from the operator's seat.
Authoritative tables — what's shipped this release, and the TPC-H numbers vs DuckDB / Polars / PySpark.
A workflow names a DAG of jobs and declares when it fires. Member jobs declare where they sit inside the DAG.
Cron, event, message, and boolean combinations — AND, OR, nested — evaluated against last successful run.
Long-running Kafka / RabbitMQ / Pub/Sub / Kinesis consumers with at-least-once delivery and DLQ support.
Workflows / Jobs / Runs / DAG tabs, restart-from-step, live throughput on streaming pipelines.
TPC-H at SF=1/10/100/1000 on AWS — single node vs DuckDB / Polars, 4-node mesh vs Trino / PySpark, 1 TB on one box — plus the strict M4 Max protocol and concurrent-stream throughput. Reproducer commands included.
Backend matrix and v0.11.0 surface. What's stable, what's still in motion.
A workflow with a composite trigger (event + cron) plus within-DAG ordering.
from ematix_flow import ematix, ManagedTable, Annotated, BigInt, Text, pk
@ematix.connection
class warehouse:
kind = "postgres"
url = "${WAREHOUSE_URL}"
class OrdersExtracted(ManagedTable):
__schema__ = "analytics"; __tablename__ = "orders_extracted"
order_id: Annotated[BigInt, pk()]
customer_id: BigInt
amount_cents: BigInt
class OrdersEnriched(ManagedTable):
__schema__ = "analytics"; __tablename__ = "orders_enriched"
order_id: Annotated[BigInt, pk()]
amount_bucket: Text
@ematix.job(name="extract_orders",
target=OrdersExtracted, target_connection="warehouse",
mode="merge", keys=("order_id",))
def extract_orders(conn):
return "SELECT order_id, customer_id, amount_cents FROM raw.orders"
@ematix.job(name="enrich_orders",
target=OrdersEnriched, target_connection="warehouse",
mode="merge", keys=("order_id",),
depends_on=["extract_orders"])
def enrich_orders(conn):
return "SELECT order_id, CASE WHEN amount_cents < 10000 THEN 'small' ELSE 'large' END AS amount_bucket FROM analytics.orders_extracted"
# Workflow declares the trigger; member jobs declare their DAG position.
ematix.workflow(
name="orders_etl",
triggered_by=["upstream_workflow"],
schedule="0 21 * * *",
timezone="America/New_York",
jobs=["extract_orders", "enrich_orders"],
)
On PyPI as ematix-flow. The public API is stable
under semantic versioning — no breaking changes without a
major bump. Bug reports and design pushback are still exactly
what we want — file issues on
GitHub.