Python

The datum-stream package lets Python code build Datum stream blueprints while the Rust runtime does the execution. Construction is still Datum construction: creating a Source, Flow, Sink, or graph starts nothing. Execution begins when you materialize the blueprint with run() or run_with().

Use it when you want Python orchestration around a Rust stream runtime, or when your data is already Arrow-shaped and a Python batch UDF is the right boundary.

Install

pip install datum-stream

The package imports as datum:

import datum

The current package policy is CPython 3.13+ with abi3 wheels, Linux only, for x86_64 and aarch64 (M13 ratification). pyarrow and cloudpickle==3.1.2 are package dependencies.

The package ships py.typed plus PEP 561 stubs. CI can type-check user code with Source[int], Flow[int, int], Sink[int, Mat], RunnableGraph[Mat], Inlet[int], and Outlet[int]; Datum's own Python test gate runs both mypy and pyright over positive and negative examples. The first Python surface is intentionally honest about what exists today: linear user callables are integer-element callables, and Arrow work uses the separate batch stream types.

First Pipeline

import datum

with datum.Runtime() as runtime:
    graph = (
        datum.Source.range(0, 1_000)
        .map_multiply(3)
        .filter_greater_than(1_000)
        .to_mat(datum.Sink.fold())
    )
    total = graph.run(runtime).wait()

print(total)

Sink.fold() is the sum terminal, with Sink.fold_sum() as its explicit spelling. Sink.fold_product() multiplies elements and defaults its initial value to 1. These named terminals replace the former fold(op=...) string selector.

The Python operator ladder has three tiers:

• Callable convenience: map(lambda x: x + 1), filter(lambda x: x > 5), and flat_map(lambda x: [x, -x]). Datum batches integer elements into single-column Arrow RecordBatch values and executes one Arrow UDF per batch; the per-element loop runs inside the Python wrapper.
• Named Rust kernels: map_add(), map_subtract(), map_multiply(), and the named comparison methods such as filter_equal() and filter_greater_than(). Use these for hot integer paths.
• Batch/vectorized work: use col() expressions with select(), with_column(), and expression filter() for typed, lowerable Arrow kernels. Use map_batches(udf, output_schema=...) when the operation needs arbitrary Python over whole pyarrow.RecordBatch values.

Batch plans expose their selected expression backend through execution_report(). Passing metrics_level="counts", "timing", or "stalls" returns immutable per-node metric snapshots; metrics remain off by default.

Guides

Guide	What it covers
Pipelines And Graphs	Linear DSL, immutable blueprints, GraphDSL junctions, materialized values, Keep, completion handles, and errors
Arrow UDFs	RecordBatch -> RecordBatch UDFs, schemas, tracebacks, zero-copy notes, GIL behavior, and the measured performance model
Connect	datum.connect.serve() / connect(), token auth, trust model, subprocess UDF workers, wire formats, and current transport limits

Scope

The linear surface is deliberately narrow: integer streams, callable map / filter / flat_map through Arrow UDF batches, named arithmetic and compare kernels, take / drop, GraphDSL junctions with PartitionStrategy.MODULO, and named sink terminals. The batch surface adds typed col() expressions and explicit map_batches() UDFs. That keeps the Python builder close to Datum's Rust blueprint model.

For arbitrary Python logic that must be fast, prefer Arrow map_batches() and vectorized PyArrow kernels. The callable tier exists for ergonomics; it still pays Python per-element loop cost inside each Arrow batch.

Python construction is always strict. Type and schema mismatches are validated by the build call that creates the edge or blueprint, not deferred to materialization. Arrow batch edges always carry a known schema: non-empty inputs infer one, empty inputs require schema=..., and map_batches() requires output_schema=... before it returns a flow or source.