Datum

Appearance

Arrow Flight Adapter ​

datum-flight serves Datum plan results over Apache Arrow Flight, an Arrow-native gRPC transport. It is an adapter over the engine, not a new engine: a Flight Ticket carries a protobuf-encoded DatumPlan, and DoGet streams the plan's result Arrow RecordBatches back as FlightData. Plan execution reuses the datum-connect engine seam (datum_connect::execute_plan_batches); the crate adds only the Flight/gRPC surface.

Arrow Flight is the alternative adapter for ADBC/BI-tool-style Arrow access. Datum's native Python-connect transport is Datum-native by design (see the Python Connect guide); Flight is the interop surface for tools that already speak Flight, and it is Datum's Arrow Flight transport comparison baseline. This is also the one place gRPC/tonic enters the workspace.

Scope (server-side DoGet MVP) ​

text
Flight client --Ticket(DatumPlan)--> do_get --> execute_plan_batches --> stream<FlightData>
  • FlightServerHandle::start binds a Flight server (its own OS thread + Tokio runtime, mirroring the datum-connect server handle) and shuts down gracefully on drop.
  • DoGet executes a ticketed DatumPlan and streams its result batches.
  • GetFlightInfo returns a FlightInfo with the plan's schema, a redeemable endpoint ticket, and row count. ListFlights returns an empty listing — clients drive execution by ticket.
  • Optional bearer-token auth (authorization: Bearer <token> metadata).

Building a plan and its Flight ticket:

rust
use datum_connect::{DatumPlan, PrimitiveSinkKind, PrimitiveSourcePlan};
use datum_flight::{FlightServerHandle, FlightServerOptions, plan_ticket};

// Start a Flight server on an ephemeral loopback port.
let server = FlightServerHandle::start(FlightServerOptions::default())?;

// A Datum plan: range(0, 10) collected as an int64 column.
let plan = DatumPlan::primitive(
    PrimitiveSourcePlan::range(0, 10, 1),
    Vec::new(),
    None,
    PrimitiveSinkKind::Collect,
    0,
);

// Hand this ticket to any Arrow Flight client's DoGet against `server.uri()`.
let ticket = plan_ticket(&plan);

Any Arrow Flight client (Rust arrow-flight, PyArrow, ADBC, …) can then DoGet the ticket and decode the returned FlightData into RecordBatches. See the crate's integration tests for a full arrow-flight client round trip.

Bearer-token auth ​

Set FlightServerOptions { token: Some(...), .. } to require an authorization: Bearer <token> metadata header on every RPC; requests without a valid token are rejected with UNAUTHENTICATED. This matches the datum-connect localhost-plus-token trust posture; TLS termination is expected via a reverse proxy for non-local deployments.

Performance ​

Flight DoGet runs at parity with the native connect path for small results and is ~1.1–1.3x slower (wall) on larger payloads — the honest cost of the gRPC/HTTP2 + Arrow-IPC stack over Datum's lean connect protocol. Full numbers, including CPU, are in the Arrow Flight benchmark record.

Future work ​

Flight SQL, a Flight client library, DoPut/DoExchange/DoAction, true incremental streaming, and plans that ship Python UDFs are out of scope for this MVP. UDF plans require the datum-connect server's subprocess worker pool and are rejected by the Flight adapter.