Quack HTTP Adapter (datum-quack)

datum-quack is a Quack-shaped HTTP adapter for Datum. It serves Datum plans over HTTP and streams Arrow RecordBatch results back. It is an adapter over the datum-connect engine seam — not a reimplementation and not Datum's native transport: each submitted plan is executed in-process through datum-connect's execute_plan_batches seam (the same executor the native connect server runs plans through, shared with the datum-flight Arrow Flight adapter), and that seam owns plan execution, validation, and the Arrow boundary. See Adding A Satellite for how satellite crates relate to the core, and Python Connect for the native transport that shares this engine.

Quack-shaped, not DuckDB Quack

The adapter borrows DuckDB Quack's transport shape, not its serialization:

• HTTP(S), token-secret (Authorization: Bearer …) auth, localhost-first defaults, and a reverse-proxy-friendly posture (the server does not terminate TLS itself).
• A submit/execute call plus a FETCH-chunked results call, client-driven throughout.
• The Quack default port 9494 and quack:-style positioning.

Results are framed as an Arrow IPC stream (uncompressed arrow-ipc by default; arrow-ipc-lz4 / arrow-ipc-zstd negotiable), not DuckDB's application/duckdb internal serialization.

Not DuckDB Quack compatible (yet)

Upstream Quack is in beta until DuckDB v2.0 (~September 2026), with a protocol still subject to change. datum-quack makes no compatibility claim — it is "a Quack-shaped HTTP endpoint for Datum plans." The handshake response reports quack_compatible: false. DuckDB v2.0 compatibility is tracked as a future rung.

Endpoints

All requests are POSTs under /quack/v1 and require the bearer token:

Endpoint	Request	Response
/quack/v1/handshake	JSON: client version, wire formats	JSON: server version, session id, offered wire formats, quack_compatible: false
/quack/v1/execute	body: a datum_connect::DatumPlan protobuf	JSON: execution id, fetch token, result wire format
/quack/v1/fetch	JSON: execution id, batch_budget	Arrow IPC stream chunk + x-quack-complete / x-quack-fetch-token headers
/quack/v1/cancel	JSON: execution id	JSON: terminal acknowledgement

The plan runs once at execute; each fetch returns up to batch_budget buffered result batches as one Arrow IPC stream body. A x-quack-fetch-token header is present until the stream completes; the final chunk carries x-quack-complete: true.

Starting a server

use datum_quack::{QuackServer, QuackServerOptions};

fn main() -> Result<(), datum_quack::QuackError> {
    // Localhost-first: defaults to 127.0.0.1:9494 and a generated token.
    let server = QuackServer::start(QuackServerOptions {
        addr: "127.0.0.1:0".to_owned(), // ephemeral port for tests
        token: Some("super-secret".to_owned()),
    })?;

    println!("serving at {} (token {})", server.base_url(), server.token());
    Ok(())
}

The in-process engine seam serves Rust-native and Arrow-batch plans with no Python dependency. Plans that ship Python UDFs are rejected — serving those would need the full connect server's subprocess worker pool behind the adapter, a future rung.

Submitting a plan

A client submits a datum_connect::DatumPlan protobuf to /quack/v1/execute, then repeatedly calls /quack/v1/fetch and decodes each Arrow IPC stream body until x-quack-complete: true:

1. POST /quack/v1/handshake — authenticate and learn the offered wire formats.
2. POST /quack/v1/execute with the plan protobuf as the body → returns an execution_id.
3. POST /quack/v1/fetch { "execution_id": …, "batch_budget": … } → an Arrow IPC stream chunk; repeat until complete.
4. POST /quack/v1/cancel { "execution_id": … } to stop an execution early.

The result stream is self-describing: each fetch body is a standard Arrow IPC stream that any Arrow reader (PyArrow, Polars, arrow-rs) can decode, carrying the schema in its header.

What is deferred

• A Rust HTTP client and DuckDB ATTACH interop (the MVP is server-side only).
• DuckDB Quack v2.0 compatibility once the upstream protocol stabilizes.
• Native in-server TLS/SASL (use a TLS-terminating reverse proxy for non-local deployments).

Performance

The adapter's cost is priced honestly against the seam it wraps. The benchmark (cargo bench -p datum-quack --bench quack) compares the HTTP adapter to the in-process execute_plan_batches engine seam and to the raw datum-core engine floor, on wall-clock and CPU. On a localhost loopback the adapter adds roughly ~12–18× wall over the in-process seam — the cost of two HTTP round trips (no keep-alive in the bench client), protobuf decode, result buffering, and an Arrow IPC re-encode — while the seam itself is only ~1.3× the raw engine floor. Full numbers and the levers that would close the gap (chiefly HTTP keep-alive) are recorded in the roadmap benchmark table for the crate.