Cluster Operator Console

datum-tui is the full-screen, cluster-aware operator console for Datum. It is a thin DCP client: it connects to one datum-agent, uses that agent's cluster fan-out, and never embeds the stream engine or cluster runtime in the console process.

The paper theme is the default. Press t to switch to the machine theme; both require a truecolor (24-bit RGB) terminal for the intended palette.

Connect

Run the console against the default loopback DCP listener:

cargo run -p datum-tui

Select another agent with --addr:

cargo run -p datum-tui -- --addr 10.0.0.12:9555

The connection flags mirror the datum CLI:

• --addr <HOST:PORT> — DCP listener, default 127.0.0.1:9555.
• --tls-ca <PATH> — trusted CA certificate for QUIC+mTLS.
• --tls-cert <PATH> — client certificate for QUIC+mTLS.
• --tls-key <PATH> — client private key for QUIC+mTLS.

Pass all three TLS flags together. With none of them, the console uses plain TCP (loopback by default).

What it shows

The header summarizes membership and the placement coordinator. The nodes panel shows member, session, role, locality, coordinator, reachability state, and live per-node CPU, memory (RSS), and throughput. The jobs panel is a cluster-wide ps view, and the inspect panel expands the selected job's lifecycle, placement, element count, and derived live throughput. Placement changes caused by failover are labelled as relocations, with job_replaced events showing the source node, target node, reason, and generation when the cluster reports them.

The cluster-events pane follows the live SubscribeClusterEvents DCP stream. It includes membership changes, coordinator changes, and placement events such as job_replaced moves. An event sequence gap causes an immediate cluster snapshot resync so the table state catches up with the event feed.

Snapshots are polled every 1.5 seconds. Throughput is derived client-side from timestamped deltas between cumulative elements_through samples. DCP metric delivery is intentionally lossy under backpressure, so rates, sparklines, CPU, and RSS are live operational gauges, not accounting totals. On the wire, elements_through is the maximum of a stage's element, row, and batch counts; scalar stages report elements, while Arrow-batch stages report rows. The compact n/a marker in cluster capacity gauges and the longer unavailable marker in job panels and the CLI both mean unmeasured. Peer-session reconnect and coordinator failover are bounded by the cluster backoff settings; in the default local and Docker setups, the console normally resynchronizes within a few seconds.

Keys

Key	Action
j, Down	Select the next job (clamped at the last row)
k, Up	Select the previous job (clamped at the first row)
Enter	Re-fetch the selected cluster job's status
d	Ask to drain the selected job; confirmation is required
r	Ask to restart the selected job; confirmation is required
y, Enter	Confirm a pending control action
n, Esc	Cancel a pending control action
t	Toggle paper/machine theme
left-click a node	Toggle that node as a jobs-table filter; other rows are dimmed
q, Ctrl-C	Quit

Drain is cluster-aware. Restart currently acts on the connected agent's local registry because DCP does not yet expose a cluster-aware restart call.

The terminal's raw mode, mouse capture, cursor, and alternate screen are restored on ordinary exit. On panic, a restore-first hook chains to the previous hook so the crash reason prints on the restored screen; a drop guard also covers unwinding.

Live and pending telemetry

Available now through DCP:

• cluster node membership/session snapshots;
• cluster-wide job lifecycle and placement status;
• per-node CPU, memory (RSS), throughput, and throughput history;
• cumulative job elements, derived job throughput, and throughput history;
• GraphDSL fused-stage stall time, queue depth, processing time, and p99 when the agent job records its FusedNodeMetrics through the job-context bridge;
• membership, coordinator-change, and placement events;
• control-action responses.

Linear DSL instrumentation produces element counts and lifecycle only, so stage name, stall, depth, processing time, and p99 render as unavailable for those jobs. Watermark lag is integration-owned for both DSLs and is unavailable until an integration supplies it. Per-job CPU and RSS are also unavailable because jobs share worker threads and the process heap.

The TUI's max stage p99 value is computed from exactly one latency sample per completed fused report. For a factory that emits one element per report, it is per-element latency. For a multi-element graph run, it is a percentile over per-run stage-processing totals, not over the individual elements in each run.

Cluster capacity lines begin with nodes m/N; Σreport (or Σr in a narrow pane) means the following CPU/memory capacities are summed over the m metric-reporting nodes only, not silently presented as a total for all N members. If a capacity tail cannot fit in the pane, the tail is dropped before the mandatory Σr scope marker. The panel and gauge chrome stays visible so missing instrumentation cannot be mistaken for a healthy zero.