Membership, Downing & Placement

datum-cluster is Datum's cluster membership layer: a set of nodes that discover each other, agree on who is Up, notice when a node goes away, and expose a single ordered view of that membership to the rest of the stack. It is the substrate the cluster-aware datum-agent builds on for node-to-node control and job placement, and the foundation datum-cluster-sharding needs before it can place shards.

cargo add datum-cluster

It is deliberately a membership layer only. It mirrors the Akka Cluster member states Datum needs before placement, and nothing more: there is no leader with special duties beyond a deterministic coordinator convention, and no split-brain resolver yet (see Downing).

What runs under the hood (and stays there)

Membership is SWIM gossip, implemented with foca as the private failure detector. foca never appears in the public API — no foca identities, timers, configuration, or notifications leak out. You configure a ClusterConfig, start a ClusterNode, and read Datum types (Member, MemberState, MemberEvent). UDP is the v0.10 gossip transport; the API is transport-neutral so a future secure-gossip carrier does not change your code.

The node runs in two planes, matching Datum's two-plane rule:

A membership actor owns the registry: state transitions, current-state publication, ordered event publication, and downing decisions.
A gossip task — one Tokio task — owns the UDP socket and the foca instance. It receives datagrams and drains foca's runtime sends, timers, and notifications. It never spawns an actor per packet.

Quickstart

rust

use datum_cluster::{ClusterConfig, ClusterNode, MemberState};

// One member's configuration. Roles are advertised in gossip; seed nodes are
// ADDRESSES only (identities are discovered through gossip). This example
// starts a single loopback node with no seeds.
let config = ClusterConfig::new("orders-1").with_roles(["orders", "eu-west"]);

// Starting a node begins gossiping immediately and returns a live handle.
let node = ClusterNode::start(config)
    .await
    .expect("cluster node starts");
assert_eq!(node.node_id(), "orders-1");

// Two membership feeds, both ordinary Datum concurrency primitives:
//  - state():  a Signal<ClusterState>       — latest snapshot, no replay.
//  - events(): a Subscription<MemberEvent>  — every transition, in order.
let state = node.state();
let _events = node.events();

// The local member is in the snapshot immediately; peers converge async.
let snapshot = node.current_state();
assert_eq!(snapshot.self_node, "orders-1");
assert!(snapshot.member("orders-1").is_some());

// Wait for the local member to reach `Up`. With one member, this node is
// then the deterministic placement coordinator (oldest Up member, by
// incarnation then node id).
loop {
    if state
        .get()
        .member("orders-1")
        .is_some_and(|member| member.state == MemberState::Up)
    {
        break;
    }
    tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(node.current_state().is_placement_coordinator("orders-1"));

// Graceful leave: Leaving -> Exiting -> Removed, gossiped to peers.
node.leave().await.expect("graceful leave");

ClusterConfig::new(node_id) starts from loopback defaults; with_roles([...]) advertises role labels and with_seed_nodes([...]) lists the gossip addresses of existing members to join (identities are discovered, not configured). Everything else — gossip cadence, probe timeout, suspect timeout, downing timeout, event-buffer size — has a field on ClusterConfig with a sensible default.

The two membership feeds

A ClusterNode publishes membership through two Datum concurrency primitives, chosen for what each consumer actually needs:

Method	Type	Use it when you want
`node.state()`	`Signal<ClusterState>`	the latest immutable snapshot, no replay — the common case for "who is up right now". `node.current_state()` is the one-shot `Arc<ClusterState>` read.
`node.events()`	`Subscription<MemberEvent>`	every accepted transition, in order. The subscription is bounded and backpressures the publisher rather than dropping membership events.

ClusterState gives you member(node_id), up_members() (reachable Up members), and the coordinator helpers below. MemberEvent carries a monotonic sequence, so a consumer that falls behind can detect the gap and resync from current_state().

Member states

Every member moves through the Akka-style lifecycle. Reachability is an orthogonal flag (member.unreachable), not a state: a member stays Up while it is flagged unreachable, and only becomes Down once a downing provider decides.

   (discovered)
        │  MemberJoining
        ▼
     Joining ──MemberUp──▶  Up  ──MemberLeaving──▶ Leaving ──▶ Exiting ──▶ Removed
                            │ ▲                     (graceful leave; gossiped to peers)
              MemberUnreachable │ MemberReachable
                            ▼ │
                        (unreachable=true)
                            │  downing decides
                            ▼
                           Down ─────────────────────────────▶ (Removed)
                            │
                            │  same node id returns with a newer incarnation
                            ▼
                        MemberRejoined ──▶ Up

The MemberEventKind values name each edge: MemberJoining, MemberUp, MemberUpdated (roles or advertised address changed without a state change), MemberUnreachable, MemberReachable, MemberLeaving, MemberExiting, MemberDown, MemberRemoved, and MemberRejoined (a downed/removed node id came back with a newer incarnation). The incarnation counter distinguishes a restarted process from the old one at the same address.

Joining → Up. A newly discovered member is published Joining, then Up.
Up (unreachable). foca suspicion flags the member unreachable and emits MemberUnreachable; the state does not change yet.
Reachable again. If reachability is restored before downing fires, the flag clears (MemberReachable).
Down. A DowningProvider decides an unreachable member is Down.
Graceful leave. node.leave() walks Up → Leaving → Exiting → Removed and gossips the departure so peers converge to Removed instead of waiting for a failure-detector timeout.

Downing and the split-brain caveat

Failure detection and downing are deliberately separate. foca suspicion decides a member is unreachable; a DowningProvider decides when an unreachable member becomes Down:

rust

pub trait DowningProvider: Send + Sync + 'static {
    fn name(&self) -> &'static str;
    fn should_down(&self, member: &Member, unreachable_for: Duration, now: SystemTime) -> bool;
}

v0.10 ships exactly one provider, TimeoutDowning, which downs a member after it has been unreachable for a fixed timeout (ClusterConfig::downing_timeout, used by ClusterNode::start; pass your own with ClusterNode::start_with_downing).

Split-brain caveat — stated plainly. TimeoutDowning is not a split-brain resolver. v0.10 has no quorum, lease, or majority strategy. Under a network partition, timeout downing runs independently on each side: each side downs the members it cannot see and keeps the ones it can. There is no fencing — during a partition, more than one side can transiently satisfy the "oldest Up member" coordinator convention below and believe it is the coordinator. Treat automatic downing as appropriate for tests and small, well-connected deployments; a quorum/lease split-brain resolution provider is a named v0.11+ follow-up before placement can make strong singleton claims. This is the same trade-off Akka documents for its timeout-based downing.

The placement coordinator

Placement (both cluster jobs and sharding) needs a single node to make allocation decisions. Rather than run an election protocol, Datum uses a deterministic convention: the coordinator is the oldest reachable Up member, tie-broken by node id. C1 does not assign Akka up-numbers, so the age key is the propagated member incarnation.

rust

// Every node computes the same answer from the same view.
if state.is_placement_coordinator(node.node_id()) {
    // this node makes placement decisions
}
let coordinator = state.placement_coordinator(); // Option<&Member>

Because it is computed from the local ClusterState, every node with the same view agrees. The caveat is the same as downing: under a partition with timeout downing, different views can transiently produce different coordinators, and there is no fencing to prevent it.

Node sessions and the cluster-aware agent

Membership tells you who is in the cluster; node sessions are how nodes talk. The cluster-aware agent, ClusterAgent (in datum-agent), bundles the pieces together:

membership (ClusterNode),
the local job registry and DCP server,
a NodeSessionManagerHandle that maintains long-lived per-peer DCP sessions (reconnecting with bounded backoff) to every member advertising the agent role,
a placement coordinator.

Node sessions carry node-to-node control over DCP — the same protocol the CLI and TUI speak — so cluster control reuses one wire format. The transport mirrors DCP's rule: plaintext TCP on loopback for local dev, QUIC + mTLS for anything remote. Members that run a DCP agent endpoint advertise the agent role so peers know where to open a session.

rust

use datum_agent::{ClusterAgent, ClusterAgentConfig, dcp::DcpJobFactories};
use datum_cluster::ClusterConfig;

let agent = ClusterAgent::start(
    ClusterAgentConfig {
        cluster: ClusterConfig::new("orders-1").with_seed_nodes([/* peer gossip addrs */]),
        ..ClusterAgentConfig::default()
    },
    DcpJobFactories::new(),
)
.await?;

let membership = agent.cluster();            // &ClusterNode — state()/events()/current_state()
let sessions = agent.sessions();             // &NodeSessionManagerHandle

Cluster jobs: submit, placement, re-placement

A cluster job is an ordinary supervised datum-agent job whose placement is decided by the coordinator instead of being started on one specific node. Because closures don't cross the wire, cluster jobs use the same registered-factory model as DCP: each daemon registers named JobSpec factories at startup, and you submit an instance by factory name + parameters.

You submit through the datum CLI (or the typed DCP client):

# Least-loaded eligible node (default strategy).
datum submit --cluster --factory ingest --name ingest-eu --param shard=0

# Constrain to a role.
datum submit --cluster --factory ingest --role worker

# Pin to a specific node.
datum submit --cluster --factory ingest --node node-2

Placement is driven by a PlacementSpec:

Strategy	CLI	Meaning
`PlacementStrategy::LeastJobs`	(default)	pick the eligible node with the fewest cluster jobs, tie-broken by node id
`PlacementStrategy::Pinned { node_id }`	`--node <id>`	place on a specific node

--role <r> sets the role_constraint: only members advertising that role are eligible.

Re-placement on node loss. When the node a cluster job was assigned to leaves or is downed, the coordinator re-places the job on another eligible node, advancing the job's placement_generation and appending a ClusterPlacementHistory entry (from_node, to_node, reason, timestamp). Delivery of the work is not resumed mid-flight: re-placement rematerializes the whole job blueprint on the new node — it is job-level failover, not per-message state migration. (Sharding, covered in its own guide, is the layer that moves entities and their in-flight buffers.)

Observing the cluster from the CLI

Two datum subcommands read the cluster view over DCP (they fan out to peer node sessions with bounded partial-result semantics, so one slow node can't hang the command):

datum nodes — the membership + session view:

$ datum nodes
NODE    MEMBER  UNREACHABLE  SESSION    AGENT_ADDR       ROLES
------  ------  -----------  ---------  ---------------  -------------
node-0  Up      false        connected  127.0.0.1:9555   agent,orders
node-1  Up      false        connected  127.0.0.1:9556   agent,worker

datum ps --cluster — jobs aggregated across every node, with placement columns:

$ datum ps --cluster
NODE    ADDRESS         JOB         ID  STATE    DESIRED  GEN  STARTS  RESTARTS  CLUSTER  PLACED_ON  PGEN
------  --------------  ----------  --  -------  -------  ---  ------  --------  -------  ---------  ----
node-0  127.0.0.1:9555  placed-job  3   Running  Running  1    1       0         yes      node-1     1
node-1  127.0.0.1:9556  remote-job  1   Running  Running  1    1       0         -        -          -

CLUSTER marks jobs the coordinator placed; PLACED_ON and PGEN (placement generation) come from the job's cluster metadata. Both commands take --json for scripting, and both surface unreachable peers in a trailing NODE / ERROR table rather than failing outright. See the CLI guide for the full surface.

Performance

The datum-cluster numbers are recorded honestly in roadmap/benchmarks/cluster.md. The membership and control-plane paths are fast (all loopback, 2026-07-05):

Scenario	Nodes	Result
Join convergence (all nodes see all members `Up`)	5	43 ms wall
`ListClusterJobs` fan-out (empty registries)	3 → 5	195.6 µs → 372.2 µs p50
`SubmitClusterJob` → `Running` (through the coordinator)	3 → 5	438 µs → 488 µs p50
Cluster job re-placement after node kill	3 → 2	439 ms (kill → replacement `Running`, placement generation 2)

Read the re-placement row honestly. 439 ms is job-level failover — a whole blueprint rematerialized on a surviving node. It is not comparable to the frozen Pekko rebalance row (13,175 ms), which moves sharded entities under a 1,000,000-message workload. The apples-to-apples sharding rebalance comparison lives in the sharding guide. The CPU column for the membership rows is deferred until the cluster compare harness gains the shared /proc/self/stat sampler, and the record says so.

Where this is going

Named follow-ups, all recorded in the M10 roadmap: a quorum/lease split-brain resolution DowningProvider (v0.11+) to fence the coordinator under partitions, and QUIC + mTLS as a secure gossip transport (the control plane already uses it).

Membership, Downing & Placement ​

What runs under the hood (and stays there) ​

Quickstart ​

The two membership feeds ​

Member states ​

Downing and the split-brain caveat ​

The placement coordinator ​

Node sessions and the cluster-aware agent ​

Cluster jobs: submit, placement, re-placement ​

Observing the cluster from the CLI ​

Performance ​

Where this is going ​

See also ​

Membership, Downing & Placement

What runs under the hood (and stays there)

Quickstart

The two membership feeds

Member states

Downing and the split-brain caveat

The placement coordinator

Node sessions and the cluster-aware agent

Cluster jobs: submit, placement, re-placement

Observing the cluster from the CLI

Performance

Where this is going

See also