Docker Multi-Node Cluster

The repository includes a three-node Docker Compose harness for exercising Datum cluster membership, placement, live demo jobs, and the operator console without a broker or other external service. Membership uses UDP gossip. Every DCP connection that leaves a process—including peer, CLI, and TUI traffic—uses QUIC with rustls + ring mutual TLS; plaintext DCP TCP remains disabled.

Prerequisites

Install Docker Engine with Compose v2 and OpenSSL 1.1.1 or newer. Run commands from the repository root. The host also needs the datum and datum-tui binaries; build the deliberately lean package set if necessary:

cargo build --release --locked -p datum-agent -p datum-cli -p datum-tui

Do not substitute cargo build --workspace: that pulls unrelated satellite crates and dev tooling into an otherwise standalone harness. The cluster harness does not use Kafka, and Datum's Kafka client is native-only.

Generate development identities

./docker/gen-certs.sh

The script creates two self-signed identities, matching the trust topology proven by the three-node cluster_sessions.rs test:

• a server certificate whose DNS SAN is exactly localhost;
• a client certificate trusted by every listener.

The six DER files used by datum-agent and the PEM files used by host tools are written under docker/certs/. They are ignored by git. The CLI, TUI, and agent session client hardcode localhost as the QUIC server name, so the SAN is not interchangeable with a container hostname or numeric address.

OpenSSL is used only by this development certificate script. The DCP TLS stack itself is C-free: peer sessions, CLI, and TUI all use rustls with ring, not aws-lc/CMake.

Start three nodes

docker compose -f docker/docker-compose.yml up --build -d
docker compose -f docker/docker-compose.yml logs -f

The compose network assigns 172.30.0.11 through 172.30.0.13. Static numeric addresses are load-bearing: cluster seed and advertised settings parse directly as SocketAddr, so service names do not work. Each DCP QUIC listener also binds its own static IP because Datum advertises the bound listener address to peers; an unspecified 0.0.0.0 peer endpoint would not be routable between containers.

The first node's 9556/udp is published to the host. Docker forwards that port even though the listener binds the container's static IP.

Verify, submit, and open the TUI

Use the PEM client material for all operator connections. The address must be numeric for CLI parsing; certificate verification still uses localhost and therefore matches the server SAN.

datum \
  --addr 127.0.0.1:9556 \
  --tls-ca docker/certs/ca.pem \
  --tls-cert docker/certs/client.pem \
  --tls-key docker/certs/client-key.pem \
  nodes

datum \
  --addr 127.0.0.1:9556 \
  --tls-ca docker/certs/ca.pem \
  --tls-cert docker/certs/client.pem \
  --tls-key docker/certs/client-key.pem \
  submit --cluster --factory ticker --name demo-1

datum-tui \
  --addr 127.0.0.1:9556 \
  --tls-ca docker/certs/ca.pem \
  --tls-cert docker/certs/client.pem \
  --tls-key docker/certs/client-key.pem

If the tools are not installed, use target/release/datum and target/release/datum-tui. Submit demo-2, demo-3, and further names to populate the cluster view. The node contacted by the host serves its local membership view and routes cluster job requests over its QUIC peer sessions, so one published port provides cluster-wide visibility.

DCP peer sessions use QUIC keepalive, and a slow peer request timeout no longer tears down the session. First-attempt nodes, ps --cluster, submit, drain, and stop operations should therefore remain reliable while the cluster is healthy; when a peer or coordinator does fail, the bounded reconnect loop normally restores the operator view within a few seconds in this harness.

Stop the harness with:

docker compose -f docker/docker-compose.yml down

Ports

Address	Protocol	Scope	Use
172.30.0.11:25520–172.30.0.13:25520	UDP	Compose network	Membership gossip
172.30.0.11:9556–172.30.0.13:9556	QUIC/UDP	Compose network	mTLS DCP peer sessions
127.0.0.1:9556	QUIC/UDP	Host to node 1	mTLS CLI/TUI entrypoint
9555	TCP	Disabled	Plaintext DCP remains loopback-only by design and is not published

Agent environment

The compose file sets only keys understood by the datum-agent bootstrap parser.

Setting	Value or pattern	Why it is set
DATUM_AGENT_CLUSTER	1	Enables the cluster entrypoint
DATUM_AGENT_CLUSTER_NODE_ID	datum-node-1 … datum-node-3	Stable node identity
DATUM_AGENT_CLUSTER_ROLES	agent,worker	Session discovery and job placement roles
DATUM_AGENT_CLUSTER_BIND_ADDR	Static IP plus 25520	Gossip socket bind
DATUM_AGENT_CLUSTER_ADVERTISE_ADDR	Static IP plus 25520	Numeric peer-visible gossip endpoint
DATUM_AGENT_CLUSTER_SEEDS	172.30.0.11:25520	Node 1 seed; the local identity is filtered on node 1
DATUM_AGENT_CLUSTER_GOSSIP_INTERVAL_MS	60	Proven three-node test cadence
DATUM_AGENT_CLUSTER_PROBE_TIMEOUT_MS	15	Proven probe timeout
DATUM_AGENT_CLUSTER_DOWNING_TIMEOUT_MS	150	Proven downing timeout
DATUM_AGENT_CLUSTER_AGENT_ROLE	agent	Selects members with DCP endpoints
DATUM_AGENT_CLUSTER_REQUEST_TIMEOUT_MS	300	Bounds peer fan-out
DATUM_AGENT_CLUSTER_RECONNECT_MIN_BACKOFF_MS	20	Initial session reconnect delay
DATUM_AGENT_CLUSTER_RECONNECT_MAX_BACKOFF_MS	100	Maximum session reconnect delay
DATUM_AGENT_CLUSTER_DCP_TRANSPORT	quic	Selects authenticated cross-container sessions
DATUM_AGENT_CLUSTER_DCP_QUIC_SERVER_NAME	localhost	Matches the server certificate SAN
DATUM_AGENT_DCP_TCP	0	Disables plaintext TCP
DATUM_AGENT_DCP_METRICS_INTERVAL_MS	20	Keeps live demo metrics responsive
DATUM_AGENT_DCP_QUIC_ADDR	Static IP plus 9556	Listener and advertised DCP endpoint
DATUM_AGENT_DCP_QUIC_KEEP_ALIVE_INTERVAL_MS	2000 (default)	Idle-session liveness probe interval for both QUIC endpoints
DATUM_AGENT_DCP_QUIC_MAX_IDLE_TIMEOUT_MS	10000 (default)	Bounds dead-peer detection on an inactive QUIC connection
DATUM_AGENT_DCP_QUIC_CERT_DER	server-cert.der	Listener identity certificate
DATUM_AGENT_DCP_QUIC_KEY_DER	server-key.der	Listener PKCS#8 key
DATUM_AGENT_DCP_QUIC_CLIENT_CA_DER	client-ca.der	Client identity trust anchor
DATUM_AGENT_CLUSTER_DCP_QUIC_SERVER_CA_DER	server-ca.der	Server identity trust anchor
DATUM_AGENT_CLUSTER_DCP_QUIC_CLIENT_CERT_DER	client-cert.der	Peer session client identity
DATUM_AGENT_CLUSTER_DCP_QUIC_CLIENT_KEY_DER	client-key.der	Peer session PKCS#8 key

The timing values mirror every corresponding environment-configurable field in the proven cluster_sessions.rs three-node setup. Fields without bootstrap environment keys retain the daemon defaults.

Why QUIC+mTLS is required

Datum deliberately refuses non-loopback plaintext TCP listeners and rejects plaintext node sessions to non-loopback peers. A Docker bridge therefore cannot use TcpLoopback. QUIC provides the non-loopback transport, while mutual TLS authenticates both the server and the connecting peer or operator. The same identities can be reused across these development containers; use managed, node-specific credentials for a production deployment.

Troubleshooting

Membership never reaches three nodes

Inspect all logs and check that 172.30.0.0/24 does not overlap an existing host route. Gossip is UDP on 25520; DCP peer sessions are QUIC/UDP on 9556. Keep seed, bind, advertise, and DCP listener values numeric. A service name such as datum-node-1:25520 will fail SocketAddr parsing.

QUIC handshake fails

Recreate credentials and containers, then inspect the SAN:

./docker/gen-certs.sh
openssl x509 -in docker/certs/ca.pem -noout -ext subjectAltName
docker compose -f docker/docker-compose.yml up --build -d --force-recreate

The SAN output must contain DNS:localhost. Confirm the mounted DER paths are readable and supply all three --tls-* flags together to operator tools.

Host tools cannot connect

Use 127.0.0.1:9556, confirm node 1 publishes 9556/udp (not TCP), and check the host firewall's UDP rules. docker compose ... ps should show the UDP mapping. Do not try the disabled TCP port.

Dev-server E2E

The harness is authored and locally validated without assuming access to a Docker daemon. The maintainer owns the live three-node E2E on the dev server: confirm all members become Up, all remote sessions show connected, ticker jobs remain running and accumulate throughput on placed nodes, and datum-tui renders the populated cluster. Environment-specific findings from that run should feed back into the harness.

See the repository-local docker/README.md for the compact command reference.