Datum

Appearance

Persistence schema evolution ​

Persistent bytes outlive the code that wrote them. Datum makes the durable identifiers explicit so compatibility is a design choice you can test:

  • SerializerId chooses a decoder from SerializerCatalog.
  • The serializer manifest identifies a type/version within that serializer.
  • The adapter manifest identifies the domain-to-journal or domain-to-state mapping.

Serializer and adapter manifests are separate fields. Do not overload one as a replacement for the other, and never silently reuse a durable id for incompatible bytes.

Serializer catalogs ​

SerializerCatalog<J> has one primary serializer for new writes and any number of additional id-addressed decoders for historical frames. Registration rejects duplicate ids.

rust
use datum_persistence::{SerializerCatalog, SerializerId};

let mut catalog = SerializerCatalog::new(I64Serializer);
catalog.register(LegacyI64Serializer).unwrap();
assert_eq!(
    catalog.primary_serializer_id(),
    SerializerId::new(41),
);

I64Serializer is the current serializer from the event-sourcing quickstart, and the application-defined LegacyI64Serializer also implements PersistenceSerializer<i64>; they use distinct stable ids, and only the primary serializer is used by encode. decode looks up the id stored in SerializedPayload and passes the stored manifest and bytes to that serializer.

When rolling out a new serializer:

  1. Give it a new stable id if the byte codec changes incompatibly.
  2. Make it the catalog primary for new writes.
  3. Keep every old serializer registered as a decoder while its frames may exist in journals, snapshots, durable state, backups, or replicas.
  4. Remove an old decoder only after an explicit, verified data migration and retention window.

Protobuf serializer ​

The default protobuf feature exports ProstSerializer<J> for prost::Message + Default journal types. The caller supplies both durable identifiers:

rust
use datum_persistence::{ProstSerializer, SerializerCatalog, SerializerId};

let catalog = SerializerCatalog::new(ProstSerializer::<AccountOpened>::new(
    SerializerId::new(41),
    "account-opened-v2",
));

The manifest string is not inferred from a Rust type name, which keeps refactors from silently changing stored protocol.

Event adapters ​

EventAdapter<D, J> separates the current domain event from the stable journal representation. Writes are always one domain event to one journal value. On recovery, from_journal_events may map one old journal value to zero or more current domain events.

rust
use datum_persistence::{EventAdapter, PersistenceResult};

enum DomainEvent {
    Added(i64),
    Audited,
}

enum JournalEvent {
    Delta(i64),
    LegacyDeltaWithAudit(i64),
}

struct CounterAdapter;

impl EventAdapter<DomainEvent, JournalEvent> for CounterAdapter {
    fn to_journal(&self, event: &DomainEvent) -> PersistenceResult<(String, JournalEvent)> {
        let journal = match event {
            DomainEvent::Added(delta) => JournalEvent::Delta(*delta),
            DomainEvent::Audited => JournalEvent::Delta(0),
        };
        Ok(("counter-event-v2".into(), journal))
    }

    fn from_journal(
        &self,
        _adapter_manifest: &str,
        event: JournalEvent,
    ) -> PersistenceResult<DomainEvent> {
        match event {
            JournalEvent::Delta(delta) => Ok(DomainEvent::Added(delta)),
            JournalEvent::LegacyDeltaWithAudit(delta) => Ok(DomainEvent::Added(delta)),
        }
    }

    fn from_journal_events(
        &self,
        adapter_manifest: &str,
        event: JournalEvent,
    ) -> PersistenceResult<Vec<DomainEvent>> {
        match (adapter_manifest, event) {
            ("counter-event-v1", JournalEvent::LegacyDeltaWithAudit(delta)) => {
                Ok(vec![DomainEvent::Added(delta), DomainEvent::Audited])
            }
            (manifest, event) => self.from_journal(manifest, event).map(|event| vec![event]),
        }
    }
}

The same recovery mapping is used by event-sourced recovery and decode_envelopes, so write-side and projection interpretations stay aligned. Filtering an obsolete event is Ok(Vec::new()).

StateAdapter<S, J> provides a one-to-one to_journal / from_journal mapping for snapshots and durable state. There is no 0..n state recovery because exactly one state value must be installed.

Manifests are routing data ​

Treat both manifests as enums encoded in strings, even if your first version has only one case. Match known values explicitly and return PersistenceError::Serialization for unknown or malformed frames. Avoid Rust module paths, debug formatting, timestamps, and deployment-specific names.

IdentityAdapter returns an empty adapter manifest and clones the value. It is appropriate only when domain and journal types are intentionally the same durable contract.

The committed-fixture pattern ​

A round-trip test proves only that today's encoder agrees with today's decoder. It does not prove that today's code can read yesterday's bytes. Commit a small byte fixture produced by each released schema and decode it with the current code.

The crate's protobuf tests use this exact pattern:

rust
use datum_persistence::{PersistenceSerializer, ProstSerializer, SerializerId};
use prost::Message;

#[derive(Clone, PartialEq, Message)]
struct AccountOpened {
    #[prost(string, tag = "1")]
    account_id: String,
    #[prost(uint64, tag = "2")]
    opening_balance: u64,
    #[prost(string, tag = "3")]
    source: String,
}

const ACCOUNT_OPENED_V1: &[u8] = &[
    0x0a, 0x06, b'a', b'c', b'c', b't', b'-', b'7', 0x10, 0x7d,
];

let serializer =
    ProstSerializer::<AccountOpened>::new(SerializerId::new(41), "account-opened-v2");
let event = serializer
    .deserialize("account-opened-v1", ACCOUNT_OPENED_V1)
    .unwrap();

assert_eq!(event.account_id, "acct-7");
assert_eq!(event.opening_balance, 125);
assert_eq!(event.source, "");

Keep the fixture constant literal or in a checked-in binary file; do not regenerate it during the test. Test serializer-id routing, serializer manifests, adapter manifests, and recovery expansion in addition to the raw codec.

Evolution checklist ​

  • Keep PersistenceId identity rules stable; changing ids creates new streams rather than migrating old ones.
  • Never renumber a SerializerId or reuse it for incompatible bytes.
  • Keep old catalog decoders registered.
  • Preserve recognized serializer and adapter manifests.
  • Exercise old journal events through from_journal_events, including filtered and expanded cases.
  • Exercise old snapshots/durable values through StateAdapter::from_journal.
  • Commit old bytes and decode them with current code in CI.
  • Run recovery from a snapshot plus an old-format journal tail, not just isolated codec tests.

See also ​