Schema Versioning
Scope
Schema versioning applies to the Numax-managed sync namespaces stored in sled. Application keys and runtime identity metadata have separate ownership.
The managed tables are:
- materialized and durable state for each CRDT family;
- seen-operation metadata;
- the durable operation log;
- sequence allocation, receipt tracking, and retained replay positions.
These are logical tables implemented with key prefixes in one sled database.
Table metadata
Each logical table has a metadata key under:
__nx/schema/<table-name>Its value is an eight-byte header:
The magic value is NXDB. Versions and table identifiers use big-endian
encoding.
The table identifier prevents valid metadata for one namespace from being accepted for another. Each table owns its schema version independently.
Manual candidate metadata
The coordinator owns an additive version-1 namespace, independent of sync tables and membership. It consumes no CRDT operation or sequence number:
| Key | Value |
|---|---|
__nx/discovery/manual/schema | Eight bytes: NXMP, version u16 BE = 1, reserved u16 BE = 0 |
__nx/discovery/manual/cluster | Validated discovery cluster identifier |
__nx/discovery/manual/peer/<canonical-endpoint> | Empty value |
Endpoints use the existing coordinator canonicalizer and at most 259 bytes.
Loading uses pages of 64 records/32 KiB, records up to 512 bytes and at most
max_candidates manual endpoints. Unknown keys, unsupported headers, invalid
or noncanonical endpoints, nonempty peer values, partial namespaces and scope
mismatches fail explicitly. Missing namespace means an empty manual source.
Changing capacity must still accommodate the union with configured sources.
The coordinator prepares the entire candidate mutation before storage. An exclusive write lease covers the atomic header/scope/endpoint batch and its flush. Only then is the prepared state published and the request confirmed. No live candidate is published on write/flush failure. An uncertain flush or crash can leave the complete record; a fresh start validates and confirms a covering flush before publishing survivors. A retry of the canonical address is idempotent. Cancellation of an HTTP waiter is not transaction cancellation.
The namespace stores only candidates, never identity verification, membership,
timers, sockets or replica readiness. It supplements configuration without
changing CLI > NX_* > TOML > default precedence. No removal or TTL is provided.
No historical manual format exists, so no migration is introduced. Future
formats require an explicit migration. Existing sync schemas, wire protocol,
nx migrate and historical fixture bytes are unchanged. This additive namespace
alone is not a downgrade fence.
Membership metadata
The private membership storage adapter owns a separate version-1 namespace.
SyncManager invokes it during runtime startup. It does not add a sync-table id,
change CRDT schemas or consume replication sequence numbers.
| Key | Value |
|---|---|
__nx/membership/schema | Eight bytes: NXMB, version u16 BE equal to 1, reserved u16 BE equal to 0 |
__nx/membership/local | JSON object with bound local member and nonzero epoch |
__nx/membership/floor/<member> | JSON object with nonzero epoch, incarnation, terminal status (dead or left) and nullable closed_epoch |
Identities are UTF-8 strings of 1 through 256 bytes. Floor key suffixes contain the identity bytes directly, not a filesystem path. Epochs and incarnations are unsigned 64-bit counters; epoch zero is invalid. The local identity in metadata must match the caller’s configured identity, which is a trusted adapter input, not proof of authentication.
For example, a floor with epoch: 8, incarnation: 0, status: "dead" and
closed_epoch: 7 rejects old terminal pairs and retains the closure of epoch 7.
Floor merge keeps the maximum pair and maximum closed epoch independently.
When the maximum pair belongs to the closed epoch, its effective status is left, regardless of incarnation.
This is a rejection floor, not a newly owner-issued announcement. The merge is idempotent, commutative and associative.
The adapter rejects malformed/inconsistent fields, unknown namespace keys, unsupported headers, missing metadata and identity mismatches.
left requires its own epoch to be closed; a dead floor may retain a lower closed epoch but
cannot reopen a closed one. A local floor must be left and cannot name an epoch beyond the stored local allocator.
An empty membership namespace is initialized additively: the schema header and first bound epoch share one batch.
A partial or unknown namespace is never interpreted as empty or auto-repaired.
Restoration reads pages of at most 64 records and 128 KiB, rejects records over 2 KiB including their keys, and retains at most the configured number of remote floors (up to 4,096), one local floor and the schema/local records. Exhaustion fails visibly without deleting rejection protection. There is no TTL or compaction. Floor restoration establishes terminal ordering constraints only; it does not restore sessions, authentication, timers or active membership.
An exclusive store write lease covers read/merge/write and flush. Startup
advances the recovered epoch with checked arithmetic and confirms initial
Alive only after a flush covering its atomic header/local batch and all
surviving floors. Terminal persistence validates the pending action before
writing and checks that the stored local epoch still matches the instance.
Only a successful covering flush permits terminal publication. Runtime terminal
writes use one blocking worker and at most 64 queued actions. The core remains
available to serve control while I/O is pending; completion must still match
the pending token and pair before publication. A superseded in-flight floor
can remain durable without authorizing a stale announcement.
A storage failure fences the affected core. A rejected write publishes nothing; an uncertain flush may have persisted the whole batch, so reopening reads and retains surviving records instead of assuming rollback. The existing datastore durability fence prevents further writes on a handle whose flush failed. The next successful start confirms a new covering flush before publication. Losing the datastore or restoring an older backup does not guarantee a newer epoch and is outside this recovery contract.
No earlier membership format is supported. Future
versions require an explicit migration path; this adapter rejects them without
mutation. nx migrate still governs sync tables, not this separate namespace.
Historical CRDT fixtures and current sync-table versions are unchanged. Older
runtimes may ignore membership metadata: its presence alone is not a downgrade
fence. Protocol 9 supplies an exact-version network boundary; the existing
sync schema supplies the tested datastore downgrade boundary against v0.1.5.
Sequenced replication in v0.1.6
The op-log table (id 14) now uses schema 3. Its 1 -> 2 migration validates
existing operation payloads without assigning historical sequences or rewriting
those payloads. The version change also makes v0.1.5 reject an upgraded store
before it can mutate CRDT state without updating sequence metadata.
The additive sequences table (id 15, schema 3) uses __nx/crdt/sequence/:
local: the bound origin, current generation, and allocated sequence;origin/<origin>: contiguous prefix, known high sequence, sparse identities, optional unavailable-history evidence (source, interval and request ID), and legacy baseline requirement;usage: origin count, serialized receipt bytes, and sparse-position count;replay/<OpId>: original origin/generation/sequence, retained alongside the corresponding replay record and deleted when that record expires;baseline/<origin>: installed coverage of that origin’s pre-sequence state;failure/<origin>: the current source/reason rejection tuple, or JSONnullafter invalidation (at most one record per tracked origin);baseline-unconfirmed: a marker installed with a baseline batch and removed only after a covering flush succeeds.
The sequence-table 1 -> 2 migration preserves contiguous coverage, sparse
identities, known maxima and baseline requirements. It discards the old
uncorrelated unavailable boolean instead of inventing a source or interval.
Migration is resumable; receipt-byte accounting is recomputed and committed
atomically with the schema-2 header and checkpoint removal at completion.
The op-log and sequence-table 2 -> 3 steps validate and preserve payloads,
then advance their headers to fence runtimes that do not preserve transferred
baselines. Other CRDT tables remain at schema 1. Initialization establishes only
the store’s own legacy baseline provenance; receiving an advertisement from
another origin never establishes receipt of that baseline.
Baseline state, materialized values, per-origin receipt coverage and installed
legacy provenance commit together. A rejected batch publishes nothing. After
a failed flush, the complete batch may survive and its in-memory state is kept
coherent, but baseline-unconfirmed remains set and the store rejects writes
until reopen. A successful restart flush covers surviving state before clearing
that marker. An accepted transfer does not synthesize or replay old UUID-only
operations, and known maxima beyond its cut remain pending.
Failure records describe one source under the current recovery target; they do not prove permanent cluster-wide loss. Local pending requests include a receipt fingerprint, so a still-live refusal after another source repaired the gap is ignored. Recorded failures are invalidated when receipt coverage or known history changes. The fingerprint is local request metadata; the schema-3 record layout is unchanged. Storage faults retain their original cause rather than becoming data rejections.
Local state and sequence allocation commit in one batch before the covering flush. Remote state, positions, and receipt metadata likewise commit atomically before publication. Compacted receipt prefixes survive replay and seen-ID retention. Missing or corrupt authoritative sequence records fail closed.
Existing stores require explicit offline migration with
nx migrate --datastore-path /path/to/data. UUID-only records remain legacy;
their automatic remote replay is rejected, and the need for a legacy state
baseline is advertised separately. Historical fixture payloads remain intact.
Upgrade, historical state, and downgrade contract
Package, wire and table-schema versions are independent. The current storage transition is deliberately explicit:
| Input | Current runtime / migration result |
|---|---|
| Empty store | Initialize current table headers |
| v0.1.0 unversioned sync records | Runtime rejects; offline migration validates 0 -> 1, then upgrades op log and sequences to 3 |
| v0.1.5 schema-1 store | Runtime rejects the old op-log schema; offline migration validates its payloads and upgrades op-log and sequence headers to 3 |
| Development sequence-table schema 1 | Offline 1 -> 2 migration converts receipts, drops uncorrelated unavailable flags and recomputes byte usage, then validates 2 -> 3 |
| Development schema-2 op-log/sequence tables | Runtime rejects; offline 2 -> 3 validates payloads and advances the headers |
| Current schema-3 store | Open normally; migration is idempotent |
| Unknown future schema or malformed source | Reject; do not interpret it as empty data |
| v0.1.5 opening an upgraded store | Old runtime rejects op-log schema 3 before CRDT mutations |
Stop the runtime and preserve an offline backup before running nx migrate --datastore-path /path/to/data. Reopen with the new runtime only after migration
succeeds. A failed migration can have completed earlier table steps; it is
resumable, not a whole-database rollback transaction. Each record transformation
and checkpoint advance is atomic. Each final table header and checkpoint
removal is atomic, together with sequence usage accounting where required.
The path-based CLI migration confirms a flush before reporting success.
A failed flush has an uncertain outcome: reopen and inspect/retry migration;
never assume that an error means no batch reached storage.
Historical CRDT state and its materialized values survive migration. Old UUID-only operations retain their original payloads and local log indices; those indices are not origin sequences. Migration does not synthesize sequence positions or certify that a bounded historical log is complete. Unrelated application records remain unchanged.
On first sequence initialization, an existing pre-sequence store is marked as requiring a legacy baseline. The flag survives restart and new contiguous sequenced writes. It is advertised to other nodes independently of their new receipt frontiers. Old operations are excluded from sequenced replay; incoming legacy replay is rejected even after its UUIDs have left the seen cache. A connected new peer therefore cannot infer historical convergence from a successful handshake or complete coverage of new operations. Recovery of that baseline uses the bounded state-transfer contract. A source with incomplete coverage, an oversized payload or an incompatible receiver cut may refuse recovery explicitly; no unrestricted lossless recovery is promised.
A downgrade cannot safely write this store: the old writer would bypass sequence tracking. There is no reverse migration. Restoring an offline pre-upgrade backup is a separate operator action and does not preserve writes accepted since that backup. Mixed protocol-5/protocol-9 replication is also rejected; plan deployment for that communication interruption rather than assuming a rolling upgrade continues replication across the version boundary.
Compatibility verification
Historical fixtures are immutable release-generated records. Their original encoders, source tags and payload bytes are preserved; migration does not regenerate them using the current format. Reproduction procedures are in the storage and wire compatibility test guide.
Startup validation
The sync manager validates every managed table before hydrating persisted data:
| State | Result |
|---|---|
| Empty table without metadata | Initialize current metadata |
| Current metadata | Continue |
| Data without metadata | Reject as legacy |
| Older version | Reject; migration required |
| Future version | Reject as unsupported |
| Invalid magic, length, or table id | Reject as corrupted metadata |
Missing metadata is initialized in one atomic sled batch, after all tables have passed validation. Validation never partially initializes a store.
The runtime uses the fallible sync-manager constructor and propagates schema errors to the caller. It does not continue with empty in-memory CRDT state after a persistence error.
Migration boundary
Schema validation does not modify existing table data. Legacy and older schemas require an explicit migration.
A migration must:
- validate the source version;
- transform records deterministically;
- update data and schema metadata atomically where possible;
- be safe to retry;
- reject unknown future versions;
- have fixture-based tests for both successful and interrupted migration.
Migration registry
Each table owns an independent ordered registry of N -> N+1 steps. A step
may replace, create, move, or delete records. Its mutations and checkpoint
advancement use the same atomic sled batch. Every completed step writes its
schema version before the next step starts.
Inside the table being scanned, a record step may only mutate its own source key. New keys and moves must target a different namespace, preventing checkpoint gaps and double-processing.
Legacy migration
The migration engine supports v0 -> v1 for the original tables and
v1 -> v2 -> v3 for op log and sequences. Version 0 means either
legacy data without metadata or an explicit version-zero header.
The original-table v0 -> v1 and op-log steps validate existing payloads without
rewriting them. The sequence-table 1 -> 2 receipt conversion is described
above. Each original-table validation step proceeds as follows:
- process at most a configured number of records per batch;
- validate each key and payload with its table-specific decoder;
- persist the last validated key under
__nx/migration/<table-name>; - resume after that key when restarted;
- write the version-one schema header and remove the checkpoint atomically after the whole table has been validated.
The default limits are 512 records and 4 MiB per batch. Both limits apply:
- the record limit prevents unbounded table scans;
- one shared byte budget covers scanned input and generated mutations;
- a single record larger than the byte limit is rejected before its payload is cloned into the migration batch.
Memory use is bounded by the configured batch limits rather than the total table size.
Migration acquires an exclusive in-process write lease for its entire execution. Store clones may continue reading, but their mutations wait until the migration session ends. The sled database lock separately prevents another process from opening the same datastore.
The engine does not perform implicit migration during Runtime::new.
If validation fails, the checkpoint is not advanced and the schema header is not written. Existing payload bytes remain unchanged.
The v0.1.0 logical-store fixture supplies the original layout for the
compatibility verification, including byte
preservation and validation of the resulting CRDT state and operation log.