Skip to content
Numax

Gossip Protocol

This page explains what gossip means in Numax, what the current sync layer already does, and what will arrive in the peer-discovery releases.

The short version: Numax discovers connection candidates, broadcasts directly to active peers, and repairs missed operations through periodic anti-entropy. Discovery is dynamic in v0.1.5; SWIM-style membership and K-fanout data gossip remain future work.

What gossip is

A gossip protocol is a way to spread information through a distributed system without requiring one central coordinator.

Instead of sending every update through a leader, each node talks to some peers. Those peers talk to other peers. Over time, the information spreads through the cluster.

For Numax, the information being spread is mostly CRDT operations:

pub struct Op {
    pub id:     OpId,
    pub origin: NodeId,
    pub kind:   OpKind,
}

Each operation has a globally unique OpId, the node that produced it, and the CRDT change itself. Peers use OpId to deduplicate messages they have already seen.

What exists today

The current data-replication implementation remains intentionally simple and deterministic. A node obtains endpoint candidates from static, bootstrap, mDNS, DNS-SRV or file providers. The same bounded, updateable candidate snapshot feeds initial dialing and reconnect. Anti-entropy uses all active connections, including inbound peers and connections whose candidates have been removed. Starting with no candidates is valid; later provider updates wake the connection machinery. Local readiness does not imply peer convergence.

Candidates are not members or peers yet. A candidate becomes an active peer only after connection admission, the normal wire handshake, TLS identity binding when configured, and allowlist authorization. When an operation is produced locally, the sync manager queues it and sends it to the currently connected peers.

local CRDT host call
       |
       v
op queued in SyncManager
       |
       v
broadcast loop batches ops
       |
       v
PushOps sent to connected peers
       |
       v
peer applies unseen ops and persists state

This is not SWIM and it is not K-fanout yet. It is a full broadcast to connected peers, bounded by max_peers, with batching and backpressure through the operation queue.

Wire messages

Peer communication is handled by nx-net. The current wire protocol defines these message kinds:

MessagePurpose
HelloStart the handshake, declare node id, protocol version and supported serialization formats.
HelloAckAccept the handshake and choose the serialization format.
PushOpsSend one or more CRDT operations to a peer.
PushOpsAckAcknowledgement message for received operations. The current sync path does not rely on it as a causal frontier.
PullSinceAsk a peer for retained operations. Today this is usually sent with None.
Ping / PongKeepalive message types. A received Ping is answered with Pong.
ErrorStructured wire error sent before rejecting a request or closing a connection.
BootstrapHelloStart a one-shot authenticated bootstrap request with cluster, advertisement and result limit.
BootstrapAckReturn the seed identity, matching cluster, negotiated format, bounded endpoint suggestions and their lease.

The protocol version is currently 5. Peers negotiate either Bincode or Json, with Bincode as the production default and Json available for debug-style interoperability. Version 5 is deliberately incompatible with the version 4 wire contract from Numax v0.1.4.

Handshake and identity

When a node connects to a peer, the first exchange is:

client -> server: Hello(node_id, protocol_version, supported_formats, preferred_format)
server -> client: HelloAck(node_id, protocol_version, selected_format)

After that, both sides know the peer NodeId and the selected serialization format.

If TLS is enabled, the claimed NodeId is checked against the peer certificate. This prevents a node from claiming an identity that does not match its certificate. Optional allowlists can further restrict which peer ids are accepted.

Protocol compatibility

Numax currently requires an exact protocol-version match, there is no implicit forward or backward compatibility:

Local nodePeer nodeResult
NNCompatible; continue the handshake
NN - 1Incompatible; reject the handshake
NN + 1Incompatible; reject the handshake

Both Hello and HelloAck carry an explicit protocol_version, a mismatch is rejected before the peer is registered or CRDT operations are exchanged. When possible, the rejecting peer sends Error(ProtocolMismatch) before closing. Serialization-format negotiation does not override protocol compatibility. The rules for evolving this contract are defined in Wire Versioning.

Bootstrap handshake

Bootstrap uses the same listener but a separate, one-shot first message:

client -> seed: BootstrapHello(
  node_id, protocol_version, supported_formats, preferred_format,
  cluster_id, advertised_endpoint?, max_results
)
seed -> client: BootstrapAck(
  node_id, protocol_version, selected_format,
  cluster_id, candidates, candidate_ttl_ms
)
connection closes

The seed authenticates the requesting node using the same TLS certificate binding and allowlist checks as a normal peer handshake, validates cluster and advertised endpoint, then records that endpoint under a bounded lease. The client likewise authenticates the seed and validates the complete response. Cluster mismatch or an invalid request yields BootstrapRejected.

The one-shot exchange never enters the active peer map and emits no PeerConnected event. Authentication proves only who answered and who made the request; it does not vouch for any endpoint in candidates. Each suggestion is fed into normal reconnection and must authenticate independently before CRDT traffic can flow. Response count, cache size, candidate TTL, message size, socket time and concurrent client queries are all bounded.

Broadcast path

Local CRDT writes are applied locally first. Then the corresponding operation is queued for network propagation.

The broadcast loop drains that queue, groups operations into batches, records them in the seen-op and op-log metadata, and sends a PushOps message through nx-net.

The important boundaries are:

BoundaryCurrent default
Maximum connected peersnx_net::DEFAULT_MAX_PEERS
Queued local ops10,000
Retained op-log entries10,000
Retained seen OpIds100,000
Socket timeoutnx_net::DEFAULT_SOCKET_TIMEOUT

If a peer is disconnected, it does not receive the immediate push. That is why anti-entropy exists.

Anti-entropy

Anti-entropy is the repair loop.

Every anti_entropy_interval, a node asks each active connection for retained operations using PullSince. This cadence is independent of discovery churn; missed ticks are skipped rather than replayed in a burst. Candidate removal stops future reconnect attempts, not repair over an already admitted connection.

Today the request is conservative: it asks for the bounded op-log rather than relying on a single “last seen op id” as a causal frontier. That matters because one newer operation does not prove that every older operation arrived.

The receiving side deduplicates by OpId, applies only unseen operations, and persists the resulting CRDT state.

node A missed op-7 during a temporary disconnect
       |
       v
node A reconnects
       |
       v
anti-entropy sends PullSince(None)
       |
       v
node B returns retained ops
       |
       v
node A applies only unseen OpIds

The op-log and deduplication history are bounded, so anti-entropy is a practical catch-up mechanism, not an infinite historical archive or state transfer. Rediscovery alone cannot guarantee recovery when the required history is gone.

Peer health and reconnect

Current candidates have a small health state:

StateMeaning
HealthyThe candidate is connected or recently connected successfully.
SuspectA connection attempt failed, but the peer has not crossed the failure threshold.
DeadConsecutive failures reached peer_dead_after_failures.

Reconnect uses exponential backoff:

SettingCurrent default
First reconnect delay500ms
Maximum reconnect delay30s
Dead after failures3
Anti-entropy interval30s

This is simple failure tracking for discovery candidates, including configured peers. It is not a full membership protocol yet.

What is not implemented yet

The current release line does not yet provide:

  • SWIM membership,
  • Lifeguard-style failure detection,
  • phi-accrual failure detection,
  • K-fanout dissemination,
  • adaptive gossip rate,
  • NAT traversal,
  • causal frontier metadata for precise incremental pulls.

If you see “gossip” in the current docs, distinguish bootstrap gossip — a bounded exchange of endpoint suggestions — from data gossip. Current CRDT propagation is still a broadcast to all active peers, with anti-entropy as its repair path. Bootstrap suggestions are not membership state.

Current foundations and next steps

Peer discovery foundations are implemented in the current v0.1.5 release; membership and K-fanout remain planned for v0.1.6.

v0.1.5 - Peer Discovery: Foundations

This release introduces the PeerDiscovery contract and five Rust provider implementations: static configuration, authenticated bootstrap, LAN mDNS, DNS-SRV and an externally updated peer file. Snapshot/watch handoff is atomic, delivery is bounded with explicit overflow, and provider tasks are owned and stopped by runtime shutdown.

All five modes (static, bootstrap, mdns, dns-srv, file) are selectable through --discovery-mode, NX_DISCOVERY_MODE and the [discovery] TOML section, with precedence CLI > environment > TOML > defaults. Explicit --peer entries continue to contribute a static source alongside the selected dynamic provider; they are not reinterpreted as bootstrap seeds. Embedders can also compose the public providers through the nx-core Rust API. The detailed semantics are in the Peer Discovery Contract.

v0.1.6 - Peer Discovery: SWIM & Gossip K-fanout

This is where the protocol becomes a real dynamic cluster protocol.

Planned split:

ChannelResponsibility
MembershipSWIM / Lifeguard-style view of who is in the cluster.
Failure detectionSuspicion and dead-peer detection without relying only on configured addresses.
Data disseminationK-fanout gossip for CRDT operations.

K-fanout means a node does not send every update to every peer. Instead, it sends each update to K selected peers. Those peers forward it further. With a good value of K, the cluster gets fast propagation without every operation becoming a full-cluster broadcast.

The planned default is based on cluster size:

K = ceil(log2(N) + c)

where N is the known cluster size and c is a small safety constant.

The roadmap also includes adaptive fanout based on load and RTT, controlled backpressure, periodic anti-entropy as a repair path, and seedable randomness so tests can reproduce gossip behavior.

Why both gossip and anti-entropy

Gossip is the fast path. It spreads new operations quickly.

Anti-entropy is the repair path. It catches up nodes that were offline, partitioned, slow, or unlucky.

Numax needs both because local-first systems must tolerate temporary disconnection. CRDTs define convergence semantics; dissemination moves operations between peers, and anti-entropy repairs missed operations while the required operation and deduplication history remains available.