Skip to content

nx-net

nx-net owns everything below the sync layer: TCP connections, TLS/mTLS handshakes, wire message framing, serialization format negotiation, peer slot management, and the cooperative shutdown of all network tasks. It surfaces events upward to nx-core via an async channel.

It depends on nx-sync for Op and NodeId types. It does not depend on nx-core or nx-store.


Responsibilities

ResponsibilityWhere
TCP listener and inbound connection handlingnode.rs - Node::start_listener, handle_incoming
Outbound connections and handshakenode.rs - Node::connect_to_peer
Encode-once, generation-bound History framesnode.rs - Node::prepare_history_to_addr, Node::send_prepared_history, PreparedHistory, ConnectionGeneration
Correlated control timing excluding connection setupnode.rs - Node::control_request_timed_tracked, ControlTiming; existing request wrappers remain available
Identity-guarded topology slot releasenode.rs - Node::disconnect_peer; cancels the connection generation before event delivery
Wire message framing (length-prefixed)node.rs - read_message, write_message
Serialization format negotiationnode.rs - negotiate_serialization_format
TLS/mTLS acceptor and connectortls.rs - TlsConfig::accept_stream, TlsConfig::connect_stream
NodeId binding to TLS certificatetls.rs - derive_protocol_node_id_from_cert
Peer slot enforcement (semaphore)node.rs - connection_slots, ensure_peer_slot_available
Broadcast and targeted op sendnode.rs - Node::broadcast_ops, Node::send_ops_to_addr
Anti-entropy pull requestsnode.rs - Node::send_pull_since_to_addr
Authenticated one-shot bootstrap exchangebootstrap.rs, node.rs - BootstrapClient, inbound bootstrap handling
Bounded bootstrap advertisement cachebootstrap.rs - BootstrapServerConfig, BootstrapServer
Cooperative shutdown via watch channelnode.rs - Node::shutdown, shutdown_tx
Wire message types and encode/decodemessage.rs - Message, MessageKind
Peer state trackingnode.rs - PeerConnection, peer.rs - PeerInfo, PeerState
Error typeserror.rs - NetError

Node

Node is the main struct. The sync manager in nx-core creates one, starts it, and consumes events from it.

pub struct Node {
config: NodeConfig,
peers: Arc<RwLock<HashMap<String, PeerConnection>>>,
event_tx: mpsc::Sender<NodeEvent>,
event_rx: Option<mpsc::Receiver<NodeEvent>>,
shutdown_tx: watch::Sender<bool>,
connection_slots: Arc<Semaphore>,
tasks: Arc<Mutex<Vec<JoinHandle<()>>>>,
}

NodeConfig

NodeConfig::new(node_id, "0.0.0.0:9000")
.with_peers(vec!["127.0.0.1:9001".into()])
.with_tls(tls_config)
.with_max_peers(64)
.with_max_message_size(16 * 1024 * 1024)
.with_socket_timeout(Duration::from_secs(30))
.with_serialization_format(SerializationFormat::Bincode)
.with_event_channel_capacity(1024)

NodeConfig::validate() checks limits before channel/semaphore allocation or network startup: max_peers cannot exceed Tokio’s semaphore capacity, event_channel_capacity must be positive and within that capacity, and socket_timeout must be positive and form a representable deadline. max_peers = 0 is valid and disables connection admission.

Prefer Node::try_new(config), which returns NodeConfigError for invalid limits without binding sockets. The legacy infallible Node::new(config) retains its 0.1.4 signature and behavior for source compatibility. Validation does not establish that a listen address can be bound or a peer reached.

Node lifecycle

Node construction, receiver ownership and listener startup precede independent data and one-shot control work.

MethodResponsibility
try_new(config)Validate without binding sockets.
try_new_with_bootstrap_server(config, bootstrap)Also configure the optional one-shot bootstrap service.
take_event_receiver() / take_control_receiver()Transfer ownership of data events / control requests to consumers.
start_listener()Bind TCP, own the listener task and return its actual socket address.
connect_to_peer(addr)Dial, apply TLS/handshake policy and admit a data connection.
connection_info(addr)Inspect transport direction, identity and verification.
announce_bootstrap_endpoint(addr) / withdraw_bootstrap_endpoint()Manage the local bootstrap advertisement.
broadcast_history(message) / send_history_to_addr(addr, message)Send sequenced history over data connections.
prepare_history_to_addr() / send_prepared_history_with_timeout()Encode for the negotiated connection generation, then send within a writer budget.
broadcast_ops() / send_ops_to_addr() / send_pull_since_to_addr()Legacy unsequenced codec helpers; rejected by the current sync path.
shutdown()Stop admissions, retire owned tasks and close connections.

NodeEvent

Events emitted to the sync manager via mpsc::Sender<NodeEvent>:

pub enum NodeEvent {
HistoryReceived { from: NodeId, addr: String, message: HistoryMessage },
OpsReceived { from: NodeId, ops: Vec<Op> },
PullRequested { from: NodeId, addr: String, since_op_id: Option<String> },
PeerConnected { node_id: NodeId, addr: String, peers_connected: usize },
PeerDisconnected{ node_id: NodeId, addr: String, peers_connected: usize },
}

take_event_receiver() must be called once before start_listener. The receiver is moved out of the Node so the sync manager owns it.


Connection flow

Data connection admission

A dialer and listener exchange Hello and HelloAck before admitting the data connection.

The dialer rejects duplicate endpoint attempts and acquires bounded attempt and connection slots before opening TCP/TLS. Both sides validate protocol version, serialization format and peer identity before admitting a data connection. TLS certificate binding and the allowlist apply when configured; a plaintext handshake does not authenticate the claimed identity.

Inbound routing

The listener applies the socket deadline, optional TLS and a bounded first-frame read, then routes the request by its opening message:

Opening messageProcessingResult
HelloValidate version, negotiate format, verify identity and acquire data admission capacity.HelloAck, active peer registration and a data read loop.
BootstrapHelloValidate identity, bootstrap service, cluster and request bounds.BootstrapAck, then close; no active data peer registration.
ControlHelloValidate version, identity and control owner; use independent bounded control capacity.Correlated ControlAck, then close; no active data peer registration.

PeerConnectionInfo into the peers map 12. emit PeerConnected event and spawn the read loop

### Inbound (listener)

handle_incoming(stream, addr, context)

  1. TLS accept (if configured), capture peer_cert
  2. receive Hello or BootstrapHello
  3. validate protocol version and negotiate_serialization_format
  4. reject the local NodeId, then perform TLS identity binding and allowlist checks 5a. normal: send HelloAck, insert PeerConnection, emit PeerConnected, run read_loop 5b. bootstrap: validate service/cluster/request, send BootstrapAck, close without peer admission
`PeerConnectionInfo` keeps the TCP transport address separate from the outbound
endpoint that was dialed. It also records inbound/outbound direction and whether
the handshake NodeId was certificate-bound or unverified. These are runtime
facts only and do not change the wire format.
---
## Wire format
Every message is framed as:
![A four-byte big-endian length precedes the format tag and encoded message.](/numax/diagrams/reference/wire-frame.svg)
- Length is the total of `format byte + payload`, encoded as big-endian `u32`.
- Format byte: `0x01` = JSON, `0x02` = bincode.
- Payload is the serialized internal protocol message. For normal replication
frames its layout is byte-for-byte equivalent to the public `Message`; the
bootstrap-only variants remain private so the exhaustive public enums retain
their 0.1.4 source-compatible shape.
`PROTOCOL_VERSION = 9`. Version mismatch during a recognized handshake causes a structured
`WireError::ProtocolMismatch` and immediate disconnect.
### Protocol message variants
| Variant | Direction | Purpose |
|---|---|---|
| `Hello` | dialer -> listener | Open handshake: node identity, protocol version, supported formats |
| `HelloAck` | listener -> dialer | Accept handshake: protocol version and selected format |
| `PushOps` | both | Legacy unsequenced operations; rejected by the sync manager |
| `PushOpsAck` | unused by current exchange | Encodable reception-count variant; not emitted or handled as an acknowledgement |
| `PullSince` | both | Legacy UUID-based replay request; rejected by the sync manager |
| `Ping` / `Pong` | both | Keepalive |
| `Error` | both | Structured wire error: `ProtocolMismatch`, `OpRejected`, `RateLimited`, `NotAuthorized`, `Internal` |
| `BootstrapHello` *(private wire variant)* | client -> seed | One-shot identity, format, cluster, optional endpoint advertisement and result limit |
| `BootstrapAck` *(private wire variant)* | seed -> client | Seed identity, format, cluster, bounded candidates and lease |
| `History` *(private wire variant)* | both | Sequenced pushes, summaries, bounded range and baseline recovery |
| `ControlHello` / `ControlAck` *(private wire variants)* | one-shot RPC | Membership exchange and correlated probing with independent control budgets |
The private `History` variant carries `HistoryMessage` exchanges.
Range requests and responses carry a required `request_id`; replies from an
expired or superseded attempt cannot complete a current request.
Use `Node::broadcast_history` / `send_history_to_addr`; received messages become
`NodeEvent::HistoryReceived`. The sync manager accepts sequenced envelopes and
rejects legacy `PushOps` / `PullSince` replay. The old public codecs remain
available for compatibility and inspection, without granting legacy replay a
new deduplication guarantee.
### Delivery and acknowledgement boundary
`broadcast_history` and `send_history_to_addr`, like the legacy
`broadcast_ops` and `send_ops_to_addr`, wait for local transport writes and
stream flushes, not for remote CRDT application or storage flushes.
`broadcast_ops` can succeed with zero connected peers; a failed broadcast may
already have delivered to some peers. Neither result identifies durable remote
copies.
Receiving `History` emits an internal `NodeEvent::HistoryReceived` for `nx-core`
to validate and apply. Legacy `PushOps` emits `NodeEvent::OpsReceived`, which
the sync manager rejects before applying state. Neither sends `PushOpsAck`.
Incoming `PushOpsAck` messages fall
through the read loop's unhandled-message branch; no receipt count is used to
confirm replication. `HelloAck` and `Pong` concern handshake and ping handling,
respectively, and do not acknowledge CRDT operations.
The [acknowledgement contract](/numax/design/peer-discovery/#acknowledgement-semantics)
distinguishes batch application, flush-confirmed local acceptance, and remote
replication. The current API provides no remote durability acknowledgement.
### WireError semantics
| Error | Retry policy | Meaning |
|---|---|---|
| `ProtocolMismatch` | Fatal | Different wire contracts. Upgrade/downgrade one side before reconnecting. |
| `NotAuthorized` | Fatal for that peer/config | Credentials, certificate identity, or allowlist must change before retrying. |
| `RateLimited` | Retryable | Back off. Use `retry_after_ms` when present, otherwise use normal reconnect backoff. |
| `OpRejected` | Fatal for those ops | Do not resend the same rejected ops unchanged. Current generic error handling closes the peer connection. |
| `BootstrapRejected` *(private wire error)* | Fatal for that request | Bootstrap is disabled or its cluster, advertisement or request bounds are invalid. |
| `Internal` | Retryable with backoff | Treat as transient unless it repeats; record metrics/logs. |
The configured-peer reconnect loop uses this policy: fatal wire errors stop
automatic reconnect for that peer, `RateLimited.retry_after_ms` is honored up to
the configured reconnect max delay, and retryable errors keep the normal
exponential backoff.
### Serialization format negotiation
When a bincode node connects to a JSON-only debug node:
| Stage | Value |
| --- | --- |
| Dialer advertises | `[Bincode, Json]`, preferred `Bincode` |
| Listener supports | `Json` |
| Negotiated format in `HelloAck` | `Json`, the first common format in the dialer's list |
A `--debug-protocol` node (JSON only) always negotiates JSON with any peer.
A standard node advertises both and prefers bincode.
### Bootstrap transport
`BootstrapClient::query(seed, request)` opens a bounded, one-shot connection,
sends `BootstrapHello`, authenticates the `BootstrapAck` seed identity and
returns `BootstrapResponse`. `BootstrapClientConfig` reuses `NodeId`, optional
`TlsConfig`, message-size, socket-timeout and serialization controls. Its
defaults permit one concurrent query, at most 128 returned candidates and a
maximum accepted candidate TTL of 300s.
The server is enabled through `Node::try_new_with_bootstrap_server`. Keeping the
bootstrap policy outside `NodeConfig` preserves source compatibility with 0.1.4
struct literals. By default it
retains at most 1024 authenticated requester advertisements for 60s and returns
at most 128 candidates. Its own advertised endpoint is returned first, followed
by cached requester endpoints in stable insertion order; responses are
deduplicated and exclude the current requester. A request without an advertised
endpoint withdraws that requester's cache entry.
Cluster IDs, endpoint strings, response count and leases are validated on both
sides. A bootstrap socket holds an inbound connection slot while the request is
processed but is never inserted into the active peer map, never enters a read
loop and never emits `PeerConnected`. Suggestions in `BootstrapResponse` are
not authenticated identities; the normal dialer must authenticate each one in
a separate `Hello`/`HelloAck` exchange.
---
## TLS and mTLS
TLS is optional. When `TlsConfig` is provided:
- Outbound: `TlsConfig::connect_stream(tcp, server_name)` via `tokio-rustls`.
- Inbound: `TlsConfig::accept_stream(tcp)`.
- Both sides extract the peer DER certificate from the completed TLS session.
### TlsConfig
```rust
pub struct TlsConfig {
pub cert_path: Option<String>, // this node's PEM cert
pub key_path: Option<String>, // this node's PEM key
pub ca_path: Option<String>, // CA cert for peer verification (enables mTLS)
pub allowed_peers: Option<HashSet<String>>, // optional allowlist of NodeId strings
pub insecure: bool, // skip cert verification (dev only)
}

NodeId binding

After TLS handshake, the node verifies the claimed NodeId in Hello/HelloAck against the identity derived from the peer’s X.509 certificate:

Derive the protocol NodeId from certificate SPKI using SHA-256, 16-byte truncation and lowercase hexadecimal.

If the claimed NodeId does not match the cert-derived one, the connection is rejected with NetError::TlsError("node_id mismatch ...").

Allowlist enforcement

When TlsConfig.allowed_peers is set, the cert-derived NodeId is checked against the set. A peer not in the allowlist is rejected after the TLS handshake, before ops are exchanged.

Test utilities

TestPki in tls.rs generates an in-memory CA + two node certs for use in tests:

let pki = TestPki::generate().unwrap();
let node1_cfg = pki.node1_config(); // TlsConfig for node 1
let node2_cfg = pki.node2_config(); // TlsConfig for node 2

Peer slot management

Peer capacity is enforced with a tokio::sync::Semaphore initialized to max_peers.

  • Inbound: try_acquire_owned() at accept time - the permit is held in PeerConnection._slot. If the semaphore is exhausted, the connection is dropped before the TLS/handshake cost.
  • Outbound: try_acquire_owned() before TCP connect - fails fast with PeerLimitReached.

The permit is dropped when the PeerConnection is removed from the peers map (on disconnect or shutdown).


Cooperative shutdown

Shutdown serializes callers, closes admissions and signals owned tasks through a tokio::sync::watch channel. All network tasks share one total three-second grace period; the deadline is not renewed for each task. Remaining tasks are aborted and joined before peer writers are released.

Network tasks share a single three-second grace period before residual tasks are aborted and joined.

Read loops check the shutdown signal on every iteration via tokio::select!. Listener loop checks it between accept calls. This avoids waiting for socket timeouts during clean shutdown.


Error types

pub enum NetError {
Io(std::io::Error),
Serialization(serde_json::Error),
BinarySerialization(wincode::WriteError),
BinaryDeserialization(wincode::ReadError),
ConnectionFailed(String),
PeerDisconnected(String),
InvalidMessage(String),
Wire(WireError),
MessageTooLarge { len: usize, limit: usize },
Timeout,
ChannelClosed,
TlsError(String),
PeerNotAllowed(String),
PeerLimitReached(usize),
NodeIdMismatch { expected: String, got: String },
}

Configuration validation uses the additive, #[non_exhaustive] NodeConfigError. Bootstrap-only construction and queries use the additive, #[non_exhaustive] BootstrapError, which distinguishes invalid configuration, query concurrency, authenticated rejection, invalid responses, node configuration and transport failures. NetError retains exactly its 0.1.4 variants so existing exhaustive matches remain source-compatible.


Defaults

ConstantValueDescription
DEFAULT_MAX_PEERS64Maximum simultaneous peers
DEFAULT_MAX_MESSAGE_SIZE16 MiBMaximum wire message size
DEFAULT_SOCKET_TIMEOUT30sRead/write timeout per operation
DEFAULT_EVENT_CHANNEL_CAPACITY1024Event channel buffer size
DEFAULT_BOOTSTRAP_CACHE_CAPACITY1024Seed-side advertised endpoint cache
DEFAULT_BOOTSTRAP_RESPONSE_CAPACITY128Results returned by one bootstrap exchange
MAX_BOOTSTRAP_RESPONSE_CAPACITY4096Hard limit for one bootstrap response, not the combined discovery snapshot
DEFAULT_BOOTSTRAP_CANDIDATE_TTL60sSeed-side advertisement lease
DEFAULT_MAX_CONCURRENT_BOOTSTRAP_QUERIES1Simultaneous queries per bootstrap client
TASK_SHUTDOWN_GRACE3sCooperative shutdown grace per task

Test coverage

Tests live in node.rs and message.rs (#[cfg(test)]), plus integration tests in tests/.

TestWhat it covers
test_node_configdefault values and builder
node_config_allows_custom_peer_limitwith_max_peers
negotiation_prefers_local_format_when_peer_supports_itbincode-bincode -> bincode
negotiation_falls_back_to_json_for_debug_peerbincode node + json-only peer -> json
negotiation_rejects_empty_peer_formatsno common format -> None
peer_slot_limit_rejects_new_peer_when_fullsemaphore exhausted -> PeerLimitReached
peer_slot_limit_allows_replacing_same_addrre-connect to same addr is allowed
mark_peer_failed_returns_updated_connected_countpeer state transition + count
track_task_prunes_finished_handles_before_pushtask list stays clean
is_connected_addr_tracks_connected_stateonly Connected peers return true
read_message_rejects_payload_over_configured_limitMessageTooLarge
read_message_times_out_waiting_for_lengthTimeout on stalled read
connect_to_peer_times_out_during_handshakeTimeout during plain handshake
connect_to_peer_times_out_during_tls_handshakeTimeout during TLS handshake
connect_to_peer_rejects_protocol_version_mismatchold version in HelloAck
incoming_rejects_protocol_version_mismatchold version in Hello
protocol_v5_binary_encoding_matches_bincode_golden_hashesstable binary encoding for normal and bootstrap messages
one_shot_query_returns_candidates_without_registering_a_peerbootstrap response without active peer admission or events
cluster_mismatch_is_rejected_without_populating_the_cachecluster isolation before advertisement caching
incoming_idle_handshake_consumes_peer_slotslot held before handshake completes
incoming_idle_tls_handshake_releases_peer_slot_after_timeoutslot released after timeout
active_peer_shutdown_does_not_wait_for_socket_timeoutcooperative shutdown timing
incoming_ping_gets_pong_responsePing/Pong keepalive
bincode_node_negotiates_json_with_debug_peercross-format negotiation E2E
message_roundtrip_json / message_roundtrip_bincodeencode/decode roundtrip
rejects_unknown_serialization_formatunknown format byte -> InvalidMessage
Terminal window
cargo test -p nx-net

Use this page together with the sync model and runtime docs: