CRUMB a card from devarno-cloud

Clock Discipline & Peer Sync

weave intermediate 6 min read

ELI5

Clock discipline is like a group of musicians tuning to the same A440 before playing: each musician (peer) listens to the group, calculates how far their pitch (clock) drifts, and adjusts. If any musician is more than 5 semitones off (5 ms), they’re flagged as out-of-sync and their notes arrive in the wrong bar.

Technical Deep Dive

ClockStatus State Machine

stateDiagram-v2
[*] --> OutOfSync: ClockDiscipline new
OutOfSync --> Synchronized: sync_with_peer, abs_offset <= 1000µs
OutOfSync --> Drifting: sync_with_peer, 1000 < abs_offset <= 5000µs
Synchronized --> Drifting: subsequent sync shows 1000 < abs_offset <= 5000µs
Drifting --> Synchronized: sync brings abs_offset back <= 1000µs
Drifting --> OutOfSync: abs_offset > 5000µs
Synchronized --> OutOfSync: abs_offset > 5000µs

Source: mesh-node/src/convergence.rs lines 44–66.

Sync Sequence

sequenceDiagram
participant LocalNode
participant ClockDiscipline
participant RemotePeer


RemotePeer->>LocalNode: heartbeat(peer_time_us)
LocalNode->>ClockDiscipline: sync_with_peer(peer_id, peer_time_us)
ClockDiscipline->>ClockDiscipline: offset = peer_time_us - local_time_us (as i32)
ClockDiscipline->>ClockDiscipline: peer_time_us.insert(peer_id, peer_time_us)
ClockDiscipline->>ClockDiscipline: offset_us.insert(peer_id, offset)
ClockDiscipline->>ClockDiscipline: status = match abs(offset)
ClockDiscipline-->>LocalNode: (status updated)
LocalNode->>ClockDiscipline: consensus_time()
ClockDiscipline-->>LocalNode: median(peer_time_us values)
LocalNode->>ClockDiscipline: adjust_local_time(consensus)

ClockStatus Thresholds

StatusConditionImplication
Synchronizedabs_offset <= 1000 µs (±1 ms)Peer within Theorem 1 budget
Drifting1000 < abs_offset <= 5000 µsWarning: delivery guarantee degrades
OutOfSyncabs_offset > 5000 µs (> ±5 ms)Peer should be quarantined or excluded from fast path

Consensus Time Algorithm

consensus_time() computes the median of all peer_time_us values. If no peers have been synced, it returns local_time_us as a fallback. Median is robust to a single Byzantine peer reporting an extreme timestamp.

within_tolerance() Guard

pub fn within_tolerance(&self, tolerance_ms: i32) -> bool {
    let tolerance_us = tolerance_ms * 1000;
    self.offset_us.values().all(|&offset| offset.abs() <= tolerance_us)
}

Default WeaveConfig passes clock_tolerance_ms: 1. A single peer offset of 2 ms would cause within_tolerance(1) to return false, which should trigger message rejection at the LCB layer.

Relationship to Theorem 1

Theorem 1 (≤8 ms P99) is conditional on clock discipline holding within ±1 ms. If clock_tolerance_ms = 1 but actual offsets reach ±5 ms, the effective delivery bound degrades to ±18 ms. The ClockDiscipline struct provides the measurement machinery; enforcement is the caller’s responsibility.

Key Terms

  • ClockDiscipline → Per-node clock synchronisation state: tracks offsets from each peer, computes consensus, reports sync status
  • ClockStatus → Enum: Synchronized (±1 ms), Drifting (±5 ms), OutOfSync (> ±5 ms)
  • offset_us → BTreeMap<PeerID, i32> of microsecond offsets per peer; > 0 means peer is ahead
  • consensus_time → Median of peer_time_us values; resistant to one Byzantine peer spoofing time
  • within_tolerance → All-peers guard; returns false if any peer’s offset exceeds the configured ms tolerance

Q&A

Q: Why is ClockStatus set on every sync_with_peer call rather than tracking a history? A: The status reflects the most recent sync sample, not a moving average. This is a design simplification — it means a single bad measurement can flip status. Production deployments should wrap ClockDiscipline with an EMA on the offset, similar to LatencyMetric in network.rs.

Q: What does adjust_local_time() do and is it safe to call repeatedly? A: It sets local_time_us to the provided value — a hard override. It does not apply gradual slewing (NTP-style adjtime). Calling it with the consensus time on every tick would cause jumps; callers should implement their own slewing logic.

Q: Does ClockDiscipline need one instance per peer or one per node? A: One per local node. The peer_time_us and offset_us maps hold one entry per remote peer. The single status field reflects the worst sync state across all peers (last-write-wins per sync call).

Examples

From convergence.rs tests (lines 282–305):

  • sync_with_peer(peer, 1000500): offset = +500 µs → Synchronized ✓
  • sync_with_peer(peer, 1003000): offset = +3000 µs → Drifting ✓
  • sync_with_peer(peer, 1010000): offset = +10000 µs → OutOfSync ✓

For the ±1 ms default tolerance: any production environment achieving ±500 µs (LAN with NTP) stays firmly in Synchronized. A WAN peer with ±3 ms offset triggers Drifting but not rejection unless within_tolerance(1) is called by the application.

neighbors on the map