Decentralization Architecture — P2P Pure, Pluggable SyncBackend, CRDT per Domain

Context

ADR-024 establishes operations as the agent's only project-touching path. ADR-025 establishes that authoritative state lives in Rust on the substrate (G1-G10) with NCL as manifestation. ADR-026 establishes the validation architecture with witness-based local verification. All three were written assuming a single-actor, single-node operational model — the substrate runs locally, the operation dispatches locally, the validators verify locally.

The current production reality contradicts this assumption: ontoref-daemon ships with stratum-db (feature-gated) that connects to a remote SurrealDB server for the query cache. The 19 ontoref-onboarded projects share tables in this remote DB via {slug}--{id} prefixes. This is a centralized model: a remote authority holds the materialized state; outage of the DB outage breaks discovery and cross-project query for every consumer.

A deeper structural deficit was identified during the session of 2026-05-26 (this ADR's session): when the ontology lives inside the project's filesystem, cross-project operations or verifications require dragging the entire other project into local scope. This repeats the H7 saturation pattern at a different layer — instead of agent-saturated-with-context, we have verifier-saturated-with-other-project. The substrate's witness mechanism (G5 Merkle proofs) was designed precisely to escape this saturation — a verifier holding only state_root and witness can validate a claim without holding the source. But the witness mechanism is useless if there is no decentralized exchange channel and no separation of ontology from project.

This ADR closes the decentralization question. ADR-028 closes the matching ontology-from-project separation question. Together they reorganize the operational model from "centralized substrate + remote DB" to "P2P pure + per-actor substrate + content-addressed ontology layer + witness-based cross-actor verification".

The decision is irreversible-but-incremental: P2P pure is the architectural commitment; the sync protocol implementation is deferred via a trait abstraction that lets candidates (Iroh, Hypercore, Radicle, NATS) be evaluated when empirical signals demand. The reversibility lives in the backend choice, not in the model.

Decision

Adopt P2P-pure decentralization as the operational model: no central authority node; each actor maintains its own substrate locally; sync is peer-to-peer between actors who explicitly follow each other.

Four load-bearing capabilities that the architecture must support:

1. Multi-actor concurrent with sovereignty per actor — multiple actors (humans + AI agents) operate simultaneously on the same logical project; each holds its own copy; decides whom to follow; no coordination authority.

2. Verifiability without complete state access — any actor can verify a claim by another with only state_root + witness; the substrate's G5 Merkle proofs are the mechanism; ADR-026's validators rely on this.

3. Decentralized discovery — actors find each other via DHT, gossip, or social protocol (Radicle); no central registry. The current ~/.config/ontoref/projects.ncl becomes "peers I follow", not "registry I depend on".

4. Resilience to partition / offline-first — an actor offline retains full local operation; reconnection syncs lazily; no synchronous coordination required for correctness.

SyncBackend trait (pluggable backend, decision deferred):

trait SyncBackend { async fn announce(&self, oplog_head: OpId, public_key: PublicKey); async fn discover_peers(&self) -> Vec<Peer>; async fn fetch_op(&self, peer: &Peer, op_id: OpId) -> Result<Op>; async fn subscribe_peer(&self, peer: &Peer) -> Stream<Op>; }

Candidate impls behind trigger-based assessment (same pattern as ADR-026 CommitmentBackend): - FilesystemSync — default, trivial impl (peers share a filesystem directory; useful for development and testing; not really decentralized but zero-dependency) - IrohSync — n0-iroh; Rust-native; gossip + content-addressed blobs + DHT; aligned with Ed25519 identity (ADR-005) - HypercoreSync — Hypercore via Rust bindings if mature; append-only log primitive aligned with oplog - RadicleSync — git-protocol-over-Radicle for repos; potentially different layer from oplog sync - NatsSync — NATS JetStream as fabric (G7 of the substrate plan, previously skipped)

The decision among these is deferred. The trigger conditions: - cross_actor_sync_required > 0 (i.e., the pilot has reached the point where actual P2P sync is needed; until then, FilesystemSync suffices) - actor_count > 1 (a real second actor exists, not a self-loop) - peer_discovery_latency or sync_throughput become measurable concerns

The fit-assessment (type-fit, not benchmark) for each candidate runs as ADR-026 spike: does the candidate adapt to SyncBackend trait without invasive wrappers. Candidates that do not fit are eliminated.

CRDT per-domain (selective CRDT, not global):

D6 of the old plan ("HLC without CRDT") was a sound default for the substrate's structural correctness but is insufficient when multi-actor concurrent edits target the same cell. This ADR modifies D6 selectively: domains where concurrent edits over the same cell are plausible declare a CRDT merge strategy; other domains retain HLC last-writer-wins.

Identified domains needing CRDT (this session's identification): - Backlog (BacklogItem): multiple actors add/status/done concurrently; MergeStrategy::OrSet for items; MergeStrategy::Lww per status with HLC tiebreak - Ontology nodes (description, edges): collaborative description enrichment + edge addition by multiple contributors; MergeStrategy::TextMerge for description (LSEQ or similar); MergeStrategy::OrSet for edges - QA entries + search bookmarks: append-only accumulators; MergeStrategy::GSet trivially

CrdtMergeStrategy trait (pluggable, similar to SyncBackend and CommitmentBackend):

trait CrdtMergeStrategy { type Item; fn merge(&self, a: Self::Item, b: Self::Item) -> Self::Item; fn merge_witness(&self, ...) -> MergeWitness; }

Default impl: HlcLastWriterWins (preserves current D6 behavior). Per-domain impls declared in the domain's NCL schema.

SurrealDB position under P2P pure:

SurrealDB becomes opt-in via the existing feature flag (cargo build -p ontoref-daemon --features db). The runtime does not depend on SurrealDB; consumers that want SurrealDB's query expressiveness configure it locally. The stratum-db dependency moves from "feature-gated default" to "feature-gated optional with explicit choice". Existing 19 onboarded projects continue to work unchanged; the migration to non-SurrealDB query layer (datalog G6 + ontoref-query) is opt-in per project.

Crucially: SurrealDB is local-only under P2P pure. The connect_remote path becomes connect_local (embedded mode) or is deprecated when the project chooses opt-out.

Constraints

• Hard crates/ontoref-ops or new crate ontoref-sync MUST declare trait SyncBackend with methods announce, discover_peers, fetch_op, subscribe_peer. A default impl FilesystemSync MUST exist that satisfies the trait without external dependencies (suitable for development and single-actor pilot phase).
• Hard crates/ontoref-ops MUST declare trait CrdtMergeStrategy with method merge(a, b) -> Item + merge_witness(...). A default impl HlcLastWriterWins MUST exist. Domain-specific impls (OrSet for backlog, TextMerge for ontology descriptions, GSet for QA accumulators) MUST be declared when the corresponding domain catalog operations are implemented.
• Hard ontoref-daemon MUST build successfully with --no-default-features --features mcp (or equivalent) — i.e. without the db feature that pulls stratum-db. The runtime MUST function for query, mutation, validation, and witness emission without SurrealDB present.
• Hard This ADR engages and synthesizes the voluntary-adoption + internal-coherence-enforced axioms (ADR-025) and the formalization-vs-adoption tension. The synthesis is the per-actor sovereignty model: adoption of P2P is voluntary per project (opt-in tier), but a project that adopts P2P operates within enforced internal coherence (witness-verified ops, signed identity, CRDT per declared domain).

Alternatives considered

• Keep centralized SurrealDB as authoritative query layer; add P2P as optional layer above — rejected: Reproduces the H7 saturation pattern at the storage layer: actors depend on a central authority for queries, defeating the substrate's local-verifiability purpose. The 'optional P2P' layer becomes a courtesy feature rather than the operational backbone; usage in practice falls back to the centralized path. The decision to decentralize must be in the model, not an additional layer.
• Federation model (ActivityPub / Matrix / NATS cluster) — nodes federate but each is a hub — rejected: Federation introduces 'nodes' as a category distinct from 'actors', creating a power asymmetry: node operators control sync topology. This contradicts the per-actor sovereignty capacity. Federation is the right model for some systems (Mastodon, Matrix), but not for ontoref's protocol-of-self-knowledge use case where every actor is logically equivalent.
• CRDT globally for all domains — rejected: Forces CRDT complexity on domains that do not need it. ADR transitions are sequential by their definition (Proposed → Accepted is causally ordered; there is no semantic merge of 'two simultaneous Accepts'). Forcing CRDT here creates a richer data model without operational benefit. The per-domain selective model is the right grain.
• Pick a sync protocol now (e.g. Iroh) and commit before having multi-actor scenarios — rejected: Same mistake as ADR-026 commitment-backend would be without trigger-based deferral. Sync protocol choice without operational scenarios is a guess; the cost of remaking the choice later (after seeing actual sync patterns) outweighs the benefit of committing now. The trait abstraction is cheap; the deferral is honest.
• Remove SurrealDB unilaterally; force all consumers to datalog G6 — rejected: Breaks 19 onboarded projects without consultation. The capacity to use SurrealDB for SurrealQL expressiveness is legitimate for projects that have invested in it. The opt-in demotion lets each project decide; coexistence is sustainable under P2P pure because SurrealDB is local-only.

Related ADRs

ADR-001 · ADR-005 · ADR-013 · ADR-017 · ADR-023 · ADR-024 · ADR-025 · ADR-026