Validation Architecture — Three Planes, First-Class Validators, SLA per Operation, Pluggable Commitment Backend

Context

ADR-024 establishes the operations layer as the agent's only project-touching path. ADR-025 establishes that the authoritative state lives in the verifiable substrate and NCL is its manifestation. Both decisions leave one question deliberately open: how is the coherence of a mutation actually verified? The plan that implements ADR-024 + ADR-025 mentions "validate" as a step inside the operation runtime but does not articulate where validators live, what kinds of validation apply, or what verifiability guarantees are emitted alongside the mutation.

Without this articulation, the operations layer collapses to a thin wrapper: the runtime loads a precondition slice, runs an opaque validate() body inside the operation, and emits a witness that no external party can meaningfully check. This is the failure pattern of the current pre-commit hooks at scale — validators that run inside the producer's process, producing pass/fail signals that consumer projects must trust on faith. The H7 diagnosis is repeated at the validator layer: validators that consume the full state saturate, drift, and lose locality.

Three structural deficits the operations layer alone does not resolve:

1. Validators that require complete knowledge of project state cannot run locally without paying the saturation cost. The substrate's witness mechanism (G5) provides local verifiability for cells, but the operations layer does not exploit it — validators read the live state instead of reading a witnessed slice.

2. The same validator predicate runs on every invocation regardless of contextual relevance. An IPv4-shape validator runs identically for every IP; a "this IP is in this pool given the FSM state" validator needs slice + context. The current pre-commit pattern conflates the two — both run as NuCmd processes loading whatever they want.

3. Validation latency and concurrency requirements are not expressible per operation. A move_fsm_state must validate synchronously (the state transition is the contract); a cross-project impact assertion may legitimately validate eventually (consumer in B asserts the consequence of A's op asynchronously). The current model imposes one mode (sync, pre-commit) on all validations.

The substrate G1-G10 was designed precisely to support locally-verifiable witnesses — bitemporal triples (G4), state root commitments (G5), proof paths for cells (witness API). These primitives have been built and tested but not consumed by an articulated validation layer. This ADR closes that gap.

Decision

Validation is articulated as a first-class architecture, parallel to the operation catalog. The architecture has three load-bearing properties:

Three planes of validation:

Pre-validation — runs after precondition slice load and before the operation body executes. If any pre-validator returns Verdict::Reject, the operation does not execute; the runtime returns a structured block message naming the validator and the unmet predicate.

Inline validation — runs during the operation body. The body may invoke additional validators that depend on values computed mid-execution (e.g. a cross-project query whose result is itself validated). Failures abort the body with rollback at the substrate level (the op is not appended to the oplog).

Post-validation — runs after the operation body has applied to the state. Two modes: - Synchronous: the runtime invokes registered validators inline; the op is committed only if all return Accept. - Asynchronous: validators are external processes consuming the oplog via the existing subscription channel (ontoref-query G6 subscribe()); they emit AttestationOp entries materialized in the same substrate. The state can carry not-yet-attested ops; consumer queries can filter by attestation status.

Two categories of validators:

Structural validators — predicates over input or local data with no dependency on project state or context. Examples: IPv4 shape, CBOR canonical, Ed25519 signature valid, NCL schema contract held. Universally applicable, cheap, deterministic.

Contextual validators — predicates over a slice of project state, possibly parameterized by actor, dimension, or domain. Examples: "this entity is in this pool given the current FSM state", "this actor may invoke this op given their role", "cross-project op respects constraint declared in project B". The slice query is part of the validator declaration; the runtime loads exactly that slice — no more — before running the predicate.

Validators are first-class entities, declared in two combinable forms:

catalog/validators/*.ncl — NCL declaration: id, description, category (Structural | Contextual), slice_query (for contextual), predicate_ref, witness_shape, actor_scope. Reusable across operations.

#[onto_validator(id, kind, slice, ...)] — Rust macro on the validator implementation function. Emits inventory::submit!(ValidatorEntry { ... }) at link time. The validator function signature is fn validate(slice: &Slice, ctx: &ValidationCtx) -> Verdict.

An operation references validators by id in its constraints field; alternatively, an operation may declare inline constraints when the validator is single-use and reuse is not anticipated. Both forms coexist.

SLA per operation:

Each operation declares a validation_sla field: 'Synchronous — all validators (pre, inline, post) run sync; op only commits on full Accept 'EventualWithin(Duration) — pre/inline sync; post validators run async with deadline; consumers query attestation status with as_of clauses 'Background — pre/inline sync; post validators run background; no deadline; ops are eventually-attested

The runtime applies the declared mode; mixing modes within a single op is not permitted (to keep dispatch unambiguous).

Pluggable commitment backend:

The substrate's commitment layer (G5, currently hand-rolled binary Merkle) is abstracted behind a CommitmentBackend trait with methods apply, root, witness, verify_witness. The G5 hand-rolled implementation becomes the BinaryMerkle impl. The trait permits alternative impls (NOMT, jmt, sparse-merkle-tree) without modifying call sites.

The choice of backend is deferred: the BinaryMerkle impl is the default and proven (10 tests, clippy clean). A trigger-based spike runs only when empirical signals from the pilot indicate a backend with different trade-offs may be beneficial:

- validate_latency_p99 > 500ms (sustained) - state_cells_count > 100k - commit_root_time_p99 > 100ms (sustained) - daemon memory pressure > 1GB on substrate alone

A one-off type-fit assessment (1-2 hours, no benchmark) checks whether candidate backends (NOMT 1.0.4, jmt latest, sparse-merkle-tree latest) adapt to the CommitmentBackend trait without invasive wrappers. Backends that do not fit are eliminated without benchmark cost. The assessment is optional and may be deferred indefinitely; absence of assessment leaves BinaryMerkle as the conscious choice.

Local verifiability:

Every operation emits a witness containing op_id, post-apply state_root, signed by the actor (ADR-005 keypair model), plus Merkle proof paths for cells the operation's effects touched. An external verifier holding only the state_root + witness can check predicate(slice, witness, state_root) for any contextual validator without needing access to the full state. This is the Sigstore-like property the substrate was built to enable.

Constraints

• Hard crates/ontoref-ops MUST declare the trait Validator with method fn validate(&self, slice: &Slice, ctx: &ValidationCtx) -> Verdict; crates/ontoref-commit MUST refactor the existing commit module to extract trait CommitmentBackend with methods apply, root, witness, verify_witness. BinaryMerkle (the existing hand-rolled impl) MUST be the default impl.
• Hard Every operation in catalog/operations/*.ncl MUST declare a validation_sla field with one of the values 'Synchronous, 'EventualWithin(duration), or 'Background. Operations that lack the field are rejected by the catalog schema at parse time.
• Hard .ontoref/config.ncl MUST declare an ops.commitment_backend field with enum value 'BinaryMerkle | 'Nomt | 'Jmt | 'SparseMerkleTree. Default value is 'BinaryMerkle. The runtime selects the impl based on this field.
• Soft docs/validation/spike-triggers.md MUST exist documenting the empirical thresholds that activate the commitment-backend spike (validate_latency_p99, state_cells_count, commit_root_time_p99, daemon memory pressure). The thresholds MUST be revisable via PR; the document records the current values plus rationale.
• Hard This ADR engages and synthesizes the named tensions ontology-vs-reflection, formalization-vs-adoption, and the internal-coherence-enforced axiom; the synthesis state and direction of motion MUST be discoverable in the decision and rationale.

Alternatives considered

• Single-plane validation: pre-only (as in current pre-commit hooks) — rejected: Pre-only cannot handle validators that depend on values computed during op execution (inline) and cannot handle expensive validators that would block throughput unacceptably (async post). The current model fails precisely because it has only this one plane — the H7 diagnosis traces some of its dysfunction to validators that try to do everything pre.
• Validators as Rust traits only, no NCL declaration — rejected: Loses the Self-Describing property: validators become invisible to non-Rust consumers (NCL readers, GraphQL clients, MCP agents querying catalog). Also loses the ability for validators to be cross-project queryable — a project that wants to know what validators apply to its mutations would need to read source code rather than declarative artifacts.
• Validators as NCL only, no Rust impl bridge — rejected: NCL is declarative; predicate evaluation logic must run somewhere. Forcing all predicates into Nickel contract syntax limits expressiveness severely (no graph traversal, no FFI, no I/O). The dual-write (NCL declares, Rust implements) is the only way to satisfy both Self-Describing (NCL is the catalog) and effective implementation (Rust runs the code).
• Force synchronous validation for all ops (no SLA per op) — rejected: Equates correctness with synchronous correctness. Many legitimate validators (cross-project, statistical, graph-traversal) cannot reasonably run synchronously without throttling the runtime. The sync-only model would either reject these validators (impoverishing the architecture) or require their bodies to be approximations of the real predicate (sacrificing correctness for latency).
• Commit immediately to NOMT or jmt without spike or trait abstraction — rejected: No prior implementation experience exists. Either choice without data is gambling. The trait abstraction costs minimal effort (the G5 implementation needs trait extraction; the other backends, if ever evaluated, just implement the trait). Deferring the decision preserves optionality at near-zero cost; committing now sacrifices optionality for no benefit.
• Defer all of validation architecture to a later ADR, ship operations layer with thin validate() — rejected: The operations layer without articulated validators is the failure pattern this entire trajectory is trying to escape. Shipping ADR-024 + ADR-025 + the operations-layer plan without ADR-026 would produce a runtime that has the surface of validation (a validate step in the dispatch pipeline) but the substance of the current pre-commit model (opaque body running ad-hoc checks). Validation architecture is co-constitutive with operations architecture; they share the load-bearing.

Related ADRs

ADR-001 · ADR-002 · ADR-005 · ADR-007 · ADR-014 · ADR-023 · ADR-024 · ADR-025