feat: write-verdict foundations — rejection spike, error taxonomy, uniform /health
- commit
- 3efb3159ec58cb56fca6e91edf3dfad211f523d9
- parent
- 4cdab00eb626f5bb430d808dec32644050c3b894
- parent
- 7a7975f7c4b6719d1d9e3dd721d92ffa46d30a62
- author
- FelineStateMachine <dami.etoile@gmail.com>
| M | docs/ | 2 | + |
| M | docs/ | 40 | ++++- |
| A | docs/ | 79 | ++++++++++ |
| M | mod.ts | 6 | + |
| A | src/ | 52 | ++++++ |
| A | tests/ | 147 | ++++++++++++++++++ |
| A | tests/ | 28 | +++ |
| A | tests/ | 249 | ++++++++++++++++++++++++++++++ |
@@ -31,6 +31,8 @@ What the node gives you:
3131 data: [Provision a store](provision-a-store.md).3232 - The mental model behind all of it — slices, shared stores, and why apps can't clobber each other:3333 [Sliceable apps and shared stores](sliceable-apps-and-shared-stores.md).34+- What the store guarantees when it refuses a write, and how apps observe it:35+ [Write verdicts](write-verdicts.md).3436 3537 The app-side enrollment flow (pasting a ticket into a lofi app) is part of the framework docs:3638 [Sync and recovery](/docs/sync-and-recovery).@@ -9,18 +9,42 @@ loopback-only Jazz server. This page is the route-level reference for a **ticket
99 1010 ## Routes1111 12-| Route | Ticket | Behavior |13-| ------------------------------------- | ----------- | ------------------------------------------------------------------------- |14-| `GET /health` | none | Liveness; proxied to Jazz. `200`, or `502` when the store is unreachable. |15-| `GET /t/<secret>/store-status` | any scope | Metadata-only schema preflight (below). |16-| `/t/<secret>/apps/<appId>/ws` | any scope | WebSocket sync; re-originated toward Jazz with subprotocol forwarded. |17-| `/t/<secret>/apps/<appId>/…` | any scope | Catalogue reads and other app routes, prefix-stripped and proxied. |18-| `/t/<secret>/apps/<appId>/admin/…` | `provision` | Store administration; the node's admin secret is injected server-side. |19-| `POST /t/<secret>/derive-sync-ticket` | `provision` | Mints a parent-linked, sync-scoped ticket (below). |12+| Route | Ticket | Behavior |13+| ------------------------------------- | ----------- | ---------------------------------------------------------------------- |14+| `GET /health` | none | Liveness (below). `200`, or `502` when the store is unreachable. |15+| `GET /t/<secret>/health` | any scope | Same liveness under a ticket base; `401` once the ticket is revoked. |16+| `GET /t/<secret>/store-status` | any scope | Metadata-only schema preflight (below). |17+| `/t/<secret>/apps/<appId>/ws` | any scope | WebSocket sync; re-originated toward Jazz with subprotocol forwarded. |18+| `/t/<secret>/apps/<appId>/…` | any scope | Catalogue reads and other app routes, prefix-stripped and proxied. |19+| `/t/<secret>/apps/<appId>/admin/…` | `provision` | Store administration; the node's admin secret is injected server-side. |20+| `POST /t/<secret>/derive-sync-ticket` | `provision` | Mints a parent-linked, sync-scoped ticket (below). |2021 2122 The `<secret>` is a 43-character base64url path segment; the gate compares digests in constant time,2223 strips the prefix, preserves the query string, and forwards.2324 25+## health26+27+```jsonc28+// 200 — the store answered29+{ "status": "healthy" }30+```31+32+One liveness contract, every connection mode:33+34+- **Open mode** — Jazz serves `/health` directly on the public URL.35+- **Ticket mode** — the gate proxies it, unauthenticated at the top level (reachable before36+ enrollment) and under any valid ticket base, so `serverUrl`-relative polling works unchanged. A37+ revoked ticket gets `401` on its own base while the top level stays `200`: "node up, access gone"38+ stays distinguishable from "node unreachable".39+- **Peer mode** — the request rides the iroh tunnel to the peer's Jazz like any other HTTP.40+41+Responses are cheap in all modes (one loopback hop at most) and never hang on a dead store: the gate42+answers `502` when Jazz is unreachable. The client contract is composition, not a new primitive — an43+app derives its single connection observable from WebSocket lifecycle events (connect, close, auth44+failure) plus periodic `/health` polls, and uses it to label pending writes `offline` rather than45+leaving them silently waiting. Verdict semantics for those pending writes are on the46+[write verdicts](write-verdicts.md) page.47+2448 ## store-status2549 2650 ```jsonc@@ -0,0 +1,79 @@
1+# Write verdicts2+3+<!-- Source: FelineStateMachine/lofi-node src/verdict.ts, tests/write_verdict_test.ts;4+ jazz-tools batch-fate surface at the pinned alpha. -->5+6+A lofi app treats every mutation as a durable write that eventually settles: accepted into the7+store, or rejected by it. Rejection is what compensation logic (refunds, releasing holds) hangs on,8+so its semantics are a contract of the node, not an implementation detail. This page states that9+contract as it holds at the pinned Jazz alpha, verified by the lofi-node test suite.10+11+## Where verdicts come from12+13+Every write becomes a batch with a client-generated batch id. The store adjudicates the batch and14+the client runtime records the outcome as the batch's fate. Two client surfaces carry a rejection:15+16+- **Awaited writes.** `write.wait({ tier })` rejects with a typed error carrying the originating17+ `batchId`, a machine-readable `code`, and a human-readable `reason`.18+- **Un-awaited writes.** A fallback `onMutationError` event fires with the same `code`/`reason` plus19+ the full batch record (`batch.batchId`, `batch.latestSettlement`), for writes whose awaiter no20+ longer exists.21+22+Both surfaces correlate to the originating write by batch id. The `reason` string is diagnostics23+only; its wording changes between Jazz versions and must never be matched on.24+25+## Local refusal is not a verdict26+27+The client runtime enforces the policy it currently knows synchronously, at the call site: a write28+that the local policy denies throws immediately and no batch ever exists. The store only adjudicates29+writes made under a **stale** policy — accepted locally, then denied by a store whose permissions30+tightened after the client last synced them. Author-facing effect handlers therefore see two31+distinct failure shapes: a synchronous throw (a programming or policy error against known state) and32+an asynchronous `rejected` verdict (the store overruled a write it had never seen).33+34+## Rejection properties35+36+Three properties hold, and the node's harness keeps them pinned:37+38+1. **Correlation.** A permission-denied write surfaces `code: "permission_denied"` with the39+ originating write's batch id, on both verdict surfaces.40+2. **Re-derivability.** A write unacknowledged at disconnect is re-proposed on reconnect, and the41+ rejection fires then. A disconnect cannot swallow a verdict; it only defers it.42+3. **Determinism.** Waiting again on a rejected batch re-derives the same verdict, across further43+ reconnects. An independent device issuing the same write against the same store state derives the44+ same code.45+46+## At-least-once, exactly-once: the consumer's duty47+48+Delivery is at-least-once from the store's point of view (unacked batches are re-proposed on every49+reconnect) but the fallback event fires **once per runtime lifetime** — a later reconnect does not50+re-fire events for batches already settled. A write journal must therefore:51+52+- persist the verdict when it first arrives, keyed by batch id;53+- treat any duplicate delivery (for example after an app restart) as idempotent by batch id;54+- re-derive a verdict on demand by waiting on the batch, not by expecting the event again.55+56+Compensation handlers keyed by batch id run at most once even when the verdict is observed twice.57+58+## The taxonomy: permanent or transient59+60+A consumer classifies every mutation-error code into one of two actions, using the classification61+the package exports (`classifyMutationError`, `MUTATION_ERROR_CLASSES`):62+63+| Class | Meaning | Consumer action |64+| ----------- | ---------------------------------------------------- | ------------------------------------------- |65+| `permanent` | The store adjudicated the write and never accepts it | Verdict is `rejected`; compensation may run |66+| `transient` | No adjudication happened (unreachable, timeout) | Write stays pending; keep waiting |67+68+The pinned alpha emits exactly one rejection code over the sync protocol: `permission_denied`,69+classified permanent. Codes not in the registry classify as transient by design — an unrecognized70+code must never trigger irreversible compensation, because waiting is recoverable and a wrong71+`rejected` verdict is not. As the upstream code space grows (validation, schema enforcement), codes72+enter the registry here rather than in every consumer.73+74+## Pending is not offline75+76+A transient-pending write needs one more input before an app can honestly say "waiting for your sync77+node": whether the node is reachable at all. That signal is the78+[`/health` endpoint](http-surface.md#health), available in every connection mode, composed79+client-side with WebSocket lifecycle events into a single connection observable.@@ -46,3 +46,9 @@ export { decodeAppTicket, encodeAppTicket, looksLikeAppTicket } from "./src/appt
4646 export type { AppTicket } from "./src/appticket.ts";4747 export { CLOSE_TICKET_REVOKED } from "./src/gate.ts";4848 export { MeshUnavailableError } from "./src/errors.ts";49+export {50+ classifyMutationError,51+ isPermanentMutationError,52+ MUTATION_ERROR_CLASSES,53+} from "./src/verdict.ts";54+export type { MutationErrorClass } from "./src/verdict.ts";@@ -0,0 +1,52 @@
1+// Write-verdict taxonomy: classify mutation-error codes so consumers act on2+// the code, never on the human-readable reason string.3+//4+// A Jazz write that reaches the sync server settles into a per-batch fate;5+// when that fate is a rejection it carries a machine-readable `code` plus a6+// free-text `reason`. The reason is diagnostics only — its wording changes7+// between Jazz versions. The code is the contract, and this module is the one8+// place that maps codes onto the two actions a consumer can take:9+//10+// - `permanent`: the store has adjudicated the write and will never accept it11+// (permission denied, and any future validation/schema-enforcement codes).12+// The write's verdict is `rejected`; compensation may run.13+// - `transient`: the outcome is not adjudicated (store unreachable, timeout,14+// ticket hiccup). The write stays pending; keep waiting and retrying.15+//16+// Codes not in the registry classify as `transient` by design: an17+// unrecognized code must never trigger irreversible compensation. Waiting is18+// recoverable; a wrong `rejected` verdict is not.19+20+/** How a consumer must treat a mutation-error code. */21+export type MutationErrorClass = "permanent" | "transient";22+23+/**24+ * The registry of known mutation-error codes.25+ *26+ * The pinned Jazz alpha emits exactly one rejection code over the sync27+ * protocol: `permission_denied` (policy denies the operation on the table for28+ * this session). Writes that violate the CURRENT locally-known policy or29+ * schema never produce a code at all — the client runtime refuses them30+ * synchronously at the call site, before any batch exists.31+ */32+export const MUTATION_ERROR_CLASSES: Readonly<Record<string, MutationErrorClass>> = {33+ permission_denied: "permanent",34+};35+36+/**37+ * Classify a mutation-error code from a rejected batch fate — the `code` on38+ * `MutationErrorEvent`, on `PersistedWriteRejectedError`, or on a39+ * `BatchFate` of kind `rejected`.40+ *41+ * Unknown codes classify as `transient`: the consumer keeps the write42+ * pending rather than compensating on a code it cannot interpret.43+ */44+export function classifyMutationError(code: string): MutationErrorClass {45+ return MUTATION_ERROR_CLASSES[code] ?? "transient";46+}47+48+/** True when {@link classifyMutationError} classifies `code` as permanent —49+ * the write is `rejected` and compensation may run. */50+export function isPermanentMutationError(code: string): boolean {51+ return classifyMutationError(code) === "permanent";52+}@@ -0,0 +1,147 @@
1+// The connection signal: GET /health answers in every connection mode —2+// open (Jazz serves it directly), ticket (the gate proxies it, both at the3+// top level and under a ticket base path), and peer (carried over the iroh4+// tunnel to the node's Jazz). A failing store degrades to a non-200, never a5+// hang, so clients can fold periodic health checks into a connection6+// observable alongside WS lifecycle events.7+8+import { assert, assertEquals } from "@std/assert";9+import { createSyncNode } from "../src/node.ts";10+import { startGate } from "../src/gate.ts";11+import { AppTicketStore, decodeAppTicket } from "../src/appticket.ts";12+import { loadIrohAddon } from "../src/native/addon.ts";13+import { IrohNode } from "../src/iroh/node.ts";14+import { startTunnelListener } from "../src/tunnel.ts";15+import { resolveIrohLib } from "../src/native/loader.ts";16+17+async function fetchHealth(base: string): Promise<{ status: number; body: unknown }> {18+ const res = await fetch(new URL("/health", base));19+ const text = await res.text();20+ let body: unknown = text;21+ try {22+ body = JSON.parse(text);23+ } catch {24+ // non-JSON error bodies stay as text25+ }26+ return { status: res.status, body };27+}28+29+Deno.test({30+ name: "health: open mode — Jazz answers on the public URL",31+ sanitizeOps: false,32+ sanitizeResources: false,33+ fn: async () => {34+ const node = await createSyncNode({35+ appId: crypto.randomUUID(),36+ backendSecret: "lofi_backend_health_open",37+ adminSecret: "lofi_admin_health_open",38+ inMemory: true,39+ mesh: "off",40+ });41+ try {42+ const { status, body } = await fetchHealth(node.url.replace(/^ws/, "http"));43+ assertEquals(status, 200);44+ assertEquals((body as { status: string }).status, "healthy");45+ } finally {46+ await node.stop();47+ }48+ },49+});50+51+Deno.test({52+ name: "health: ticket mode — unauthenticated at the top level, and under a ticket base",53+ sanitizeOps: false,54+ sanitizeResources: false,55+ fn: async () => {56+ const node = await createSyncNode({57+ appId: crypto.randomUUID(),58+ backendSecret: "lofi_backend_health_gate",59+ adminSecret: "lofi_admin_health_gate",60+ inMemory: true,61+ mesh: "off",62+ access: "ticket",63+ });64+ try {65+ // Top level: no ticket needed — reachable before enrollment.66+ const top = await fetchHealth(node.url.replace(/^ws/, "http"));67+ assertEquals(top.status, 200);68+ assertEquals((top.body as { status: string }).status, "healthy");69+70+ // Under a ticket base: serverUrl-relative polling works unchanged.71+ const issued = await node.issueTicket({ label: "health-probe" });72+ const ticket = decodeAppTicket(issued.ticket)!;73+ const relative = await fetch(`${ticket.url}/health`);74+ assertEquals(relative.status, 200);75+ assertEquals((await relative.json() as { status: string }).status, "healthy");76+77+ // Revocation closes the ticket path but never the top-level signal:78+ // a revoked client still distinguishes "node up, access gone" (401)79+ // from "node unreachable" (network error).80+ await node.revokeTicket(issued.id);81+ const revoked = await fetch(`${ticket.url}/health`);82+ assertEquals(revoked.status, 401);83+ await revoked.body?.cancel();84+ const stillUp = await fetchHealth(node.url.replace(/^ws/, "http"));85+ assertEquals(stillUp.status, 200);86+ } finally {87+ await node.stop();88+ }89+ },90+});91+92+Deno.test({93+ name: "health: gate degrades to 502 when the store is unreachable",94+ sanitizeOps: false,95+ sanitizeResources: false,96+ fn: async () => {97+ // A dead loopback port stands in for a crashed Jazz process.98+ const dead = Deno.listen({ hostname: "127.0.0.1", port: 0 });99+ const deadPort = (dead.addr as Deno.NetAddr).port;100+ dead.close();101+ const store = await AppTicketStore.load();102+ const gate = startGate({103+ port: 0,104+ hostname: "127.0.0.1",105+ target: () => `http://127.0.0.1:${deadPort}`,106+ mode: "ticket",107+ store,108+ appId: "health-app",109+ adminSecret: "health-admin",110+ });111+ try {112+ const res = await fetch(`${gate.url}/health`);113+ assertEquals(res.status, 502, "gate up, store down → 502, not a hang");114+ await res.body?.cancel();115+ } finally {116+ await gate.close();117+ }118+ },119+});120+121+Deno.test({122+ name: "health: peer mode — carried over the iroh tunnel to the node's Jazz",123+ ignore: resolveIrohLib().status !== "ok",124+ sanitizeOps: false,125+ sanitizeResources: false,126+ fn: async () => {127+ const node = await createSyncNode({128+ appId: crypto.randomUUID(),129+ backendSecret: "lofi_backend_health_peer",130+ adminSecret: "lofi_admin_health_peer",131+ inMemory: true,132+ });133+ const addon = await loadIrohAddon();134+ const dialer = await IrohNode.open(addon, crypto.getRandomValues(new Uint8Array(32)));135+ const listener = startTunnelListener(dialer, node.ticket());136+ try {137+ const { status, body } = await fetchHealth(`http://127.0.0.1:${listener.port}`);138+ assertEquals(status, 200);139+ assertEquals((body as { status: string }).status, "healthy");140+ assert(listener.stats().length >= 0, "tunnel stats observable");141+ } finally {142+ await listener.close();143+ await dialer.close();144+ await node.stop();145+ }146+ },147+});@@ -0,0 +1,28 @@
1+// The taxonomy contract: consumers act on the code's classification, never on2+// the reason string, and unknown codes fail safe (keep waiting, never3+// compensate).4+5+import { assert, assertEquals } from "@std/assert";6+import {7+ classifyMutationError,8+ isPermanentMutationError,9+ MUTATION_ERROR_CLASSES,10+} from "../src/verdict.ts";11+12+Deno.test("verdict: permission_denied is permanent", () => {13+ assertEquals(classifyMutationError("permission_denied"), "permanent");14+ assert(isPermanentMutationError("permission_denied"));15+});16+17+Deno.test("verdict: unknown codes classify transient", () => {18+ for (const code of ["", "store_unreachable", "some_future_code", "PERMISSION_DENIED"]) {19+ assertEquals(classifyMutationError(code), "transient", `code ${JSON.stringify(code)}`);20+ assertEquals(isPermanentMutationError(code), false);21+ }22+});23+24+Deno.test("verdict: the registry only holds valid classes", () => {25+ for (const [code, klass] of Object.entries(MUTATION_ERROR_CLASSES)) {26+ assert(klass === "permanent" || klass === "transient", `code ${code}`);27+ }28+});@@ -0,0 +1,249 @@
1+// Write-verdict harness: the three server-side properties a client-side write2+// journal needs to record verdicts without a server ledger.3+//4+// The server only adjudicates writes made under a STALE policy: the client5+// runtime enforces the locally-known policy synchronously at the call site,6+// so a server rejection requires the store's permissions to tighten after the7+// client last synced them. The fixture models exactly that — permission8+// revoked at the store while the writing device is offline.9+//10+// Properties exercised:11+// 1. Correlation — a rejected write surfaces `permission_denied` with the12+// originating write's batch id, on both verdict surfaces (`wait()` →13+// `PersistedWriteRejectedError`, and the `onMutationError` fallback event14+// for writes nobody awaited).15+// 2. Re-derivability — a write unacked at disconnect is re-sent on reconnect16+// and the rejection fires then; nothing is lost to the disconnect.17+// 3. Determinism — the verdict for a batch is stable across further18+// reconnects, and an independent runtime issuing the same write against19+// the same store state derives the same code. The fallback event itself20+// is delivered once per runtime lifetime (not re-fired on later21+// reconnects), so a journal must persist the verdict when it arrives;22+// re-derivation on demand goes through `wait()`.23+//24+// Schemas are defined INSIDE the test fn: jazz registers schemas process-25+// globally at definition time, and module-level definitions would clobber26+// other test files' declared hashes.27+28+import { assert, assertEquals } from "@std/assert";29+import { createDb, schema as s } from "jazz-tools";30+import { deploy } from "jazz-tools/testing";31+import { definePermissions } from "jazz-tools/permissions";32+import { createSyncNode } from "../src/node.ts";33+import { classifyMutationError } from "../src/verdict.ts";34+35+interface MutationErrorEvent {36+ code: string;37+ reason: string;38+ batch: {39+ batchId: string;40+ mode: string;41+ sealed: boolean;42+ latestSettlement: { kind: string; batchId: string; code?: string; reason?: string } | null;43+ };44+}45+46+interface Rejection {47+ outcome: "rejected";48+ name?: string;49+ batchId?: string;50+ code?: string;51+ reason?: string;52+}53+54+type Verdict = Rejection | { outcome: "settled" };55+56+function accountSecret(fill: number): string {57+ let binary = "";58+ for (const byte of new Uint8Array(32).fill(fill)) binary += String.fromCharCode(byte);59+ return btoa(binary).replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/, "");60+}61+62+async function within<T>(operation: Promise<T>, label: string, milliseconds = 20_000): Promise<T> {63+ let timer: ReturnType<typeof setTimeout> | undefined;64+ try {65+ return await Promise.race([66+ operation,67+ new Promise<never>((_resolve, reject) => {68+ timer = setTimeout(() => reject(new Error(`${label} timed out`)), milliseconds);69+ }),70+ ]);71+ } finally {72+ if (timer !== undefined) clearTimeout(timer);73+ }74+}75+76+/** Settle a write handle into a plain verdict object. */77+function settle(handle: { wait(options: { tier: "global" }): Promise<unknown> }): Promise<Verdict> {78+ return handle.wait({ tier: "global" }).then(79+ (): Verdict => ({ outcome: "settled" }),80+ (e: Error & Partial<Rejection>): Verdict => ({81+ outcome: "rejected",82+ name: e.name,83+ batchId: e.batchId,84+ code: e.code,85+ reason: e.reason,86+ }),87+ );88+}89+90+async function pollUntil(91+ label: string,92+ predicate: () => boolean,93+ milliseconds = 15_000,94+): Promise<void> {95+ const deadline = Date.now() + milliseconds;96+ while (Date.now() < deadline) {97+ if (predicate()) return;98+ await new Promise((r) => setTimeout(r, 250));99+ }100+ throw new Error(`${label} did not become true within ${milliseconds}ms`);101+}102+103+Deno.test({104+ name: "write verdict: rejection correlates by batch id, survives reconnect, and is deterministic",105+ sanitizeOps: false,106+ sanitizeResources: false,107+ fn: async () => {108+ const tables = { verdict__entries: s.table({ text: s.string() }) };109+ const app = s.defineApp(tables);110+ const permissive = definePermissions(app, ({ policy, session }) => [111+ policy.verdict__entries.allowRead.where({ $createdBy: session.userId }),112+ policy.verdict__entries.allowInsert.where({ $createdBy: session.userId }),113+ ]);114+ const readOnly = definePermissions(app, ({ policy, session }) => [115+ policy.verdict__entries.allowRead.where({ $createdBy: session.userId }),116+ ]);117+118+ const node = await createSyncNode({119+ appId: crypto.randomUUID(),120+ backendSecret: "lofi_backend_verdict",121+ adminSecret: "lofi_admin_verdict",122+ inMemory: true,123+ mesh: "off",124+ });125+ const target = {126+ appId: node.appId,127+ serverUrl: node.url.replace(/^ws/, "http"),128+ adminSecret: "lofi_admin_verdict",129+ };130+ try {131+ await deploy({ ...target, schema: app, permissions: permissive });132+133+ // Two devices of one account, both booted while inserts are allowed, so134+ // both hold the permissive policy locally.135+ const secret = accountSecret(21);136+ const connect = () =>137+ createDb({138+ appId: node.appId,139+ serverUrl: target.serverUrl,140+ secret,141+ userBranch: "main",142+ driver: { type: "memory" },143+ });144+ const deviceA = await connect();145+ const deviceB = await connect();146+ try {147+ await within(148+ deviceA.insert(app.verdict__entries, { text: "seed" }).wait({ tier: "global" }),149+ "seed write under the permissive policy",150+ );151+152+ // Both devices go offline; the store's policy tightens underneath.153+ await deviceA.disconnect();154+ await deviceB.disconnect();155+ await deploy({ ...target, schema: app, permissions: readOnly });156+157+ // Offline writes under the stale (permissive) policy: the local158+ // runtime accepts them and hands back batch ids.159+ const awaited = deviceA.insert(app.verdict__entries, { text: "awaited stale write" });160+ const unawaited = deviceA.insert(app.verdict__entries, { text: "un-awaited stale write" });161+ const fromB = deviceB.insert(app.verdict__entries, { text: "device B stale write" });162+ assert(awaited.batchId !== unawaited.batchId, "distinct writes, distinct batch ids");163+164+ const events: MutationErrorEvent[] = [];165+ deviceA.onMutationError((event) => events.push(event as MutationErrorEvent));166+167+ // Property 2 — the unacked writes are re-sent on reconnect and the168+ // rejection fires then.169+ await deviceA.reconnect();170+171+ // Property 1a — the awaited surface: a typed rejection carrying the172+ // originating batch id and a machine-readable code.173+ const verdict = await within(settle(awaited), "awaited stale write settles");174+ assertEquals(verdict.outcome, "rejected");175+ const rejection = verdict as Rejection;176+ assertEquals(rejection.name, "PersistedWriteRejectedError");177+ assertEquals(rejection.batchId, awaited.batchId, "verdict correlates to the write");178+ assertEquals(rejection.code, "permission_denied");179+ assertEquals(classifyMutationError(rejection.code!), "permanent");180+ assert(rejection.reason && rejection.reason.length > 0, "reason present (diagnostics)");181+182+ // Property 1b — the fallback event surface for writes nobody awaited:183+ // the event carries the batch record, correlating by batch id.184+ await pollUntil(185+ "onMutationError fires for the un-awaited write",186+ () => events.some((e) => e.batch.batchId === unawaited.batchId),187+ );188+ const event = events.find((e) => e.batch.batchId === unawaited.batchId)!;189+ assertEquals(event.code, "permission_denied");190+ assertEquals(event.batch.sealed, true);191+ assertEquals(event.batch.latestSettlement?.kind, "rejected");192+ assertEquals(event.batch.latestSettlement?.code, "permission_denied");193+ const eventsAfterFirstCycle = events.length;194+195+ // Property 3a — the verdict is stable across further reconnects, and196+ // re-waiting re-derives it without a new fallback event.197+ await deviceA.disconnect();198+ await deviceA.reconnect();199+ const again = await within(settle(awaited), "re-derived verdict after reconnect");200+ assertEquals(again.outcome, "rejected");201+ assertEquals((again as Rejection).code, "permission_denied");202+ assertEquals((again as Rejection).batchId, awaited.batchId);203+ const unawaitedAgain = await within(settle(unawaited), "un-awaited verdict via wait()");204+ assertEquals((unawaitedAgain as Rejection).code, "permission_denied");205+ // Delivered once per runtime lifetime: the reconnect did not re-fire206+ // events for already-settled batches. A journal must persist the207+ // verdict when it first arrives.208+ await new Promise((r) => setTimeout(r, 1500));209+ assertEquals(events.length, eventsAfterFirstCycle, "no event re-fire on reconnect");210+211+ // Property 3b — an independent runtime, same account, same store212+ // state, same write shape: the same code.213+ await deviceB.reconnect();214+ const verdictB = await within(settle(fromB), "device B stale write settles");215+ assertEquals(verdictB.outcome, "rejected");216+ assertEquals((verdictB as Rejection).code, "permission_denied");217+ assertEquals((verdictB as Rejection).batchId, fromB.batchId);218+219+ // Once the tightened policy has synced to a device, writes it denies220+ // never reach the server again: the local runtime refuses them221+ // synchronously at the call site, before any batch exists. These222+ // surface as thrown errors to the author, not as write verdicts.223+ // (Until that sync lands — fresh boot, first reconnect — the same224+ // write is accepted locally and adjudicated by the server, as above.)225+ let localRefusal: Error | null = null;226+ await pollUntil("local enforcement of the synced read-only policy", () => {227+ try {228+ deviceA.insert(app.verdict__entries, { text: "refused?" });229+ return false;230+ } catch (e) {231+ localRefusal = e as Error;232+ return true;233+ }234+ });235+ assert(236+ (localRefusal as unknown as Error).message.includes("policy denied INSERT"),237+ `local refusal names the policy: ${(localRefusal as unknown as Error).message}`,238+ );239+ } finally {240+ await Promise.allSettled([241+ within(deviceA.logout(), "device A cleanup", 3000),242+ within(deviceB.logout(), "device B cleanup", 3000),243+ ]);244+ }245+ } finally {246+ await node.stop();247+ }248+ },249+});