Add package/schema/merge_sync_test.ts: counter and g-set merge semantics across two synced clients plus a fresh observer, built on the JazzServer setup from the access sync test. It runs in its own process before the conformance suite (whose scalar test wedges the FFI driver last). G-set union merge is verified convergent across writers and fresh boots. Counter merge is pinned as inconsistent across replica lifetimes in alpha.53: the server keeps the last causally ordered update value and sums only concurrent updates, while live replicas apply every update -- including the echo of their own reconnected write -- as a delta, so a live replica diverges permanently from what a fresh boot reads. Document the full column palette in docs/data-and-ui.md, including the i32 limit, the merge-cast workaround, and merge-strategy guidance, and record the counter finding in the schema facade decision record. Findings stay internal until our testing is complete; nothing is reported upstream. Continues #105.
- commit
- 956cd12f42bccc6d5b0cfffb292fa2c10477277e
- parent
- 85720bcc6276b354e35ca0dcadd0e5d2e0ba948d
- author
- Dami <dami.etoile@gmail.com>
| M | deno.json | 2 | +- |
| M | docs/ | 56 | ++++++ |
| M | docs/ | 22 | ++- |
| M | package/ | 5 | +- |
| A | package/ | 268 | ++++++++++++++++++++++++++++++ |
@@ -56,7 +56,7 @@
5656 "docs:links": "deno run --allow-read tools/docs_links.ts",5757 "test:unit": "deno test --allow-env=LOFI_UPDATE_SNAPSHOT,CI --allow-net=127.0.0.1 --allow-read --allow-write=. --allow-run tools package/astro package/create_core_test.ts package/preact package/project_layout_doc_test.ts package/pwa_assets_test.ts package/recipes package/runtime package/schema/mod_test.ts package/testing package/tooling apps/reference/tests",5858 "test:access": "deno test --allow-env --allow-net=127.0.0.1 --allow-read --allow-write --allow-run --allow-ffi package/access/*_test.ts",59- "test:conformance": "deno test --allow-env --allow-net=127.0.0.1 --allow-read --allow-write --allow-run --allow-ffi package/schema/conformance_test.ts",59+ "test:conformance": "deno test --allow-env --allow-net=127.0.0.1 --allow-read --allow-write --allow-run --allow-ffi package/schema/merge_sync_test.ts && deno test --allow-env --allow-net=127.0.0.1 --allow-read --allow-write --allow-run --allow-ffi package/schema/conformance_test.ts",6060 "test:update:create": "LOFI_UPDATE_SNAPSHOT=1 deno test --allow-env=LOFI_UPDATE_SNAPSHOT --allow-read --allow-write=. --allow-run package/create_core_test.ts",6161 "test": "deno task test:unit",6262 "test:browser:install": "deno run -A npm:playwright@1.61.1 install chromium",@@ -47,6 +47,62 @@ query and which mutations succeed; realtime access is not a separate permission
4747 [Permissions](permissions.md), [direct sharing](examples/shared.md), and4848 [group ownership](examples/group.md).4949 50+## Column palette51+52+Every column constructor below is exercised against the real pinned engine by the conformance suite53+(`deno task test:conformance`); a member that only compiles is not part of the supported surface.54+55+| Constructor | Row value | Notes |56+| ------------------ | ------------------------ | ---------------------------------------------- |57+| `s.string()` | `string` | Unicode round-trips. |58+| `s.boolean()` | `boolean` | |59+| `s.int()` | `number` | i32 range only at runtime; see below. |60+| `s.float()` | `number` | Double precision. |61+| `s.timestamp()` | `Date` | Storage round-trips; do not filter on it yet. |62+| `s.enum("a", "b")` | Union of the literals | |63+| `s.bytes()` | `Uint8Array` | Keep payloads at 32 bytes or more; see below. |64+| `s.json()` | Any JSON value | Nested objects and arrays round-trip. |65+| `s.array(inner)` | Array of the inner value | |66+| `s.ref("table")` | Referenced row id | Filterable foreign key: `where({ parentId })`. |67+68+Constraints of the pinned alpha, each pinned by a conformance test that fails when upstream lifts69+it:70+71+- **`s.int()` accepts only the i32 range at runtime.** Despite the `number` static type, values72+ outside ±2³¹ are rejected with `InvalidArg … expected i32`.73+- **Timestamp `where`-equality matches every row** instead of filtering, so query timestamps by id74+ or another column for now.75+- **Byte payloads under 32 bytes are unreliable**; pad short payloads or store them as `s.json()`.76+77+### Column modifiers and merge strategies78+79+Modifiers chain on any constructor: `.optional()` makes the column nullable, `.default(value)` fills80+omitted inserts, and `.merge(strategy)` picks a conflict strategy for concurrent writers. In the81+pinned alpha `.merge()` returns the untyped builder (the legacy `merge(): this` signature shadows82+the typed overload), which degrades the whole table's row types — cast the result back to the83+intended column type; the runtime object is unchanged:84+85+```ts86+import { type ArrayColumn, type IntColumn, s } from "@nzip/lofi/schema";87+88+export const app = s.defineApp({89+ tagged: s.table({90+ name: s.string(),91+ tags: s.array(s.string()).merge("g-set") as unknown as ArrayColumn<"TEXT">,92+ }),93+});94+```95+96+Merge behavior is verified with two synced clients in `package/schema/merge_sync_test.ts`:97+98+- **`"g-set"` works**: concurrent writers union their elements and every replica — including a fresh99+ boot — converges on the same set. Keep a g-set table in its own single-table app for now; in the100+ pinned alpha a g-set column destabilizes writes to sibling tables in the same app.101+- **`"counter"` is not dependable yet**: the server keeps the last causally ordered update value and102+ sums only concurrent updates, while live replicas add every update as a delta, so a replica that103+ watched the history diverges permanently from what a fresh boot reads. Avoid counter columns until104+ an alpha bump clears the pins (see [the decision record](decisions/schema-facade-alpha53.md)).105+50106 ## Bind an exact typed query51107 52108 ```ts@@ -48,6 +48,8 @@ test.
4848 4949 - `package/schema/mod.ts` — the facade; `package/schema/mod_test.ts` verifies member identity5050 against `jazz-tools` and round-trips `defineApp`/`definePermissions` through the facade.51+- `package/schema/merge_sync_test.ts` — concurrent-writer merge semantics (counter, g-set) over a52+ real JazzServer with two synced clients and a fresh observer.5153 - `apps/reference/tests/author-boundary_test.ts` — rejects `jazz-tools` imports in all author5254 source, including `schema.ts` and `permissions.ts`.5355 - `deno task check` and `deno task build` pass with the reference app (and therefore the starter,@@ -88,16 +90,24 @@ pinned in the suite so an alpha bump surfaces any change:
8890 5. **Short byte payloads are unreliable.** A 2-byte insert has been observed succeeding, failing8991 with `WriteError … data too short for column value`, and hanging, depending on process context.9092 Covered by a tolerant subprocess canary.93+6. **Counter merge semantics are inconsistent across replica lifetimes.** Verified with two synced94+ clients plus a fresh observer (`package/schema/merge_sync_test.ts`): the server keeps the last95+ causally ordered update value and sums only concurrent updates, while live replicas apply every96+ update — including the echo of their own reconnected write — as a delta. A replica that watched97+ the history diverges permanently from what a fresh boot reads (observed: canonical 8, first98+ writer 13 after a concurrent 5/3 conflict on 0; canonical 3, live replicas 5 after causal 2 then99+ 3). The full matrix is pinned. G-set union merge, by contrast, is verified convergent across100+ writers and fresh boots.91101 92102 Hangs cannot be cancelled through FFI, so the canaries run in child processes that force-exit93-(`package/testdata/gset_hang_canary.ts`, `package/testdata/bytes_short_payload_canary.ts`).103+(`package/testdata/gset_hang_canary.ts`, `package/testdata/bytes_short_payload_canary.ts`). The104+concurrent-writer suite runs before the conformance suite in a separate process because the scalar105+test wedges the FFI driver last.94106 95107 ## Follow-up96108 97-- Report findings 2–5 upstream to Jazz; re-verify all pins on each alpha bump and remove the ones98- upstream fixes.99-- Document the full column palette in `docs/data-and-ui.md`, including the i32 limit and the100- merge-cast workaround, once the surface stabilizes enough to teach.101-- Concurrent-writer merge semantics (counter, g-set) need two synced clients and remain untested.109+- Re-verify all pins on each alpha bump and remove the ones upstream fixes; findings 2–6 stay110+ internal until our own testing is complete and a report is explicitly decided.111+- Merge the g-set table back into the main conformance app when the cross-table hang is gone.102112 - Revisit the curation list on each alpha bump; `defineSliceableApp` joins the surface when a use103113 case and tests exist.@@ -263,9 +263,8 @@ Deno.test("facade column modifiers (optional, default, counter, g-set) behave th
263263 `counter default: read back ${describeValue(readBack.total)}`,264264 );265265 266- // Single-session semantics only; concurrent-writer merge behavior needs267- // two synced clients and stays a follow-up (see the schema facade268- // decision record).266+ // Single-session semantics only; concurrent-writer merge behavior is267+ // pinned in ./merge_sync_test.ts, which runs first in its own process.269268 await db.update(app.modifiers, row.id, { total: 5 }).wait({ tier: "global" });270269 const bumped = (await db.all(app.modifiers.where({ id: row.id })))[0];271270 assert(bumped !== undefined, "modifiers: row lost after update");@@ -0,0 +1,268 @@
1+// Concurrent-writer merge semantics for the facade's merge strategies:2+// counter and g-set columns written by two synced clients. Single-session3+// behavior is covered by the conformance suite; this file is the multi-client4+// half, built on the JazzServer setup from package/access/access_sync_test.ts.5+//6+// Runs in its own process before the conformance suite (see the7+// test:conformance task) because the conformance scalar test deliberately8+// wedges the FFI driver last.9+import { createDb } from "jazz-tools";10+import { deploy, startLocalJazzServer } from "jazz-tools/testing";11+import { assert } from "../runtime/test-assert.ts";12+import { type ArrayColumn, type IntColumn, s } from "./mod.ts";13+14+const counterApp = s.defineApp({15+ tallies: s.table({16+ name: s.string(),17+ total: s.int().default(0).merge("counter") as unknown as IntColumn<false, true>,18+ }),19+});20+const counterPermissions = s.definePermissions(counterApp, ({ policy }) => {21+ policy.tallies.allowInsert.always();22+ policy.tallies.allowRead.always();23+ policy.tallies.allowUpdate.always();24+ policy.tallies.allowDelete.always();25+});26+27+// Single-table app: a g-set column destabilizes sibling-table writes in the28+// pinned alpha (see the conformance canary), so the g-set coverage stays29+// isolated here too.30+const gsetApp = s.defineApp({31+ tagged: s.table({32+ name: s.string(),33+ tags: s.array(s.string()).merge("g-set") as unknown as ArrayColumn<"TEXT">,34+ }),35+});36+const gsetPermissions = s.definePermissions(gsetApp, ({ policy }) => {37+ policy.tagged.allowInsert.always();38+ policy.tagged.allowRead.always();39+ policy.tagged.allowUpdate.always();40+ policy.tagged.allowDelete.always();41+});42+43+function secret(fill: number): string {44+ let binary = "";45+ for (const byte of new Uint8Array(32).fill(fill)) binary += String.fromCharCode(byte);46+ return btoa(binary).replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/, "");47+}48+49+async function within<T>(operation: Promise<T>, label: string, milliseconds = 10_000): Promise<T> {50+ let timer: ReturnType<typeof setTimeout> | undefined;51+ try {52+ return await Promise.race([53+ operation,54+ new Promise<never>((_resolve, reject) => {55+ timer = setTimeout(() => reject(new Error(`${label} timed out`)), milliseconds);56+ }),57+ ]);58+ } finally {59+ if (timer !== undefined) clearTimeout(timer);60+ }61+}62+63+type DeployOptions = Parameters<typeof deploy>[0];64+type Harness = {65+ server: Awaited<ReturnType<typeof startLocalJazzServer>>;66+ client(fill: number): Promise<Awaited<ReturnType<typeof createDb>>>;67+ stop(): Promise<void>;68+};69+70+async function bootHarness(71+ app: DeployOptions["schema"],72+ permissions: DeployOptions["permissions"],73+): Promise<Harness> {74+ const server = await startLocalJazzServer({ allowLocalFirstAuth: true });75+ await deploy({76+ appId: server.appId,77+ serverUrl: server.url,78+ adminSecret: server.adminSecret,79+ schema: app,80+ permissions,81+ });82+ const clients: Awaited<ReturnType<typeof createDb>>[] = [];83+ return {84+ server,85+ async client(fill: number) {86+ const db = await createDb({87+ appId: server.appId,88+ serverUrl: server.url,89+ secret: secret(fill),90+ userBranch: "main",91+ driver: { type: "memory" },92+ });93+ clients.push(db);94+ return db;95+ },96+ async stop() {97+ await Promise.allSettled(98+ clients.map((db, index) => within(db.logout(), `client ${index} cleanup`, 3_000)),99+ );100+ await within(server.stop(), "server cleanup", 3_000).catch(() => undefined);101+ },102+ };103+}104+105+// PINNED (alpha.53): counter columns have no single semantics across replica106+// lifetimes. The server merges an update as a replacement when it is causally107+// latest and as a sum when updates are concurrent, while live replicas apply108+// every update — including the echo of their own reconnected write — as a109+// delta. A replica that watched the history read therefore diverges110+// permanently from what a fresh boot of the same account reads. Each assert111+// below pins one cell of that matrix; when one fails, upstream changed112+// counter merge behavior — re-derive the matrix and update the column-palette113+// section of docs/data-and-ui.md.114+Deno.test("counter merge: concurrent offline updates sum on the server, diverge on a live writer", async () => {115+ const harness = await bootHarness(counterApp, counterPermissions);116+ try {117+ const alice = await harness.client(1);118+ const bob = await harness.client(2);119+ const row = await within(120+ alice.insert(counterApp.tallies, { name: "hits", total: 0 }).wait({ tier: "global" }),121+ "seed insert",122+ );123+ const bobView = await within(124+ bob.all(counterApp.tallies.where({ id: row.id }), { tier: "global" }),125+ "bob initial read",126+ );127+ assert(bobView.length === 1 && bobView[0].total === 0, "bob did not sync the seeded row");128+129+ await alice.disconnect();130+ await bob.disconnect();131+ const aliceWrite = alice.update(counterApp.tallies, row.id, { total: 5 });132+ const bobWrite = bob.update(counterApp.tallies, row.id, { total: 3 });133+ await alice.reconnect();134+ await bob.reconnect();135+ await within(aliceWrite.wait({ tier: "global" }), "alice counter durability");136+ await within(bobWrite.wait({ tier: "global" }), "bob counter durability");137+138+ const read = async (db: typeof alice, label: string) => {139+ const rows = await within(140+ db.all(counterApp.tallies.where({ id: row.id }), { tier: "global" }),141+ `${label} read`,142+ );143+ return rows[0]?.total;144+ };145+ const aliceTotal = await read(alice, "alice");146+ const bobTotal = await read(bob, "bob");147+ const charlie = await harness.client(3);148+ const canonical = await read(charlie, "fresh client");149+150+ assert(151+ canonical === 8,152+ `counter: concurrent updates no longer sum on the server (fresh client read ${canonical}, ` +153+ `not 5 + 3); the merge model changed — remap it`,154+ );155+ assert(156+ bobTotal === 8,157+ `counter: second writer read ${bobTotal}, not the canonical sum 8`,158+ );159+ assert(160+ aliceTotal === 13,161+ `counter: pinned first-writer divergence changed — read ${aliceTotal}; if 8, upstream ` +162+ `stopped double-counting the reconnected writer's own echo: remove the pin`,163+ );164+ } finally {165+ await harness.stop();166+ }167+});168+169+Deno.test("counter merge: causally ordered updates replace on the server, accumulate on live replicas", async () => {170+ const harness = await bootHarness(counterApp, counterPermissions);171+ try {172+ const alice = await harness.client(1);173+ const bob = await harness.client(2);174+ const row = await within(175+ alice.insert(counterApp.tallies, { name: "causal", total: 0 }).wait({ tier: "global" }),176+ "seed insert",177+ );178+ await within(179+ alice.update(counterApp.tallies, row.id, { total: 2 }).wait({ tier: "global" }),180+ "alice update durability",181+ );182+ const bobBefore = await within(183+ bob.all(counterApp.tallies.where({ id: row.id }), { tier: "global" }),184+ "bob pre-write read",185+ );186+ assert(bobBefore[0]?.total === 2, "bob did not sync alice's update before writing");187+ await within(188+ bob.update(counterApp.tallies, row.id, { total: 3 }).wait({ tier: "global" }),189+ "bob update durability",190+ );191+192+ const bobTotal =193+ (await within(bob.all(counterApp.tallies.where({ id: row.id })), "bob read"))[0]?.total;194+ const aliceTotal = (await within(195+ alice.all(counterApp.tallies.where({ id: row.id }), { tier: "global" }),196+ "alice read",197+ ))[0]?.total;198+ const charlie = await harness.client(3);199+ const canonical = (await within(200+ charlie.all(counterApp.tallies.where({ id: row.id }), { tier: "global" }),201+ "fresh client read",202+ ))[0]?.total;203+204+ assert(205+ canonical === 3,206+ `counter: pinned causal replacement changed — fresh client read ${canonical}; if 5, ` +207+ `upstream now sums causally ordered updates like live replicas do: remove the pin`,208+ );209+ assert(210+ aliceTotal === 5 && bobTotal === 5,211+ `counter: live replicas no longer accumulate deltas (alice=${aliceTotal}, bob=${bobTotal}, ` +212+ `expected both 5 = 2 + 3)`,213+ );214+ } finally {215+ await harness.stop();216+ }217+});218+219+Deno.test("g-set columns union concurrent offline updates and converge everywhere", async () => {220+ const harness = await bootHarness(gsetApp, gsetPermissions);221+ try {222+ const alice = await harness.client(1);223+ const bob = await harness.client(2);224+ const row = await within(225+ alice.insert(gsetApp.tagged, { name: "doc", tags: ["a"] }).wait({ tier: "global" }),226+ "seed insert",227+ );228+ const bobView = await within(229+ bob.all(gsetApp.tagged.where({ id: row.id }), { tier: "global" }),230+ "bob initial read",231+ );232+ assert(bobView.length === 1, "bob did not sync the seeded row");233+234+ await alice.disconnect();235+ await bob.disconnect();236+ const aliceWrite = alice.update(gsetApp.tagged, row.id, { tags: ["a", "b"] });237+ const bobWrite = bob.update(gsetApp.tagged, row.id, { tags: ["a", "c"] });238+ await alice.reconnect();239+ await bob.reconnect();240+ await within(aliceWrite.wait({ tier: "global" }), "alice g-set durability");241+ await within(bobWrite.wait({ tier: "global" }), "bob g-set durability");242+243+ const read = async (db: typeof alice, label: string) => {244+ const rows = await within(245+ db.all(gsetApp.tagged.where({ id: row.id }), { tier: "global" }),246+ `${label} read`,247+ );248+ return [...(rows[0]?.tags ?? [])].sort();249+ };250+ const aliceTags = await read(alice, "alice");251+ const bobTags = await read(bob, "bob");252+ const charlie = await harness.client(3);253+ const canonical = await read(charlie, "fresh client");254+255+ const union = JSON.stringify(["a", "b", "c"]);256+ for (257+ const [label, tags] of [["alice", aliceTags], ["bob", bobTags], ["fresh client", canonical]]258+ .map(([l, t]) => [l, JSON.stringify(t)] as const)259+ ) {260+ assert(261+ tags === union,262+ `g-set: ${label} read ${tags}, not the union of both writers' elements`,263+ );264+ }265+ } finally {266+ await harness.stop();267+ }268+});