identity.db changes all day, so a daily snapshot isn't enough. The container builds litestream v0.5.17 (pinned, go install, like tailcat and rclone) and runs forge identity replicate, which copies every change to the backup bucket under litestream/identity, with the FORGE_BACKUP_* keys, which stay in the environment; a local backup repository gets a file replica next to it. forge identity restore brings it back as it is or as it was at a moment of the last week (--at), keeping the one it replaces. A box without identity.db takes it from the replica at boot, before forge server init would make an empty one whose replication would bury the replica's; if the replica can't be read, nothing replicates that boot. Without backups configured, nothing changes. The daily backup carries its own copy, made by VACUUM INTO as the file's owner, never the live file with its WAL or litestream's state, and the drill runs SQLite's integrity check on it. forge restore takes whichever of the backup's copy and the replica changed last, by meta.changed, which every write bumps (deletions too), so a revoked token doesn't come back from an older copy.
- commit
- c284d8ddfd1aef6b83b75d7a4b60ea73badee9e3
- parent
- 3cd778d7f75ff2debcc4020f3f983399b978505b
- author
- cat <admin@dev.site>
- kind
- feat risk score 1.9 of 3, Jev's estimate
| M | Dockerfile | 7 | + |
| M | bin/ | 172 | +++++++++++++++++++++++++++++- |
| M | lib/ | 18 | +++- |
| M | server/ | 23 | ++++- |
| A | test/ | 43 | ++++++++ |
| A | test/ | 65 | +++++++++++ |
@@ -11,6 +11,12 @@ RUN CGO_ENABLED=0 GOBIN=/out go install github.com/tailscale/tailcat/cmd/tailcat
1111 FROM golang:1.27.2 AS rclone1212 RUN CGO_ENABLED=0 GOBIN=/out go install github.com/rclone/rclone@v1.75.21313 14+# litestream copies identity.db to the backup bucket as it changes (forge15+# identity replicate). Pinned and checked the same way; it builds without16+# cgo.17+FROM golang:1.27.2 AS litestream18+RUN CGO_ENABLED=0 GOBIN=/out go install github.com/benbjohnson/litestream/cmd/litestream@v0.5.1719+1420 # Node 22 runs the TypeScript sources directly (type stripping).1521 FROM node:22-trixie-slim1622 @@ -23,6 +29,7 @@ RUN apt-get update \
2329 2430 COPY --from=tailcat /out/tailcat /usr/local/bin/tailcat2531 COPY --from=rclone /out/rclone /usr/local/bin/rclone32+COPY --from=litestream /out/litestream /usr/local/bin/litestream2633 COPY server/container/sshd.conf /etc/ssh/sshd_config.d/forge.conf2734 2835 WORKDIR /opt/forge@@ -34,6 +34,7 @@
3434 # forge backups [--status] the last backup's health, then the snapshots3535 # forge restore [<snapshot>|latest] [--to <dir>] [--from <dir>] bring the forge back from a backup3636 # forge mirror [--dry-run] sync the pages to the mirror bucket (builds do this by themselves)37+# forge identity replicate | restore [--at <time>] [--if-missing] identity.db's litestream replica3738 # forge shell <user> forced command in authorized_keys3839 # forge hook pre-receive|post-receive3940 set -eu@@ -1779,9 +1780,9 @@ sc_release() {
17791780 esac17801781 }17811782 1782-# identity <command> [args]: the caller's web credentials, which live in1783+# credentials <command> [args]: the caller's web credentials, which live in17831784 # .forge/identity.db (lib/identity.ts): passkeys and their sessions, tokens.1784-identity() { FORGE_USER=$user node "$FORGE_HOME/lib/identity.ts" "$@"; }1785+credentials() { FORGE_USER=$user node "$FORGE_HOME/lib/identity.ts" "$@"; }17851786 17861787 # Commands reachable over ssh through git-shell-commands/.17871788 cmd_shell_cmd() {@@ -1844,7 +1845,7 @@ cmd_shell_cmd() {
18441845 public) sc_public "$@" ;;18451846 readers) sc_readers "$@" ;;18461847 release) sc_release "$@" ;;1847- passkey | passkeys | sessions | tokens) identity "$c" "$@" ;;1848+ passkey | passkeys | sessions | tokens) credentials "$c" "$@" ;;18481849 token)18491850 # A token for one repo needs a repo you can read, or it isn't there.18501851 r= next=0@@ -1854,7 +1855,7 @@ cmd_shell_cmd() {
18541855 [ "$a" != --repo ] || next=118551856 done18561857 [ -z "$r" ] || open_repo "$r"1857- identity token "$@"1858+ credentials token "$@"18581859 ;;18591860 *) printf 'commands:\n new [--private] <name> [description] create %s/<name>\n ls list your repos, and private ones you may read\n signers keys that may sign CI results (for forge verify)\n site-url <owner>/<repo> <site> [branch] where a site deploy is served\n reviews <owner>/<repo> open and draft reviews (forge reviews)\n land <owner>/<repo> <id|branch> land a review here, with the same gates as a push\n comment, approve, approvals used by forge comment, approve and land\n private <owner>/<repo> only its owner, collaborators and readers can see it\n public <owner>/<repo> [--force] anyone can, once its whole history passes the secret check\n readers <owner>/<repo> [add|rm <user>] who may read it while it is private\n release ls <owner>/<repo> its releases: tag, file, size, sha256\n release get <owner>/<repo> <tag> <file> a release file, on stdout\n passkey [name] a one-time link that adds a passkey, to sign in on the web\n passkeys [rm <id>] your passkeys\n sessions [rm <id>|--all] where you are signed in on the web; rm signs one out, --all all\n token [--repo <owner>/<repo>] [--ttl 30d] [name] a token for curl and HTTPS clones, shown once\n tokens [rm <id>] your tokens\na push to %s/<name>.git creates that repo when it is missing\n' "$user" "$user" ;;18601861 esac@@ -2731,11 +2732,14 @@ stage_repo() {
27312732 }27322733 27332734 # .forge entries a backup leaves out: rebuilt (www, store, cache,2734-# allowed_signers), regenerated at boot (secrets), or logs, locks and scratch.2735-# The lake is staged by itself (stage_lake).2735+# allowed_signers, private pages), regenerated at boot or by commands2736+# (secrets, litestream.yml), or logs, locks and scratch. The lake is staged2737+# by itself (stage_lake), and identity.db by stage_identity; its WAL and2738+# litestream's state are never copied.27362739 derived_state() {27372740 case $1 in2738- www | store | cache | backup | secrets | allowed_signers | config.restored | restore.tmp | lake | \2741+ www | store | cache | backup | secrets | allowed_signers | config.restored | restore.tmp | lake | private | \2742+ identity.db | identity.db-* | identity.db.* | .identity.db-litestream | litestream.yml | \27392743 *.log | *.lock | *.tmp | backup.* | build.* | events.* | metrics.* | mirror.* | lake.*) return 0 ;;27402744 esac27412745 return 1@@ -2758,6 +2762,23 @@ stage_lake() {
27582762 return 127592763 }27602764 2765+# stage_identity <file>: the backup's copy of identity.db, made by SQLite2766+# with VACUUM INTO: one consistent file, while the servers keep writing,2767+# never the live file and its WAL. As the file's owner: SQLite files that2768+# root opened would be root's, and the servers couldn't write them.2769+stage_identity() {2770+ db=$FORGE_STATE/identity.db2771+ [ -f "$db" ] || return 02772+ tmp=$FORGE_STATE/identity.snapshot.tmp2773+ rm -f "$tmp"2774+ if [ "$(id -u)" = 0 ] && [ "$(stat -c %u "$db" 2>/dev/null || echo 0)" != 0 ]; then2775+ runuser -u "$(stat -c %U "$db")" -- env FORGE_STATE="$FORGE_STATE" node "$FORGE_HOME/lib/identity.ts" snapshot "$tmp"2776+ else2777+ node "$FORGE_HOME/lib/identity.ts" snapshot "$tmp"2778+ fi || { berr="staging identity.db: VACUUM INTO failed"; return 1; }2779+ mv "$tmp" "$1"2780+}2781+27612782 # stage_all <dir>: copy everything a restore needs into dir. That is all of27622783 # FORGE_ROOT except caches, git-shell-commands (forge server init writes it)27632784 # and the derived parts of .forge, so state added later is kept by default.@@ -2781,6 +2802,7 @@ stage_all() {
27812802 cp -RPp "$p/www/hosts" "$st/.forge/www/" || return 127822803 fi27832804 stage_lake "$st/.forge" || return 12805+ stage_identity "$st/.forge/identity.db" || return 127842806 ;;27852807 .cache | .npm | git-shell-commands | lost+found) ;;27862808 .*) cp -RPp "$p" "$st/" || return 1 ;;@@ -2824,6 +2846,11 @@ drill() {
28242846 n=$(git --git-dir="$dr/$slug" for-each-ref </dev/null | wc -l | tr -d ' ')28252847 [ "$n" = "$refs" ] || { berr="drill: the restored $slug has $n refs, the live one had $refs"; return 1; }28262848 done <"$dr/.forge/backup.manifest"2849+ if [ -f "$dr/.forge/identity.db" ] && ! node "$FORGE_HOME/lib/identity.ts" check "$dr/.forge/identity.db" >"$BK_WORK/out" 2>&1; then2850+ cat "$BK_WORK/out" >>"$BK_LOG"2851+ berr="drill: the restored identity.db fails SQLite's integrity check"2852+ return 12853+ fi28272854 rm -rf "$dr"28282855 drilled="ok, $nrepos repos restored, git fsck clean, refs match"28292856 }@@ -3051,6 +3078,9 @@ cmd_restore() {
30513078 trap 'rm -rf "$FORGE_STATE/backup.lock"' EXIT30523079 keep=$(mktemp -d "${TMPDIR:-/tmp}/forge-restore.XXXXXX")30533080 for f in config keys; do if [ -f "$FORGE_STATE/$f" ]; then cp -p "$FORGE_STATE/$f" "$keep/$f"; fi; done3081+ # This host's identity.db (with its WAL) steps aside for the backup's.3082+ for f in identity.db identity.db-wal identity.db-shm; do if [ -f "$FORGE_STATE/$f" ]; then mv "$FORGE_STATE/$f" "$keep/$f"; fi; done3083+ rm -rf "$FORGE_STATE/.identity.db-litestream"30543084 if [ -n "$from" ]; then30553085 [ -f "$from/.forge/keys" ] || die "restore: $from doesn't look like a forge backup (no .forge/keys)"30563086 say "restoring from $from into $FORGE_ROOT"@@ -3060,6 +3090,10 @@ cmd_restore() {
30603090 say "restoring ${snap:-latest} into $FORGE_ROOT"30613091 restic restore "${snap:-latest}" --target "$FORGE_ROOT" || die "restore: restic restore failed"30623092 fi3093+ identity_pick3094+ if [ ! -f "$FORGE_STATE/identity.db" ] && [ -f "$keep/identity.db" ]; then3095+ for f in identity.db identity.db-wal identity.db-shm; do if [ -f "$keep/$f" ]; then mv "$keep/$f" "$FORGE_STATE/$f"; fi; done3096+ fi30633097 # This host's config wins: start.sh and provision.sh write it for the30643098 # domain the host serves. The backup's copy is kept next to it.30653099 if [ -f "$keep/config" ]; then@@ -3081,6 +3115,129 @@ cmd_restore() {
30813115 fi30823116 }30833117 3118+# ---------------------------------------------------------------- identity.db replica3119+#3120+# identity.db changes all day (sign-ins, tokens), so a daily snapshot isn't3121+# enough: litestream copies every change to the backup bucket within3122+# seconds, and can bring it back as of any moment of the last week. The3123+# daily backup still carries a copy of its own (stage_identity), and a3124+# restore takes whichever of the two changed last (identity_pick).3125+3126+# lt_setup: litestream's config for identity.db, from the backup settings,3127+# in $FORGE_STATE/litestream.yml (LT_CONF). An s3: FORGE_BACKUP_REPOSITORY3128+# replicates to the same bucket, under litestream/identity3129+# (FORGE_LITESTREAM_PATH), with the same keys, which stay in the3130+# environment; a local path, to <path>-litestream. Fails, saying why, when3131+# backups are off or litestream isn't installed.3132+lt_setup() {3133+ r=${FORGE_BACKUP_REPOSITORY:-}3134+ [ -n "$r" ] || { say "litestream: backups are off (FORGE_BACKUP_REPOSITORY)"; return 1; }3135+ command -v litestream >/dev/null 2>&1 || { say "litestream is not installed"; return 1; }3136+ LT_CONF=$FORGE_STATE/litestream.yml3137+ case $r in3138+ s3:http://* | s3:https://*)3139+ u=${r#s3:}3140+ host=${u#*://}3141+ bucket=${host#*/}3142+ host=${host%%/*}3143+ bucket=${bucket%%/*}3144+ [ -n "$bucket" ] && [ "$bucket" != "$host" ] || { say "litestream: no bucket in $r"; return 1; }3145+ export AWS_ACCESS_KEY_ID="${FORGE_BACKUP_KEY_ID:-}" AWS_SECRET_ACCESS_KEY="${FORGE_BACKUP_SECRET:-}"3146+ replica=$(printf ' type: s3\n endpoint: %s\n bucket: %s\n path: %s\n region: %s' \3147+ "${u%%://*}://$host" "$bucket" "${FORGE_LITESTREAM_PATH:-litestream/identity}" "${FORGE_BACKUP_REGION:-auto}")3148+ ;;3149+ /*) replica=$(printf ' type: file\n path: %s' "${FORGE_LITESTREAM_PATH:-$r-litestream}") ;;3150+ *) say "litestream: FORGE_BACKUP_REPOSITORY is neither s3: nor a local path"; return 1 ;;3151+ esac3152+ (umask 077 && printf 'dbs:\n - path: %s\n replica:\n%s\nsnapshot:\n interval: 24h\n retention: 168h\n' "$FORGE_STATE/identity.db" "$replica" >"$LT_CONF")3153+}3154+3155+# identity_pick: after forge restore, identity.db is the one that changed3156+# last (meta.changed, which every write bumps, deletions too) of the3157+# backup's copy and litestream's replica, if there is one. Litestream's is3158+# normally seconds old and the backup's up to a day; the backup's wins only3159+# if replication had stopped.3160+identity_pick() {3161+ db=$FORGE_STATE/identity.db3162+ lt_setup >/dev/null 2>&1 || return 03163+ tmp=$FORGE_STATE/identity.litestream.tmp3164+ rm -f "$tmp" "$tmp-wal" "$tmp-shm"3165+ if ! litestream restore -no-expand-env -config "$LT_CONF" -if-replica-exists -o "$tmp" "$db" >>"$BK_LOG" 2>&1; then3166+ rm -f "$tmp"3167+ say "restore: litestream couldn't restore identity.db ($BK_LOG); it is the backup's"3168+ return 03169+ fi3170+ [ -f "$tmp" ] || { say "restore: identity.db is the backup's (litestream has no replica)"; return 0; }3171+ lt=$(node "$FORGE_HOME/lib/identity.ts" changed "$tmp") bk=03172+ if [ -f "$db" ]; then bk=$(node "$FORGE_HOME/lib/identity.ts" changed "$db"); fi3173+ if [ "$lt" -ge "$bk" ]; then3174+ rm -f "$db-wal" "$db-shm"3175+ mv "$tmp" "$db"3176+ say "restore: identity.db is litestream's, which changed last ($(utc "$lt"))"3177+ else3178+ say "restore: identity.db is the backup's, which changed last ($(utc "$bk"); litestream's: $(utc "$lt"))"3179+ fi3180+ rm -f "$tmp" "$tmp-wal" "$tmp-shm"3181+}3182+3183+# forge identity replicate: litestream, copying identity.db to the replica3184+# as it changes (start.sh runs it). forge identity restore [--at <time>]3185+# [--if-missing]: identity.db from the replica, as of now or <time> (UTC,3186+# 2026-09-01T10:00:00Z), with the one it replaces kept as3187+# identity.db.before-restore; stop the forge's servers first, since they3188+# hold the file open. --if-missing does it only where there is none, and3189+# quietly does nothing without a replica (start.sh, before anything makes3190+# an empty file whose replication would bury the replica's).3191+cmd_identity() {3192+ load_config3193+ backup_env3194+ db=$FORGE_STATE/identity.db3195+ case ${1:-} in3196+ replicate)3197+ [ $# -eq 1 ] || die "usage: forge identity replicate"3198+ lt_setup || exit 13199+ [ -f "$db" ] || node "$FORGE_HOME/lib/identity.ts" init3200+ exec litestream replicate -no-expand-env -config "$LT_CONF"3201+ ;;3202+ restore)3203+ shift3204+ at= missing=03205+ while [ $# -gt 0 ]; do3206+ case $1 in3207+ --at) [ $# -ge 2 ] || die "identity restore: --at needs a time"; at=$2; shift ;;3208+ --if-missing) missing=1 ;;3209+ *) die "usage: forge identity restore [--at <time>] [--if-missing]" ;;3210+ esac3211+ shift3212+ done3213+ if [ "$missing" = 1 ]; then3214+ [ ! -f "$db" ] || return 03215+ lt_setup >/dev/null 2>&1 || return 03216+ else3217+ lt_setup || exit 13218+ fi3219+ tmp=$FORGE_STATE/identity.restore.tmp3220+ rm -f "$tmp" "$tmp-wal" "$tmp-shm"3221+ set -- -no-expand-env -config "$LT_CONF" -o "$tmp"3222+ [ -z "$at" ] || set -- "$@" -timestamp "$at"3223+ [ "$missing" = 0 ] || set -- "$@" -if-replica-exists3224+ litestream restore "$@" "$db" || die "identity restore: litestream restore failed"3225+ [ -f "$tmp" ] || { say "identity: litestream has no replica yet"; return 0; }3226+ node "$FORGE_HOME/lib/identity.ts" check "$tmp" || die "identity restore: the restored file fails SQLite's integrity check"3227+ rm -f "$tmp-wal" "$tmp-shm"3228+ if [ -f "$db" ]; then3229+ node "$FORGE_HOME/lib/identity.ts" snapshot "$FORGE_STATE/identity.db.before-restore"3230+ rm -f "$db" "$db-wal" "$db-shm"3231+ fi3232+ rm -rf "$FORGE_STATE/.identity.db-litestream"3233+ chmod 600 "$tmp"3234+ mv "$tmp" "$db"3235+ say "identity.db is back from litestream${at:+ as of $at}"3236+ ;;3237+ *) die "usage: forge identity replicate | restore [--at <time>] [--if-missing]" ;;3238+ esac3239+}3240+30843241 # ---------------------------------------------------------------- pages mirror30853242 #30863243 # The forge's own pages, without sites, previews or hostnames, are synced to@@ -3183,6 +3340,7 @@ backup) cmd_backup "$@" ;;
31833340 backups) cmd_backups "$@" ;;31843341 restore) cmd_restore "$@" ;;31853342 mirror) cmd_mirror "$@" ;;3343+identity) cmd_identity "$@" ;;31863344 shell) cmd_shell "$@" ;;31873345 shell-cmd) cmd_shell_cmd "$@" ;;31883346 hook) cmd_hook "$@" ;;@@ -22,7 +22,7 @@
2222 // node lib/identity.ts sessions [rm <id>|--all]2323 // node lib/identity.ts token [--repo <owner>/<repo>] [--ttl 30d] [name]2424 // node lib/identity.ts tokens [rm <id>]25-// node lib/identity.ts init | purge <user> | snapshot <file> | changed <file>25+// node lib/identity.ts init | purge <user> | snapshot <file> | changed <file> | check <file>2626 import fs from 'node:fs';2727 import path from 'node:path';2828 import { createHash, randomBytes, timingSafeEqual } from 'node:crypto';@@ -199,6 +199,16 @@ export class Identity {
199199 return new Identity(file, !!opts.readOnly);200200 }201201 202+ // SQLite's integrity check of a copy (a backup's): "ok", or what is wrong.203+ static check(file: string): string {204+ const db = new DatabaseSync(file, { readOnly: true });205+ try {206+ return (db.prepare('PRAGMA integrity_check').all() as { integrity_check: string }[]).map((r) => r.integrity_check).join('; ');207+ } finally {208+ db.close();209+ }210+ }211+202212 // When the copy in file last changed, without writing to it.203213 static changedAt(file: string): number {204214 const db = new DatabaseSync(file, { readOnly: true });@@ -536,11 +546,15 @@ if (process.argv[1] && fs.realpathSync(process.argv[1]) === fileURLToPath(import
536546 else if (cmd === 'purge' && args.length === 1) Identity.open(state)!.purge(args[0]);537547 else if (cmd === 'snapshot' && args.length === 1) Identity.open(state)!.snapshot(path.resolve(args[0]));538548 else if (cmd === 'changed' && args.length === 1) console.log(Identity.changedAt(args[0]));549+ else if (cmd === 'check' && args.length === 1) {550+ const r = Identity.check(args[0]);551+ if (r !== 'ok') fail(`${args[0]}: ${r}`);552+ }539553 else if (['passkey', 'passkeys', 'sessions', 'token', 'tokens'].includes(cmd)) {540554 const user = process.env.FORGE_USER;541555 if (!user) fail('no FORGE_USER');542556 sshCommand(state, user!, cmd, args, (s) => process.stdout.write(s), (s) => process.stderr.write(s));543- } else fail('usage: node lib/identity.ts passkey|passkeys|sessions|token|tokens ... | init | purge <user> | snapshot <file> | changed <file>');557+ } else fail('usage: node lib/identity.ts passkey|passkeys|sessions|token|tokens ... | init | purge <user> | snapshot <file> | changed <file> | check <file>');544558 } catch (e) {545559 if (!(e instanceof Usage)) throw e;546560 process.stderr.write(`forge: ${e.message}\n`);@@ -8,8 +8,10 @@
88 # FORGE_TAILCAT_ALLOW comma-separated tailcat client keys (nodekey:...)99 # allowed to reach sshd and Caddy through tailcat1010 # TYPESAFE_API_KEY for hooks and builds (written to .forge/secrets)11-# FORGE_BACKUP_*, FORGE_MIRROR_* backups and the pages mirror (written to12-# .forge/backup.env; README.md, "Backups and moving hosts")11+# FORGE_BACKUP_*, FORGE_MIRROR_* backups, identity.db's litestream replica12+# (FORGE_LITESTREAM_PATH, default litestream/identity)13+# and the pages mirror (written to .forge/backup.env;14+# README.md, "Backups and moving hosts")1315 # FORGE_LAKE_* the lake's data on R2 instead of the volume (also1416 # .forge/backup.env; README.md, "The lake and insights")1517 set -eu@@ -33,6 +35,17 @@ done
3335 # yet, so clear them.3436 rm -rf $root/.forge/build.lock $root/.forge/backup.lock $root/.forge/mirror.lock3537 38+# A box without identity.db (a new one) takes it from litestream's replica,39+# before forge server init makes an empty one: replicating that would bury40+# the replica's. If the replica can't be read, nothing replicates this boot.41+replicate=042+if [ -n "${FORGE_BACKUP_REPOSITORY:-}" ] && command -v litestream >/dev/null 2>&1; then43+ mkdir -p $root/.forge44+ chown git:git $root/.forge45+ if as_git $forge identity restore --if-missing; then replicate=146+ else echo "start: identity.db could not be restored from litestream, so it isn't replicated until the next boot" >&2; fi47+fi48+3649 as_git $forge server init >/dev/null 2>&13750 3851 sites=/sites/{owner}/{repo}/{site}/@@ -69,6 +82,7 @@ FORGE_BACKUP_SECRET=$(q "${FORGE_BACKUP_SECRET:-}")
6982 FORGE_BACKUP_REGION=$(q "${FORGE_BACKUP_REGION:-}")7083 FORGE_BACKUP_KEEP_DAILY=$(q "${FORGE_BACKUP_KEEP_DAILY:-}")7184 FORGE_BACKUP_KEEP_WEEKLY=$(q "${FORGE_BACKUP_KEEP_WEEKLY:-}")85+FORGE_LITESTREAM_PATH=$(q "${FORGE_LITESTREAM_PATH:-}")7286 FORGE_MIRROR_S3=$(q "${FORGE_MIRROR_S3:-}")7387 FORGE_MIRROR_KEY_ID=$(q "${FORGE_MIRROR_KEY_ID:-}")7488 FORGE_MIRROR_SECRET=$(q "${FORGE_MIRROR_SECRET:-}")@@ -122,6 +136,11 @@ touch $root/.forge/events.jsonl
122136 chown git:git $root/.forge/events.jsonl123137 runuser -u git -- env FORGE_STATE=$root/.forge node /opt/forge/lib/events-server.ts >>$root/.forge/events.log 2>&1 &124138 139+# identity.db, copied to the backup bucket as it changes (forge identity).140+if [ "$replicate" = 1 ]; then141+ as_git sh -c "exec $forge identity replicate" >>$root/.forge/litestream.log 2>&1 &142+fi143+125144 # Private repos' pages for their readers, and the 404 any other missing page126145 # gets: Caddy hands it every path www has no file for. Runs as git, which127146 # owns the private pages and identity.db.@@ -0,0 +1,43 @@
1+# A backup carries identity.db as SQLite copies it (VACUUM INTO), one2+# consistent file, never the live one with its WAL, nor litestream's state;3+# a restore brings it back, so passkeys, sessions and tokens work on the4+# new forge. Without litestream set up, the backup's copy is the one. The5+# private pages are rebuilt, not backed up.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo --private dami/secret9+clone dami dami/secret "$T/w/dami"10+at "$T/w/dami" commit "first"11+ok push_as dami "$T/w/dami" origin main12+build13+tok=$(ssh_as dami token ci 2>/dev/null)14+db=$FORGE_ROOT/.forge/identity.db15+# A reader holds the file open, as the servers do, so the WAL is there.16+node -e '17+const { DatabaseSync } = require("node:sqlite");18+const d = new DatabaseSync(process.argv[1]);19+d.prepare("SELECT count(*) FROM tokens").get();20+setTimeout(() => {}, 30000);' "$db" 2>/dev/null &21+T_PIDS="$T_PIDS $!"22+mkdir -p "$FORGE_ROOT/.forge/.identity.db-litestream"23+24+ok forge backup --dry-run25+st=$FORGE_ROOT/.forge/backup/stage26+exists "$st/.forge/identity.db"27+missing "$st/.forge/identity.db-wal"28+missing "$st/.forge/identity.db-shm"29+missing "$st/.forge/.identity.db-litestream"30+missing "$st/.forge/private"31+missing "$FORGE_ROOT/.forge/identity.snapshot.tmp"32+eq "$(stat -f %Lp "$st/.forge/identity.db" 2>/dev/null || stat -c %a "$st/.forge/identity.db")" 60033+ok node "$T_HOME/lib/identity.ts" check "$st/.forge/identity.db"34+35+r=$T/restored36+ok env FORGE_ROOT="$r" FORGE_WWW="$r/.forge/www" FORGE_CACHE="$T/cache2" "$FORGE" restore --from "$st"37+exists "$r/.forge/identity.db"38+exists "$r/.forge/private/www/dami/secret/index.html"39+export FORGE_ROOT=$r FORGE_WWW=$r/.forge/www40+private_server41+eq "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $tok" "$PRIV/dami/secret/")" 20042+ok ssh_as dami tokens43+has "$T_LOG" ' ci$'@@ -0,0 +1,65 @@
1+# With litestream installed: forge identity replicate copies identity.db2+# to a replica next to the backup repository as it changes, forge identity3+# restore brings it back as it is now or as it was at a moment, and forge4+# restore takes whichever of the backup's copy and the replica changed last.5+# FORGE_TEST_LITESTREAM names a litestream binary to use when none is on PATH.6+. "$(dirname "$0")/../lib.sh"7+if [ -n "${FORGE_TEST_LITESTREAM:-}" ]; then8+ mkdir -p "$T/bin"9+ ln -s "$FORGE_TEST_LITESTREAM" "$T/bin/litestream"10+ PATH=$T/bin:$PATH11+fi12+command -v litestream >/dev/null 2>&1 || { note "litestream is not installed: skipped"; exit 0; }13+user dami14+export FORGE_BACKUP_REPOSITORY=$T/restic FORGE_BACKUP_PASSWORD=test15+replicate() {16+ "$FORGE" identity replicate >>"$T/litestream.log" 2>&1 &17+ T_PIDS="$T_PIDS $!"18+ sleep 219+}20+# The names of dami's tokens in the forge at $1 (this one by default).21+tokens() { FORGE_STATE=${1:-$FORGE_ROOT}/.forge FORGE_USER=dami node "$T_HOME/lib/identity.ts" tokens | awk 'NR > 1 { print $NF }' | sort | tr '\n' ' '; }22+stage() { env -u FORGE_BACKUP_REPOSITORY "$FORGE" backup --dry-run --force >/dev/null 2>&1 && rm -rf "$1" && mv "$FORGE_ROOT/.forge/backup/stage" "$1"; }23+24+replicate25+ok ssh_as dami token a26+sleep 227+stage "$T/stage-a"28+mid=$(date -u +%Y-%m-%dT%H:%M:%SZ)29+sleep 230+export FORGE_NOW=$((T_NOW + 60))31+ok ssh_as dami token b32+sleep 333+stop_servers34+exists "$T/restic-litestream"35+has "$FORGE_ROOT/.forge/litestream.yml" "path: $T/restic-litestream"36+37+# As it was, then as it is.38+ok forge identity restore --at "$mid"39+eq "$(tokens)" "a "40+exists "$FORGE_ROOT/.forge/identity.db.before-restore"41+ok forge identity restore42+eq "$(tokens)" "a b "43+ok forge identity restore --if-missing44+eq "$(output)" ""45+46+# forge restore: the replica changed after the backup's copy (a, b > a)...47+r=$T/r148+ok env FORGE_ROOT="$r" FORGE_WWW="$r/.forge/www" FORGE_CACHE="$T/cache2" "$FORGE" restore --from "$T/stage-a"49+has "$T_LOG" "identity.db is litestream's, which changed last"50+eq "$(tokens "$r")" "a b "51+# ...and when replication had stopped, the backup's copy wins.52+export FORGE_NOW=$((T_NOW + 120))53+ok ssh_as dami token c54+stage "$T/stage-c"55+r=$T/r256+ok env FORGE_ROOT="$r" FORGE_WWW="$r/.forge/www" FORGE_CACHE="$T/cache3" "$FORGE" restore --from "$T/stage-c"57+has "$T_LOG" "identity.db is the backup's, which changed last"58+eq "$(tokens "$r")" "a b c "59+60+# A new box: no identity.db, so it comes from the replica before anything makes one.61+r=$T/r362+mkdir -p "$r/.forge"63+ok env FORGE_ROOT="$r" "$FORGE" identity restore --if-missing64+exists "$r/.forge/identity.db"65+eq "$(tokens "$r")" "a b "