| M | README.md | 70 | ++++++++++++++++------- |
| M | bin/ | 85 | ++++++++++++++++++++-------- |
| M | deploy/ | 15 | ++++- |
| M | guide/ | 6 | +- |
| A | lib/ | 91 | ++++++++++++++++++++++++++++++ |
| M | server/ | 1 | + |
| M | test/ | 61 | ++++++++++++++++++++ |
| A | test/ | 39 | +++++++++++++ |
| M | test/ | 14 | ++--- |
| A | test/ | 15 | +++++ |
| A | test/ | 18 | ++++++ |
| A | test/ | 38 | +++++++++++++ |
| A | test/ | 32 | +++++++++++ |
| A | test/ | 24 | ++++++++ |
| M | test/ | 21 | ++++--- |
| M | test/ | 38 | ++++++------ |
| M | test/ | 14 | ++--- |
@@ -103,7 +103,7 @@ review page, signed in ─── HTTPS ────────────> the
103103 | `lib/private-server.ts` | Serves private repos' pages to their readers, signed in with a passkey or holding a token, the sign-in pages (`/login/`, `/private/`), and the 404 of every page that isn't there; takes comments and approvals posted from review pages (see [Comments and approvals](#comments-and-approvals)); answers Caddy whether a token may clone a private repo over HTTPS (see [Private pages](#private-pages)). |104104 | `lib/pages-server.ts` | Only with a pages domain. Serves each site on its own hostname the way Cloudflare Pages does (`_headers`, `_redirects`, pretty URLs, the nearest `404.html` or a single-page-app fallback) and redirects old `/sites/...` paths there. Runs as the caddy user, since deploys are untrusted content. |105105 | `forge backup` | Daily, from a loop in `start.sh`. Snapshots the forge with restic into a bucket, then checks the snapshots and test-restores the new one (see [Backups and moving hosts](#backups-and-moving-hosts)). |106-| `forge mirror` | After builds, in the background: syncs the forge's own pages to a bucket with rclone, for the edge to serve while the forge is down. |106+| `forge mirror` | After builds, in the background, at most hourly: syncs the forge's own pages, less those of single files and folders, to a bucket with rclone, for the edge to serve while the forge is down. |107107 | `forge lake` | After builds, in the background: adds what is new in the event log, build metrics, review logs and CI notes to the lake, and writes the insights exports (see [The lake and insights](#the-lake-and-insights)). |108108 | `forge identity replicate` | With backups set up: [litestream](https://litestream.io) copies `.forge/identity.db` to the backup bucket as it changes (see [identity.db](#identitydb-backed-up-as-it-changes)). |109109 @@ -234,7 +234,7 @@ as before.
234234 forge (without `--yes` it says what would go); collaborators can't, and235235 `forge repo rm <owner>/<repo>` does the same as root. Its clones, pages in236236 either root, sites, previews, hostnames, release downloads and insights go237-at once, and the mirror's next sync deletes its pages. The bare repo waits in237+at once, and a mirror sync within 30 seconds deletes its pages. The bare repo waits in238238 `.forge/trash/<owner>/<repo>.git.<time>` for `FORGE_TRASH_DAYS` (14) days,239239 backed up like the rest of the forge: `forge repo restore <owner>/<repo>`240240 puts back the newest removal if the name is free, and `forge gc` deletes it@@ -278,7 +278,7 @@ pages and `repos.json`, and its served sites, previews and their
278278 hostnames, moves its release downloads into the private root (and their279279 files into the private store, see [Releases](#releases)), and builds280280 its pages again there. Then it starts281-a mirror sync, which deletes them from the bucket. Clones, caches and copies281+a mirror sync within 30 seconds, which deletes them from the bucket. Clones, caches and copies282282 people already have can't be recalled. `public` first runs the secret check283283 over the whole history: every commit on a branch or tag, and the old review284284 revisions that review pages show, not only what a push adds. If it finds@@ -931,7 +931,10 @@ On the edge, Caddy takes the public HTTP port and passes everything through
931931 the tunnel. When slate doesn't answer (the tunnel or the box is down, a932932 timeout, or a 502, 503 or 504), Caddy serves GET and HEAD requests for the933933 forge's own pages from the [pages mirror](#the-pages-mirror) instead, with934-`X-Forge-Offline: mirror` and `Cache-Control: no-store`. Sites, clones, live934+`X-Forge-Offline: mirror` and `Cache-Control: no-store`. A page the copy935+hasn't got, one it leaves out like a file's, or one that isn't there at all,936+is a 404: the forge's not-found page, saying the forge is offline and the937+page isn't in its offline copy. Sites, clones, live935938 updates and pushes need slate; they get a 503, or for ssh no connection,936939 until it's back. Caddy checks slate every 10 seconds, so only the first937940 request after it goes away waits. `deploy/edge/fly.toml` sets `FORGE_DOMAIN`@@ -1183,23 +1186,48 @@ stopped. Without backups set up, identity.db stays local.
11831186 11841187 ### The pages mirror11851188 1186-After a build, `forge mirror` syncs the forge's own pages (not sites, previews1187-or their hostnames) to a second bucket with rclone, in the background. It1188-waits `FORGE_MIRROR_DELAY` seconds (30) after each build and only the newest1189-one syncs, so a burst of pushes makes one sync. It never replaces the copy1190-with an empty forge. The edge serves this copy while slate is down (see1191-[above](#home-machine-behind-a-public-edge-how-srcrest-runs)). It holds1189+After a build, `forge mirror` syncs the forge's own pages to a second bucket1190+with rclone, in the background, for the edge to serve while slate is down1191+(see [above](#home-machine-behind-a-public-edge-how-srcrest-runs)). It holds11921192 public content only, the pages the forge serves anyway, and public repos'1193-release downloads. When a repo goes private, the next sync deletes its1194-pages and downloads there.1195-1196-A sync is two `rclone sync` passes. The pages go by checksum, since a1197-rebuilt page that came out the same has a new mtime and shouldn't be1198-uploaded again; that reads every file. So the downloads, which can be big1199-and never change in place, are left out of it (`/*/*/releases/download/**`)1200-and synced in a second pass by size and mtime, rclone's default. Their1201-mtime is their release commit's, so an unchanged binary isn't read. Each1202-pass leaves the other's files in the bucket alone. The edge serves1193+release downloads. It never replaces the copy with an empty forge.1194+1195+The copy only has to keep the forge readable for a while, so it leaves out1196+the pages there is one of per file. They were nine in ten of its files, and1197+since every page links the stylesheet by its hash, each style change made1198+the next sync upload all of them again. What `lib/mirror.ts` puts in:1199+1200+| In the copy | Left out |1201+|---|---|1202+| `/`, owner pages, `/topics/`, `/insights/` with its exports, `/design/`, `/assets/` (scripts, styles, fonts), `repos.json` | `/sites/`, `/previews/`, `/hosts/`: sites need slate |1203+| each repo's home (with its README), `log/`, `commit/`, `reviews/` and each review, `releases/` and their downloads, `refs/`, `sites/`, `wiki/`, `insights/`, `atom.xml`, `search.json`, `code.db`, `reviews.json`, branch menus, `files/<branch>.txt` | `/<owner>/<repo>/blob/**`: file pages |1204+| `tree/<branch>/`, each branch's top-level file list | `tree/<branch>/<folder>/**`: folder pages |1205+| the images a page in the copy shows, a README's or a wiki page's | the rest of `raw/**` |1206+1207+A page left out gets the edge's offline 404. Whatever else is in the bucket1208+goes at the next sync: pages from before they were left out (once), and1209+those of a repo that went private or away.1210+1211+**When it syncs.** Each build waits `FORGE_MIRROR_DELAY` seconds (600) and1212+only the newest goes on, so a burst of pushes makes one sync. Then it waits1213+until `FORGE_MIRROR_MIN_INTERVAL` seconds (3600) have passed since the last1214+sync started; a build meanwhile takes its place, so there is one sync1215+pending at most, and it runs when the interval ends. A sync that started1216+after a build has its pages already, and that build's wait ends there. A1217+repo that goes private, or is removed, makes a sync within 30 seconds1218+whatever the interval, so its pages leave the copy then. `forge mirror` by1219+hand syncs now. One sync at a time (`.forge/mirror.lock`); the background1220+ones log to `.forge/mirror.log`.1221+1222+A sync is three rclone steps. The pages on the list go by checksum, since1223+a rebuilt page that came out the same has a new mtime and shouldn't be1224+uploaded again; that reads every file. Then what else is in the bucket is1225+deleted, except the downloads. The downloads, which can be big and never1226+change in place, are synced on their own by size and mtime, rclone's1227+default. Their mtime is their release commit's, so an unchanged binary isn't1228+read. Uploads skip rclone's HEAD of each object afterwards (`--s3-no-head`):1229+a page is one PUT, whose response has its MD5, and the next sync compares1230+checksums with the ETags in the bucket's listing. The edge serves12031231 downloads from the copy like any file, as attachments.12041232 12051233 ### Setting it up@@ -1227,7 +1255,7 @@ What you set up yourself, once:
12271255 FORGE_MIRROR_S3=https://<account-id>.r2.cloudflarestorage.com/forge-pages12281256 FORGE_MIRROR_KEY_ID=<access key id>12291257 FORGE_MIRROR_SECRET=<secret access key>1230- # optional: FORGE_BACKUP_AT=03:30 FORGE_BACKUP_KEEP_DAILY=7 FORGE_BACKUP_KEEP_WEEKLY=5 FORGE_MIRROR_DELAY=301258+ # optional: FORGE_BACKUP_AT=03:30 FORGE_BACKUP_KEEP_DAILY=7 FORGE_BACKUP_KEEP_WEEKLY=5 FORGE_MIRROR_DELAY=600 FORGE_MIRROR_MIN_INTERVAL=360012311259 # FORGE_LITESTREAM_PATH=litestream/identity12321260 ```12331261 @@ -1448,8 +1448,8 @@ cmd_repo() {
14481448 # going private takes a repo's public side away: its pages in either root14491449 # (with its release downloads), its sites, previews and their hostnames,14501450 # its insights exports and its caches. The index, owner pages and1451-# repos.json are rebuilt without it, and the mirror's next sync deletes its1452-# pages from the bucket.1451+# repos.json are rebuilt without it, and the mirror syncs within 30 seconds,1452+# which deletes its pages from the bucket.14531453 repo_rm() {14541454 owner=${1%%/*} name=${1#*/}14551455 valid_name "$owner" && valid_name "$name" && [ "$owner/$name" = "$1" ] && [ -f "$FORGE_ROOT/$1.git/HEAD" ] || die "no repo $1"@@ -1470,7 +1470,7 @@ repo_rm() {
14701470 "${FORGE_STATE:?}/lake/export/$1" "${FORGE_STATE:?}/cache/repos/$1.json" "${FORGE_STATE:?}/cache/code/$1"14711471 rm -f "$FORGE_WWW"/insights/*.json "$FORGE_WWW"/insights/*.parquet14721472 rebuild - removed1473- mirror_kick1473+ mirror_kick soon14741474 echo "removed $1; forge repo restore $1 brings it back within ${FORGE_TRASH_DAYS:-14} days (the trash: ${trash#"$FORGE_ROOT"/}/$name.git.$t)"14751475 }14761476 @@ -1956,10 +1956,10 @@ redeploy_sites() {
19561956 # private <owner>/<repo>: only its owner, collaborators and readers can see it19571957 # from now on. Its pages go, the index, owner pages and repos.json are19581958 # rebuilt without it, its sites, previews and their hostnames go unless1959-# forge.publicsites, and the mirror's next sync deletes them from the bucket.1960-# Its pages are built again into the private root, for its readers, and its1961-# release downloads move there. The events server streams it only to them1962-# from now on (it checks the flag).1959+# forge.publicsites, and the mirror syncs within 30 seconds, which deletes1960+# them from the bucket. Its pages are built again into the private root,1961+# for its readers, and its release downloads move there. The events server1962+# streams it only to them from now on (it checks the flag).19631963 sc_private() {19641964 [ $# -eq 1 ] || die "usage: private <owner>/<repo>"19651965 open_admin "$1"@@ -1985,9 +1985,9 @@ sc_private() {
19851985 rm -f "$FORGE_WWW"/insights/*.json "$FORGE_WWW"/insights/*.parquet19861986 rebuild - private19871987 rebuild "$1" private1988- mirror_kick1988+ mirror_kick soon19891989 echo "$1 is private: only ${1%%/*}, its collaborators and its readers (readers $1 add <user>) can clone it, and see its pages once they sign in."1990- echo "Its public pages and sites are gone from here, and from the mirror at its next sync. Clones and copies people already have can't be recalled."1990+ echo "Its public pages and sites are gone from here, and from the mirror within a minute. Clones and copies people already have can't be recalled."19911991 }19921992 19931993 # public <owner>/<repo> [--force]: anyone can clone it and its pages come@@ -3717,22 +3717,44 @@ cmd_identity() {
37173717 37183718 # ---------------------------------------------------------------- pages mirror37193719 #3720-# The forge's own pages, without sites, previews or hostnames, are synced to3721-# a bucket that the edge serves while this box is down (deploy/edge). Builds3722-# start it: each stamps mirror.want and waits FORGE_MIRROR_DELAY seconds, and3723-# only the newest stamp syncs, so a burst of pushes makes one sync.3720+# The forge's own pages, without sites, previews or hostnames, and without3721+# the pages there is one of per file (lib/mirror.ts), are synced to a bucket3722+# that the edge serves while this box is down (deploy/edge). Builds start it:3723+# each stamps mirror.want and waits FORGE_MIRROR_DELAY seconds (600), and only3724+# the newest stamp goes on, so a burst of pushes makes one sync. That sync3725+# then waits until FORGE_MIRROR_MIN_INTERVAL seconds (3600) have passed since3726+# the last one started (mirror.last); a build meanwhile restarts the wait and3727+# takes its place, so one sync is pending at most. A sync that started after3728+# the stamp has the build's pages already, and the waiter leaves it at that.3729+#3730+# `mirror_kick soon` is for a repo gone private or removed: its pages leave3731+# the copy within 30 seconds (or FORGE_MIRROR_DELAY if shorter), whatever the3732+# interval. `forge mirror` by hand waits for neither.37243733 37253734 mirror_kick() {37263735 (backup_env && [ -n "${FORGE_MIRROR_S3:-}" ]) || return 03727- t=$$.$(date +%s)3728- printf '%s\n' "$t" >"$FORGE_STATE/mirror.want"3736+ now=$(date +%s)3737+ t=$$.$now3738+ w=$FORGE_STATE/mirror.want3739+ [ "${1:-}" != soon ] || w=$FORGE_STATE/mirror.soon3740+ printf '%s\n' "$t" >"$w"37293741 (37303742 # Not the build's EXIT trap: that frees the build lock.37313743 trap - EXIT37323744 backup_env37333745 cd /3734- sleep "${FORGE_MIRROR_DELAY:-30}"3735- [ "$(cat "$FORGE_STATE/mirror.want" 2>/dev/null)" = "$t" ] || exit 03746+ d=${FORGE_MIRROR_DELAY:-600}3747+ gap=${FORGE_MIRROR_MIN_INTERVAL:-3600}3748+ [ "${1:-}" != soon ] || { [ "$d" -le 30 ] || d=30; gap=0; }3749+ # Still the newest stamp, and no sync has started since it.3750+ due() { [ "$(cat "$w" 2>/dev/null)" = "$t" ] && [ "$(cat "$FORGE_STATE/mirror.last" 2>/dev/null || echo 0)" -le "$now" ]; }3751+ sleep "$d"3752+ due || exit 03753+ left=$(($(cat "$FORGE_STATE/mirror.last" 2>/dev/null || echo 0) + gap - $(date +%s)))3754+ if [ "$left" -gt 0 ]; then3755+ sleep "$left"3756+ due || exit 03757+ fi37363758 exec "$FORGE_BIN" mirror37373759 ) </dev/null >>"$FORGE_STATE/mirror.log" 2>&1 &37383760 }@@ -3754,23 +3776,40 @@ cmd_mirror() {
37543776 trap 'rm -rf "$FORGE_STATE/mirror.lock"' EXIT37553777 trim_log "$FORGE_STATE/mirror.log"37563778 t0=$(date +%s)3779+ [ -n "$dry" ] || echo "$t0" >"$FORGE_STATE/mirror.last"3780+ b=:s3:${u#*/}37573781 export RCLONE_S3_PROVIDER="${FORGE_MIRROR_PROVIDER:-Cloudflare}" RCLONE_S3_ENDPOINT="${FORGE_MIRROR_S3%%://*}://${u%%/*}" \37583782 RCLONE_S3_ACCESS_KEY_ID="${FORGE_MIRROR_KEY_ID:-}" RCLONE_S3_SECRET_ACCESS_KEY="${FORGE_MIRROR_SECRET:-}" \37593783 RCLONE_S3_REGION="${FORGE_MIRROR_REGION:-auto}" RCLONE_S3_NO_CHECK_BUCKET=true3784+ # What goes in (lib/mirror.ts), sorted as comm wants it.3785+ keep=$FORGE_STATE/mirror.keep3786+ node "$FORGE_HOME/lib/mirror.ts" "$FORGE_WWW" >"$keep.tmp" || die "mirror: listing the pages failed"3787+ LC_ALL=C sort "$keep.tmp" >"$keep" && rm -f "$keep.tmp"3788+ grep -qx index.html "$keep" || die "mirror: the list of pages has no index.html"37603789 # Pages by checksum: a rebuilt page that came out the same has a new37613790 # mtime, and isn't uploaded again. That reads every file, so release37623791 # downloads, which can be big and never change in place, are left to a37633792 # second pass by size and mtime (rclone's default): theirs is their37643793 # release commit's time (store_tree), so unchanged ones aren't read.3765- # Each pass leaves what the other's filters cover alone in the bucket.3766- rclone sync "$FORGE_WWW" ":s3:${u#*/}" --config "" --checksum --fast-list --transfers 16 --checkers 32 $dry \3767- --exclude '/sites/**' --exclude '/previews/**' --exclude '/hosts/**' --exclude '*.old/**' \3768- --exclude '/*/*/releases/download/**' ||3794+ # --s3-no-head: no HEAD after each upload; a page is one PUT, whose3795+ # response has its MD5, and the next sync compares the listing's ETags.3796+ rclone sync "$FORGE_WWW" "$b" --config "" --files-from-raw "$keep" --checksum --fast-list --s3-no-head \3797+ --transfers 16 --checkers 32 $dry ||37693798 die "mirror: rclone sync failed"3770- rclone sync "$FORGE_WWW" ":s3:${u#*/}" --config "" --fast-list --transfers 4 --checkers 32 $dry \3799+ # The sync leaves what isn't on the list alone in the bucket, so whatever3800+ # else is there goes: pages left out (once, after they were left out),3801+ # and those of repos gone private or away. Not release downloads: those3802+ # are the second pass's.3803+ rclone lsf "$b" --config "" -R --files-only --fast-list >"$keep.remote" || die "mirror: listing the bucket failed"3804+ grep -v '^[^/]*/[^/]*/releases/download/' "$keep.remote" | LC_ALL=C sort | LC_ALL=C comm -23 - "$keep" >"$keep.gone" || true3805+ if [ -s "$keep.gone" ]; then3806+ rclone delete "$b" --config "" --files-from-raw "$keep.gone" --fast-list --checkers 32 $dry ||3807+ die "mirror: deleting $(wc -l <"$keep.gone" | tr -d ' ') files from the bucket failed"3808+ fi3809+ rclone sync "$FORGE_WWW" "$b" --config "" --fast-list --s3-no-head --transfers 4 --checkers 32 $dry \37713810 --filter '- *.old/**' --filter '- *.tmp.*/**' --filter '+ /*/*/releases/download/**' --filter '- **' ||37723811 die "mirror: rclone sync of the release downloads failed"3773- say "$(date -u +%Y-%m-%dT%H:%M:%SZ) mirror: synced $FORGE_WWW to ${u#*/} in $(($(date +%s) - t0))s"3812+ say "$(date -u +%Y-%m-%dT%H:%M:%SZ) mirror: synced $(wc -l <"$keep" | tr -d ' ') files of $FORGE_WWW to ${u#*/}, deleted $(wc -l <"$keep.gone" | tr -d ' '), in $(($(date +%s) - t0))s"37743813 }37753814 37763815 cmd_hook() {@@ -3,7 +3,8 @@
33 # the forge doesn't answer (the tunnel or slate is down, a timeout, or it44 # says 502, 503 or 504), GET and HEAD for the forge's own pages are served55 # from the pages mirror instead (forge mirror; README.md), marked with6-# X-Forge-Offline. Sites, clones, live updates and pushes need the forge.6+# X-Forge-Offline; a page the copy hasn't got is a 404 that says so. Sites,7+# clones, live updates and pushes need the forge.78 # FORGE_DOMAIN the forge's hostname, e.g. src.rest; only it falls back89 # FORGE_MIRROR_URL the mirror bucket's public URL, e.g. https://mirror.src.rest;910 # without it there is no fallback@@ -59,9 +60,19 @@
5960 reverse_proxy {$FORGE_MIRROR_URL:https://mirror.invalid} {6061 header_up Host {upstream_hostport}6162 header_down -Set-Cookie63+ # Not in the copy: a page that isn't there, or one the copy64+ # leaves out (a file's or a folder's; lib/mirror.ts). The65+ # forge's own 404, saying so.6266 @miss status 403 4046367 handle_response @miss {64- respond "The forge is offline, and this page isn't in its offline copy. Try again soon." 50368+ header Content-Type "text/html; charset=utf-8"69+ respond <<HTML70+ <!DOCTYPE html>71+ <html lang="en">72+ <head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Not found</title></head>73+ <body><h1>Not found</h1><p>The forge is offline for a moment, and this page isn't in its offline copy, which leaves out the pages of single files and folders. Try again soon.</p><p><a href="/">All repositories</a></p></body>74+ </html>75+ HTML 4046576 }6677 }6778 }@@ -131,7 +131,7 @@ deleted 1788292812 dami <span class="dim"># only o
131131 <tr><th scope="row">pages server</th><td data-l="listens"><code>127.0.0.1:8092</code></td><td data-l="as">caddy</td><td>Only with <code>FORGE_PAGES_DOMAIN</code>: sites on their own hostnames. Runs as caddy because deploys are untrusted.</td></tr>132132 <tr><th scope="row"><code>forge build</code></th><td data-l="listens"></td><td data-l="as">git</td><td>Once at boot, in the background, since templates may have changed with the image; after that, after each push.</td></tr>133133 <tr><th scope="row">backup loop</th><td data-l="listens"></td><td data-l="as">root</td><td>Only with <code>FORGE_BACKUP_REPOSITORY</code>: runs <code>forge backup</code> once a day at <code>FORGE_BACKUP_AT</code>, catching up after downtime, and prints its health to the container log.</td></tr>134-<tr><th scope="row"><code>forge mirror</code></th><td data-l="listens"></td><td data-l="as">git</td><td>Only with <code>FORGE_MIRROR_S3</code>: after builds, syncs the forge's pages to the bucket the edge falls back to.</td></tr>134+<tr><th scope="row"><code>forge mirror</code></th><td data-l="listens"></td><td data-l="as">git</td><td>Only with <code>FORGE_MIRROR_S3</code>: after builds, at most hourly, syncs the forge's pages to the bucket the edge falls back to.</td></tr>135135 <tr><th scope="row">Caddy</th><td data-l="listens"><code>:8080</code></td><td data-l="as">caddy</td><td>In the foreground. Plain HTTP; TLS ends at the edge. Pages, <code>.git</code> files, <code>go get</code> meta tags, and the two proxies.</td></tr>136136 </tbody>137137 </table>@@ -202,7 +202,7 @@ deleted 1788292812 dami <span class="dim"># only o
202202 <section aria-labelledby="k-backup"><h3 id="k-backup">forge.env, for backups</h3><dl>203203 <div><dt>FORGE_BACKUP_REPOSITORY, _PASSWORD, _KEY_ID, _SECRET</dt><dd>The restic repository (an R2 bucket), its password, and the bucket's keys. Keep the password off the box.</dd></div>204204 <div><dt>FORGE_BACKUP_AT, _KEEP_DAILY, _KEEP_WEEKLY</dt><dd>When the daily backup runs (UTC) and what retention keeps: 03:30, 7 and 5.</dd></div>205-<div><dt>FORGE_MIRROR_S3, _KEY_ID, _SECRET, _DELAY</dt><dd>The pages mirror's bucket and keys, and how long it waits after a build: 30 seconds.</dd></div>205+<div><dt>FORGE_MIRROR_S3, _KEY_ID, _SECRET, _DELAY, _MIN_INTERVAL</dt><dd>The pages mirror's bucket and keys, how long it waits after the last build, 600 seconds, and how long at least between syncs, 3600 seconds.</dd></div>206206 </dl></section>207207 <section aria-labelledby="k-env"><h3 id="k-env">the environment, for tests</h3><dl>208208 <div><dt>FORGE_NOW</dt><dd>Pins the clock reviews are stamped with; the demo moves it with its commits.</dd></div>@@ -227,7 +227,7 @@ deleted 1788292812 dami <span class="dim"># only o
227227 <p><strong>Repo settings</strong> come from <code>repo_setup</code>, when a repo is created (<code>repo new</code>, <code>ssh new</code>, a push to a missing repo, an import) and for every repo on <code>forge server init</code>, which the container runs on boot: the hooks, <code>receive.advertisePushOptions</code>, <code>transfer.hideRefs refs/forge</code> so clones, fetches and pushes never see the pinned revisions (the build reads them directly; pre-receive also refuses them), and <code>commitGraph.changedPathsVersion 2</code>, filters that hash non-ASCII paths the same on every platform.</p>228228 <p><strong>Backups</strong>: <code>forge backup</code> stages a copy of <code>/srv/git</code> without what a restore rebuilds (pages, extracted sites, caches, generated keys files, hooks, logs), uploads it with restic, prunes old snapshots, runs <code>restic check</code>, and restores the new snapshot into a scratch directory to <code>git fsck</code> every repo and compare its refs. The copy is consistent mid-push: per repo it copies review logs and comments, then reads the refs, then hard-links the objects, which git writes before anything names them. It refuses a forge with no repos, or half the last backup's, so a new box can't push good snapshots out. <code>forge-admin backups</code> says how the last one went, and loudly when it failed or is a day and a half old.</p>229229 <p><strong>Moving</strong>: <code>forge restore latest</code> on an empty forge unpacks a snapshot and rebuilds the rest: what <code>server init</code> and <code>keys sync</code> write, hooks, pinned revisions, site files, pages. The restored tailcat key keeps the tunnel address, so the edge follows the forge to a new box by itself. The README walks through a new slate and a VPS.</p>230-<p><strong>The offline copy</strong>: after builds, <code>forge mirror</code> syncs the forge's own pages, without sites or previews, to an R2 bucket. When slate doesn't answer, the edge's Caddy serves them from there with <code>X-Forge-Offline: mirror</code>; sites, clones and pushes wait for slate.</p>230+<p><strong>The offline copy</strong>: after builds, at most hourly, <code>forge mirror</code> syncs the forge's own pages, without sites, previews, or the pages of single files and folders, to an R2 bucket. When slate doesn't answer, the edge's Caddy serves them from there with <code>X-Forge-Offline: mirror</code>, and a page the copy leaves out is a 404 that says so; sites, clones and pushes wait for slate.</p>231231 <p><strong>Updates</strong>: <code>deploy/slate/deploy.sh</code> rsyncs the checkout to the box and rebuilds the container; the boot build refreshes every page. A change to <code>lib/</code> or <code>site/</code> changes the code hash in every fingerprint, so that first build renders everything.</p>232232 <p><strong>Logs</strong>: <code>.forge/build.log</code> for builds, <code>.forge/events.log</code> for the events server, and <code>backup.log</code> and <code>mirror.log</code>, kept under a megabyte. <code>events.jsonl</code> is cut back to its last thousand lines once it passes a megabyte.</p>233233 </div>@@ -0,0 +1,91 @@
1+// What forge mirror puts in the offline copy (README.md, The pages mirror):2+// the forge's own pages, less the ones there is one of per file. Those are3+// most of www, and every page links the stylesheet by its hash, so each4+// style change rewrote them all and the next sync uploaded them all again.5+// The copy only has to keep the forge readable while it is down.6+//7+// in everything else: /, owner pages, topics, insights and their exports,8+// /design/, /assets/, feeds, search indexes, and each repo's home,9+// log, commits, reviews, releases, refs, sites, wiki, file lists10+// out /<owner>/<repo>/blob/** (file pages)11+// /<owner>/<repo>/tree/<branch>/<dir>/** (directory pages; each12+// branch's top-level listing stays)13+// /<owner>/<repo>/raw/** (images), except those a page in the copy14+// shows, such as a README's15+// /sites/**, /previews/**, /hosts/**, and *.old and *.tmp.* leftovers16+// and /<owner>/<repo>/releases/download/** go in a pass of their own17+//18+// node lib/mirror.ts <www> the files in, one path per line, relative to www19+import fs from 'node:fs';20+import path from 'node:path';21+import { fileURLToPath } from 'node:url';22+23+const SKIP_TOP = new Set(['sites', 'previews', 'hosts']);24+25+// Whether a directory (its path's segments) can hold anything that goes in.26+function walkInto(s: string[]): boolean {27+ const name = s[s.length - 1];28+ if (name.endsWith('.old') || name.includes('.tmp.')) return false;29+ if (s.length === 1) return !SKIP_TOP.has(name);30+ if (s.length === 3) return name !== 'blob';31+ if (s.length === 4 && s[2] === 'releases') return name !== 'download';32+ if (s.length === 5 && s[2] === 'tree') return false;33+ return true;34+}35+36+// The URLs a page's attributes point to.37+const ATTR = /\b(?:src|srcset|href)="([^"]+)"/g;38+function urlsIn(html: string): string[] {39+ const out: string[] = [];40+ for (const m of html.matchAll(ATTR)) {41+ for (const part of m[1].split(',')) out.push(part.trim().split(/\s+/)[0].replaceAll('&', '&'));42+ }43+ return out;44+}45+46+function fromUrl(u: string): string | null {47+ if (!u.startsWith('/') || u.startsWith('//')) return null;48+ try {49+ return u.replace(/[?#].*$/, '').slice(1).split('/').map(decodeURIComponent).join('/');50+ } catch {51+ return null;52+ }53+}54+55+export function mirrorFiles(www: string): string[] {56+ const keep: string[] = [];57+ const raw = new Set<string>();58+ const walk = (s: string[]): void => {59+ for (const e of fs.readdirSync(path.join(www, ...s), { withFileTypes: true })) {60+ const t = [...s, e.name];61+ // rclone skips symlinks, as it always has here.62+ if (e.isDirectory()) {63+ if (walkInto(t)) walk(t);64+ } else if (e.isFile()) {65+ if (t.length > 3 && t[2] === 'raw') raw.add(t.join('/'));66+ else if (t.length === 5 && t[2] === 'tree' ? t[4] === 'index.html' : true) keep.push(t.join('/'));67+ }68+ }69+ };70+ walk([]);71+ // The images the pages that go in show, a README's or a wiki page's.72+ const shown = new Set<string>();73+ for (const f of keep) {74+ if (!f.endsWith('.html')) continue;75+ for (const u of urlsIn(fs.readFileSync(path.join(www, f), 'utf8'))) {76+ const p = fromUrl(u);77+ if (p && raw.has(p)) shown.add(p);78+ }79+ }80+ return [...keep, ...shown];81+}82+83+// Run directly (realpath: /tmp is a symlink on macOS, and import.meta.url is resolved).84+if (process.argv[1] && fs.realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) {85+ const www = process.argv[2];86+ if (!www) {87+ process.stderr.write('usage: node lib/mirror.ts <www>\n');88+ process.exit(2);89+ }90+ process.stdout.write(mirrorFiles(www).map((f) => f + '\n').join(''));91+}@@ -89,6 +89,7 @@ FORGE_MIRROR_SECRET=$(q "${FORGE_MIRROR_SECRET:-}")
8989 FORGE_MIRROR_PROVIDER=$(q "${FORGE_MIRROR_PROVIDER:-}")9090 FORGE_MIRROR_REGION=$(q "${FORGE_MIRROR_REGION:-}")9191 FORGE_MIRROR_DELAY=$(q "${FORGE_MIRROR_DELAY:-}")92+FORGE_MIRROR_MIN_INTERVAL=$(q "${FORGE_MIRROR_MIN_INTERVAL:-}")9293 FORGE_LAKE_DATA=$(q "${FORGE_LAKE_DATA:-}")9394 FORGE_LAKE_KEY_ID=$(q "${FORGE_LAKE_KEY_ID:-}")9495 FORGE_LAKE_SECRET=$(q "${FORGE_LAKE_SECRET:-}")@@ -19,6 +19,8 @@
1919 # aside dami/secret ... back dami/secret move a repo out of the forge and back2020 # restore_into dir --from stage forge restore into a new forge at dir2121 # sync_builds hooks and ssh commands build pages before returning22+# fake_rclone a stand-in rclone with its bucket in $T/bucket, for forge mirror23+# synced [n] wait until forge mirror has finished n syncs (1)2224 # lq "<sql>" forge lake query: the rows as CSV, no header2325 # private_server lib/private-server.ts for this forge, at $PRIV2426 # signin ana ana adds a passkey and signs in at $PRIV: the session cookie@@ -247,6 +249,65 @@ restore_into() {
247249 env FORGE_ROOT="$rd" FORGE_WWW="$rd/.forge/www" FORGE_CACHE="$rd/.forge/cache" "$FORGE" restore "$@"248250 }249251 252+# fake_rclone: a stand-in for rclone on PATH, doing what forge mirror asks253+# of the real one, with the bucket a directory, $T/bucket. Each call is a254+# line in $T/rclone.log; each file it uploads (one that differs from the255+# bucket's) a line in $T/rclone.put, and each it deletes one in256+# $T/rclone.deleted. --dry-run only logs. FORGE_MIRROR_S3 points at it.257+fake_rclone() {258+ mkdir -p "$T/bin" "$T/bucket"259+ : >"$T/rclone.log"260+ : >"$T/rclone.put"261+ : >"$T/rclone.deleted"262+ {263+ printf '#!/bin/sh\nB=%s/bucket L=%s/rclone\n' "$T" "$T"264+ cat <<'EOF'265+printf '%s\n' "$*" >>"$L.log"266+op=$1 src=$2 list= dry= filters=267+shift 2268+case $1 in :s3:*) shift ;; esac269+while [ $# -gt 0 ]; do270+ case $1 in271+ --files-from-raw) list=$2; shift ;;272+ --dry-run) dry=1 ;;273+ --filter) filters=1; shift ;;274+ esac275+ shift276+done277+put() {278+ cmp -s "$src/$1" "$B/$1" && return 0279+ mkdir -p "$(dirname "$B/$1")" && cp "$src/$1" "$B/$1" && printf '%s\n' "$1" >>"$L.put"280+}281+case $op in282+lsf) cd "$B" && find . -type f | sed 's|^\./||' ;;283+delete)284+ [ -n "$dry" ] && exit 0285+ while IFS= read -r f; do rm -f "$B/$f" && printf '%s\n' "$f" >>"$L.deleted"; done <"$list"286+ # A bucket has no directories.287+ find "$B" -mindepth 1 -type d -empty -delete ;;288+sync)289+ [ -n "$dry" ] && exit 0290+ if [ -n "$list" ]; then291+ while IFS= read -r f; do put "$f" || exit 1; done <"$list"292+ elif [ -n "$filters" ]; then293+ # The release downloads' pass.294+ (cd "$src" && find . -path './*/*/releases/download/*' -type f | sed 's|^\./||') | while IFS= read -r f; do put "$f" || exit 1; done295+ fi ;;296+*) echo "fake rclone: no $op" >&2; exit 1 ;;297+esac298+EOF299+ } >"$T/bin/rclone"300+ chmod +x "$T/bin/rclone"301+ export PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages302+}303+304+# synced [n]: wait (30s at most) until forge mirror has logged n syncs.305+synced() {306+ sn=0307+ until [ "$(grep -c 'mirror: synced' "$FORGE_ROOT/.forge/mirror.log" 2>/dev/null)" -ge "${1:-1}" ] || [ "$sn" -ge 300 ]; do sleep 0.1; sn=$((sn + 1)); done308+ [ "$sn" -lt 300 ] || fail "no sync ${1:-1} in mirror.log"309+}310+250311 # From here on, what hooks and ssh commands rebuild is built before they return.251312 sync_builds() { export FORGE_NO_BUILD=0 FORGE_SYNC_BUILD=1; }252313 @@ -0,0 +1,39 @@
1+# While the forge is down, the edge serves its pages from the offline copy2+# (forge mirror's bucket), and a page the copy leaves out, such as a file's,3+# gets the forge's 404 saying so. With caddy installed: the edge's own4+# Caddyfile on free ports, the forge's port dead, the bucket a directory5+# behind a file server.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo dami/greet9+clone dami dami/greet "$T/w/d"10+at "$T/w/d" commit "first" greet.txt11+ok push_as dami "$T/w/d" origin main12+build13+fake_rclone14+ok forge mirror15+exists "$FORGE_WWW/dami/greet/blob/main/greet.txt/index.html"16+command -v caddy >/dev/null 2>&1 || { note "caddy is not installed: skipped running it"; exit 0; }17+18+mport=$(free_port)19+printf '{\n\tadmin off\n\tauto_https off\n\tpersist_config off\n}\n\n:%s {\n\troot * %s\n\tfile_server\n}\n' "$mport" "$T/bucket" >"$T/Caddyfile.bucket"20+eport=$(free_port)21+dead=$(free_port)22+sed "s/^:8080 {/:$eport {/; s/127\.0\.0\.1:8081/127.0.0.1:$dead/" "$T_HOME/deploy/edge/Caddyfile" >"$T/Caddyfile.edge"23+for c in bucket edge; do24+ FORGE_DOMAIN=src.test FORGE_MIRROR_URL=http://127.0.0.1:$mport XDG_DATA_HOME=$T/xdg XDG_CONFIG_HOME=$T/xdg \25+ caddy run --config "$T/Caddyfile.$c" --adapter caddyfile >"$T/caddy.$c.log" 2>&1 &26+ T_PIDS="$T_PIDS $!"27+done28+wait_http "http://127.0.0.1:$mport/"29+wait_http "http://127.0.0.1:$eport/"30+31+http -i -H 'Host: src.test' "http://127.0.0.1:$eport/dami/greet/" >"$T/home"32+has "$T/home" '^HTTP/1.1 200 OK'33+has "$T/home" '^X-Forge-Offline: mirror'34+has "$T/home" 'dami/greet'35+http -i -H 'Host: src.test' "http://127.0.0.1:$eport/dami/greet/blob/main/greet.txt/" >"$T/file"36+has "$T/file" '^HTTP/1.1 404 Not Found'37+has "$T/file" '^Content-Type: text/html; charset=utf-8'38+has "$T/file" '^X-Forge-Offline: mirror'39+has "$T/file" "isn't in its offline copy"@@ -42,11 +42,9 @@ for a in "" "$cat_tok"; do
4242 cmp -s "$T/missing" "$T/got" || fail "not the missing page's 404: ${a:-anonymous}"4343 done4444 45-# The mirror (a stand-in rclone that lists what it was given) sees none of it.46-mkdir "$T/bin"47-# Each run's listing whole, appended at once: the scenario waits for the file.48-printf '#!/bin/sh\nfind "$2" >"%s/rclone.$$" && cat "%s/rclone.$$" >>"%s/rclone.saw" && rm "%s/rclone.$$"\n' "$T" "$T" "$T" "$T" >"$T/bin/rclone"49-chmod +x "$T/bin/rclone"50-PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages ok forge mirror51-has "$T/rclone.saw" '/dami/open/insights/insights\.json$'52-lacks "$T/rclone.saw" 'secret'45+# The mirror (a stand-in rclone, its bucket a directory) gets none of it.46+fake_rclone47+ok forge mirror48+find "$T/bucket" >"$T/bucket.list"49+has "$T/bucket.list" '/dami/open/insights/insights\.json$'50+lacks "$T/bucket.list" 'secret'@@ -0,0 +1,15 @@
1+# forge mirror by hand syncs now, whatever the delay and the interval.2+. "$(dirname "$0")/../lib.sh"3+user dami4+repo dami/greet5+clone dami dami/greet "$T/w/d"6+at "$T/w/d" commit "first"7+ok push_as dami "$T/w/d" origin main8+build9+fake_rclone10+export FORGE_MIRROR_DELAY=600 FORGE_MIRROR_MIN_INTERVAL=360011+date +%s >"$FORGE_ROOT/.forge/mirror.last"12+build13+ok forge mirror14+has "$T_LOG" 'mirror: synced'15+exists "$T/bucket/dami/greet/index.html"@@ -0,0 +1,18 @@
1+# Builds in a row make one sync of the offline copy: each waits2+# FORGE_MIRROR_DELAY seconds after it, and a newer build restarts the wait.3+. "$(dirname "$0")/../lib.sh"4+user dami5+repo dami/greet6+clone dami dami/greet "$T/w/d"7+at "$T/w/d" commit "first"8+ok push_as dami "$T/w/d" origin main9+build10+fake_rclone11+export FORGE_MIRROR_DELAY=8 FORGE_MIRROR_MIN_INTERVAL=012+build13+build14+build15+missing "$FORGE_ROOT/.forge/mirror.last"16+synced 117+sleep 218+eq "$(grep -c '^sync .*--files-from-raw' "$T/rclone.log")" 1@@ -0,0 +1,38 @@
1+# The offline copy leaves out the pages there is one of per file: file2+# pages, directory pages below a branch's top level, and images no page in3+# the copy shows. They aren't uploaded, and those a sync from before put in4+# the bucket are deleted by the next one, once. Release downloads in the5+# bucket are their own pass's, and stay.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo dami/greet9+clone dami dami/greet "$T/w/d"10+mkdir -p "$T/w/d/lib" "$T/w/d/img"11+printf 'PNG1' >"$T/w/d/img/pic.png"12+printf 'PNG2' >"$T/w/d/img/other.png"13+printf '# greet\n\n\n' >"$T/w/d/README.md"14+printf 'x\n' >"$T/w/d/lib/x.txt"15+at "$T/w/d" git add -A16+at "$T/w/d" git commit -qm first17+ok push_as dami "$T/w/d" origin main18+build19+fake_rclone20+# What an older sync left there.21+g=$T/bucket/dami/greet22+mkdir -p "$g/blob/main/lib/x.txt" "$g/tree/main/lib" "$g/raw/main/img" "$g/releases/download/v1"23+echo old >"$g/blob/main/lib/x.txt/index.html"24+echo old >"$g/tree/main/lib/index.html"25+echo old >"$g/raw/main/img/other.png"26+echo old >"$g/releases/download/v1/greet"27+ok forge mirror28+lacks "$T/rclone.put" '/blob/|/tree/main/(lib|img)/|other\.png'29+missing "$g/blob"30+missing "$g/tree/main/lib"31+missing "$g/raw/main/img/other.png"32+eq "$(LC_ALL=C sort "$T/rclone.deleted" | tr '\n' ' ')" "dami/greet/blob/main/lib/x.txt/index.html dami/greet/raw/main/img/other.png dami/greet/tree/main/lib/index.html "33+exists "$g/releases/download/v1/greet"34+# The next sync has nothing to delete.35+: >"$T/rclone.log"36+ok forge mirror37+lacks "$T/rclone.log" '^delete '38+eq "$(wc -l <"$T/rclone.deleted" | tr -d ' ')" 3@@ -0,0 +1,32 @@
1+# The offline copy has what keeps the forge readable while it is down: the2+# home page, owner pages, topics, insights, the design page, assets and3+# fonts, and each repo's home with its README's images, its branch's file4+# list, log, commits, reviews, releases with their downloads, refs, sites,5+# feed and search indexes.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo dami/greet9+clone dami dami/greet "$T/w/d"10+mkdir -p "$T/w/d/img"11+printf 'PNG1' >"$T/w/d/img/pic.png"12+printf '# greet\n\n\n' >"$T/w/d/README.md"13+at "$T/w/d" git add -A14+at "$T/w/d" git commit -qm first15+at "$T/w/d" git tag v116+ok push_as dami "$T/w/d" origin main v117+mkdir "$T/f"18+printf 'greet\n' >"$T/f/greet"19+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/greet"20+build21+fake_rclone22+ok forge mirror23+sha=$(git -C "$T/w/d" rev-parse HEAD)24+for f in index.html repos.json dami/index.html topics/index.html insights/index.html design/index.html \25+ dami/greet/index.html dami/greet/raw/main/img/pic.png dami/greet/tree/main/index.html dami/greet/files/main.txt \26+ dami/greet/log/main/index.html "dami/greet/commit/$sha/index.html" dami/greet/reviews/index.html \27+ dami/greet/releases/index.html dami/greet/releases/download/v1/greet dami/greet/refs/index.html \28+ dami/greet/sites/index.html dami/greet/atom.xml dami/greet/search.json dami/greet/code.db; do29+ exists "$T/bucket/$f"30+done31+ls "$T/bucket/assets/" | grep -q '^style\..*\.css$' || fail "no stylesheet in the bucket"32+ls "$T/bucket/assets/" | grep -q '\.woff2$' || fail "no fonts in the bucket"@@ -0,0 +1,24 @@
1+# Syncs of the offline copy start FORGE_MIRROR_MIN_INTERVAL seconds apart at2+# least. Builds within it make one sync when it ends, not one each.3+. "$(dirname "$0")/../lib.sh"4+user dami5+repo dami/greet6+clone dami dami/greet "$T/w/d"7+at "$T/w/d" commit "first"8+ok push_as dami "$T/w/d" origin main9+build10+fake_rclone11+export FORGE_MIRROR_DELAY=1 FORGE_MIRROR_MIN_INTERVAL=1512+build13+synced 114+first=$(cat "$FORGE_ROOT/.forge/mirror.last")15+build16+build17+sleep 218+[ $(($(date +%s) - first)) -lt 15 ] || fail "the builds took too long to land within the interval"19+eq "$(grep -c 'mirror: synced' "$FORGE_ROOT/.forge/mirror.log")" 120+synced 221+second=$(cat "$FORGE_ROOT/.forge/mirror.last")22+[ $((second - first)) -ge 15 ] && [ $((second - first)) -le 18 ] || fail "the second sync started $((second - first))s after the first"23+sleep 324+eq "$(grep -c 'mirror: synced' "$FORGE_ROOT/.forge/mirror.log")" 2@@ -29,12 +29,14 @@ exists "$w/sites/dami/greet/docs/index.html"
2929 exists "$w/previews/dami/greet/docs/dami~topic/index.html"3030 eq "$(ls "$w/hosts" | sort | tr '\n' ' ')" "greet-docs-dami greet-docs-dami--dami-topic "3131 32-# The mirror's sync (a stand-in rclone) runs after the flip, and sees what is left.33-mkdir "$T/bin"34-# Written whole, then renamed: the scenario waits for the file to appear.35-printf '#!/bin/sh\nls "$2/dami" >"%s/rclone.tmp" && mv "%s/rclone.tmp" "%s/rclone.saw"\n' "$T" "$T" "$T" >"$T/bin/rclone"36-chmod +x "$T/bin/rclone"37-export PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages FORGE_MIRROR_DELAY=032+# The mirror (a stand-in rclone, its bucket a directory) syncs right after33+# the flip, though a sync has just run and the interval would hold a34+# build's back, and the repo's pages leave the bucket.35+fake_rclone36+mkdir -p "$T/bucket/dami/greet"37+echo old >"$T/bucket/dami/greet/index.html"38+date +%s >"$FORGE_ROOT/.forge/mirror.last"39+export FORGE_MIRROR_DELAY=0 FORGE_MIRROR_MIN_INTERVAL=36003840 sync_builds3941 ok ssh_as dami private dami/greet4042 has "$T_LOG" "can't be recalled"@@ -49,10 +51,9 @@ missing "$w/previews/dami/greet/docs/dami~topic"
4951 eq "$(ls "$w/hosts")" ""5052 exists "$p/dami/greet/index.html"5153 exists "$p/dami/greet/code.db"52-n=053-until [ -f "$T/rclone.saw" ] || [ "$n" -ge 50 ]; do sleep 0.2; n=$((n + 1)); done54-has "$T/rclone.saw" '^other$'55-lacks "$T/rclone.saw" 'greet'54+synced55+exists "$T/bucket/dami/other/index.html"56+missing "$T/bucket/dami/greet"5657 unset FORGE_MIRROR_S35758 5859 # Pushes to it now: no pages, no sites.@@ -1,7 +1,8 @@
1-# forge mirror syncs the pages by checksum without the release downloads,2-# then the downloads alone by size and mtime, so unchanged binaries aren't3-# read; each pass leaves the other's files in the bucket alone. rclone here4-# is a stand-in that writes down how it was called.1+# forge mirror syncs the pages on its list by checksum, without the release2+# downloads, then deletes what else is in the bucket but the downloads, then3+# syncs the downloads alone by size and mtime, so unchanged binaries aren't4+# read. Uploads skip rclone's HEAD of each object after it. rclone here is5+# a stand-in that writes down how it was called.56 . "$(dirname "$0")/../lib.sh"67 user dami78 repo dami/greet@@ -13,24 +14,23 @@ mkdir "$T/f"
1314 printf 'greet\n' >"$T/f/greet"1415 FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/greet"1516 touch "$FORGE_WWW/index.html"16-17-mkdir "$T/bin"18-cat >"$T/bin/rclone" <<EOF19-#!/bin/sh20-printf '%s\n' "\$*" >>"$T/rclone.log"21-EOF22-chmod +x "$T/bin/rclone"23-export PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages17+fake_rclone18+mkdir -p "$T/bucket/dami/greet/blob"19+echo old >"$T/bucket/dami/greet/blob/index.html"2420 ok forge mirror25-eq "$(wc -l <"$T/rclone.log" | tr -d ' ')" 221+eq "$(cut -d' ' -f1 "$T/rclone.log" | tr '\n' ' ')" "sync lsf delete sync "2622 sed -n 1p "$T/rclone.log" >"$T/pass1"27-sed -n 2p "$T/rclone.log" >"$T/pass2"28-has "$T/pass1" "^sync $FORGE_WWW :s3:pages .*--checksum"29-has "$T/pass1" "--exclude /sites/\*\* --exclude /previews/\*\* --exclude /hosts/\*\* --exclude \*\.old/\*\* --exclude /\*/\*/releases/download/\*\*$"30-has "$T/pass2" "^sync $FORGE_WWW :s3:pages "23+sed -n 4p "$T/rclone.log" >"$T/pass2"24+has "$T/pass1" "^sync $FORGE_WWW :s3:pages --config --files-from-raw $FORGE_ROOT/.forge/mirror.keep --checksum --fast-list --s3-no-head "25+has "$T/pass2" "^sync $FORGE_WWW :s3:pages --config --fast-list --s3-no-head "3126 lacks "$T/pass2" 'checksum'3227 has "$T/pass2" "--filter - \*\.old/\*\* --filter - \*\.tmp\.\*/\*\* --filter \+ /\*/\*/releases/download/\*\* --filter - \*\*$"33-# --dry-run goes to both.28+lacks "$FORGE_ROOT/.forge/mirror.keep" 'releases/download'29+has "$T/rclone.log" "^delete :s3:pages --config --files-from-raw $FORGE_ROOT/.forge/mirror.keep.gone "30+# --dry-run goes to both syncs and the deletion.31+mkdir -p "$T/bucket/dami/greet/blob"32+echo old >"$T/bucket/dami/greet/blob/index.html"3433 : >"$T/rclone.log"3534 ok forge mirror --dry-run36-eq "$(grep -c -- '--dry-run' "$T/rclone.log")" 235+eq "$(grep -c -- '--dry-run' "$T/rclone.log")" 336+exists "$T/bucket/dami/greet/blob/index.html"@@ -73,11 +73,9 @@ same -u "cat:$cat_tok"
7373 same -H "Cookie: __Host-forge=$cat"7474 same -H "Authorization: Bearer $elsewhere"7575 76-# The mirror (a stand-in rclone that lists what it was given) sees none of it.77-mkdir "$T/bin"78-# Each run's listing whole, appended at once: the scenario waits for the file.79-printf '#!/bin/sh\nfind "$2" >"%s/rclone.$$" && cat "%s/rclone.$$" >>"%s/rclone.saw" && rm "%s/rclone.$$"\n' "$T" "$T" "$T" "$T" >"$T/bin/rclone"80-chmod +x "$T/bin/rclone"81-PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages ok forge mirror82-has "$T/rclone.saw" '/index\.html$'83-lacks "$T/rclone.saw" 'secret|tool'76+# The mirror (a stand-in rclone, its bucket a directory) gets none of it.77+fake_rclone78+ok forge mirror79+find "$T/bucket" >"$T/bucket.list"80+has "$T/bucket.list" '/index\.html$'81+lacks "$T/bucket.list" 'secret|tool'