| M | .forge/ | 6 | +- |
| M | README.md | 56 | +++++++++++----- |
| M | bin/ | 105 | +++++++++++++++++++++--------- |
| A | test/ | 46 | +++++++++++++ |
| M | test/ | 13 | ++-- |
| A | test/ | 40 | +++++++++++ |
| A | test/ | 47 | +++++++++++++ |
| A | test/ | 68 | +++++++++++++++++++ |
| A | test/ | 30 | +++++++++ |
@@ -80,10 +80,12 @@ test -f "$w/sites/dami/forge/guide/index.html"
8080 # Backups: stage the demo forge the way forge backup does, restore that into8181 # an empty root, and check the forge comes back whole: hooks, keys, refs,8282 # review logs and comments, sites, pages. The restic round trip needs83-# restic, which the container image has.83+# restic, which the container image has. The restore's build would start a84+# forge lake in the background 10 seconds on, writing into $r while it is85+# removed at the end; FORGE_LAKE=0 starts none.8486 r=$(mktemp -d "${TMPDIR:-/tmp}/forge-ci-restore.XXXXXX")8587 FORGE_ROOT=$PWD/demo/root sh bin/forge backup --dry-run 2>/dev/null86-FORGE_ROOT=$r sh bin/forge restore --from demo/root/.forge/backup/stage >/dev/null 2>&188+FORGE_LAKE=0 FORGE_ROOT=$r sh bin/forge restore --from demo/root/.forge/backup/stage >/dev/null 2>&18789 test -x "$r/dami/greet.git/hooks/pre-receive"8890 grep -q 'shell ana"' "$r/.ssh/authorized_keys"8991 diff -r demo/root/dami/greet.git/forge "$r/dami/greet.git/forge"@@ -193,7 +193,8 @@ ssh git@git.example.com ls # yours, and the private o
193193 The owner and collaborators flip a repo, the way Forgejo does. `private`194194 removes the flag, the repo's public pages, its entries on the index, owner195195 pages and `repos.json`, and its served sites, previews and their196-hostnames, moves its release downloads into the private root, and builds196+hostnames, moves its release downloads into the private root (and their197+files into the private store, see [Releases](#releases)), and builds197198 its pages again there. Then it starts198199 a mirror sync, which deletes them from the bucket. Clones, caches and copies199200 people already have can't be recalled. `public` first runs the secret check@@ -210,7 +211,10 @@ any left in the wrong place.
210211 211212 A private repo's pages are built like any repo's, into212213 `.forge/private/www` instead of `www`, with the same layout: pages, search213-index, `code.db`, release downloads, insights exports. They link the same hashed assets, which the build puts in214+index, `code.db`, release downloads, insights exports. The build writes215+them in `.forge/private/build.<pid>` and swaps them in from there, so216+they are never outside `.forge/private`, which only git can read, even217+mid-build. They link the same hashed assets, which the build puts in214218 `www/assets` (they are the forge's scripts and styles, nothing of the215219 repo), and their header marks the repo private. Caddy serves a file of216220 `www` when there is one and hands every other path to@@ -484,7 +488,8 @@ The secret check skips releases. It reads branches and tags, and a
484488 release's source is its tag.485489 486490 `post-receive` extracts the tree once into `.forge/store/<tree>/`, as for a487-site, and hard-links its files into491+site (a private repo's into a store of its own, below), and hard-links its492+files into488493 `/<owner>/<repo>/releases/download/<tag>/` (a `/` in the tag becomes `~`).489494 `latest/` next to them holds the files of the newest tag that has a490495 release, so `/dami/greet/releases/download/latest/greet-linux-amd64` keeps@@ -508,10 +513,22 @@ doesn't see them and `forge.publicsites` doesn't change that. The private
508513 server gives them to its readers, signed in or showing a token (see509514 [Private pages](#private-pages)) as Caddy gives a public repo's510515 (attachments, uncompressed, byte ranges) but `Cache-Control: private,511-no-cache`, and anyone else gets the 404 of a file that isn't there. A repo512-that goes private takes its downloads there at once, and one that goes513-public brings them back to `www`. Readers can also get them over ssh, and to514-anyone else these commands answer as for a missing repo:516+no-cache`, and anyone else gets the 404 of a file that isn't there.517+518+Their files aren't extracted into `.forge/store` either, but into519+`.forge/private/store/<tree>/`, and linked from there. `.forge/store` is520+readable by the caddy user, which serves sites from it; nothing serves a521+release tree from a store, but a private repo's files shouldn't sit where522+anything but the forge can read them, so they live under `.forge/private`,523+which only git can read (mode 700). Both stores are under `.forge`, so the524+downloads stay hard links on one filesystem. A repo that goes private takes525+its downloads to the private root at once, linked from the private store526+(extracted there), and one that goes public brings them back to `www`,527+linked from `.forge/store` (extracted there again); `forge gc` frees the528+trees left in the store it no longer uses.529+530+Readers can also get them over ssh, and to anyone else these commands531+answer as for a missing repo:515532 516533 ```sh517534 curl -fO -u ana:<token> https://git.example.com/dami/notes/releases/download/v1.0/tool # token --repo dami/notes@@ -800,14 +817,17 @@ push (log: `.forge/build.log`); `forge build` rebuilds everything.
800817 801818 ### Cleaning up802819 803-Every distinct deploy is extracted once into `.forge/store/<tree>/`, and the804-old tree stays when a newer deploy takes its link. `forge gc` removes store805-trees that no `www/sites` or `www/previews` link points at (hostnames point806-at those links) and no `refs/releases/*` ref names, a private repo's too807-(release files are hard links, which name no tree), hostnames whose deploy808-is gone, previews of review branches that no longer exist, and the809-downloads of releases whose ref was deleted without the hooks, in `www` or810-the private root. A deploy extracts its tree before it links it, and820+Every distinct deploy is extracted once into `.forge/store/<tree>/` (a821+private repo's releases into `.forge/private/store/<tree>/`), and the old822+tree stays when a newer deploy takes its link. `forge gc` removes trees in823+`.forge/store` that no `www/sites` or `www/previews` link points at824+(hostnames point at those links) and no public repo's `refs/releases/*` ref825+names, and trees in `.forge/private/store` that no private repo's release826+ref names (release files are hard links, which name no tree), so a repo827+that flipped leaves nothing in the store it left. It also removes hostnames828+whose deploy is gone, previews of review branches that no longer exist,829+and the downloads of releases whose ref was deleted without the hooks, in830+`www` or the private root. A deploy extracts its tree before it links it, and811831 touches a tree it reuses, so store entries changed in the last 10 minutes812832 are kept (`--age <minutes>`), which makes gc safe to run while people push. `--dry-run` lists813833 what it would remove. A release published again leaves its old files in the@@ -919,7 +939,8 @@ Everything in `/srv/git` that can't be rebuilt:
919939 920940 Left out, because a restore rebuilds it or every boot rewrites it: the pages in921941 `.forge/www` and `.forge/private`, the extracted sites and releases in922-`.forge/store` (both are git refs),942+`.forge/store` and private repos' releases in `.forge/private/store` (all943+are git refs),923944 `.forge/cache`, `authorized_keys`, `allowed_signers`, `git-shell-commands/`,924945 hooks, `.forge/secrets` and `.forge/backup.env` (both written from `forge.env`925946 at boot), and logs, locks and scratch files.@@ -1057,7 +1078,8 @@ What you set up yourself, once:
10571078 `/srv/git`. It restores the files, rebuilds what a backup leaves out (what10581079 `forge server init` and `forge keys sync` write, hooks in every repo, pinned10591080 review revisions, extracted sites, release downloads, a private repo's in1060-the private root, with the mtimes they had, so the mirror uploads none1081+the private root, extracted into `.forge/private/store` again, with the1082+mtimes they had, so the mirror uploads none10611083 again) and runs `forge10621084 build`. identity.db is the backup's copy or litestream's replica, whichever10631085 changed last (see [above](#identitydb-backed-up-as-it-changes)). It refuses@@ -962,6 +962,8 @@ load_config() {
962962 # Private repos' pages: the same layout as www, served only to their963963 # readers (lib/private-server.ts).964964 FORGE_PRIVATE_WWW=${FORGE_PRIVATE_WWW:-$FORGE_STATE/private/www}965+ # Their release trees, beside it (release_store).966+ FORGE_PRIVATE_STORE=$(dirname "$FORGE_PRIVATE_WWW")/store965967 FORGE_AUTHORIZED_KEYS=${FORGE_AUTHORIZED_KEYS:-$FORGE_ROOT/.ssh/authorized_keys}966968 export FORGE_ROOT FORGE_STATE FORGE_WWW FORGE_PRIVATE_WWW967969 }@@ -1738,7 +1740,8 @@ sc_private() {
17381740 rm -f "$RDIR/git-daemon-export-ok"17391741 rm -rf "${FORGE_WWW:?}/$1"17401742 # Its release downloads go with its pages, and are in the private root1741- # for its readers now rather than after the build.1743+ # for its readers now rather than after the build, linked from the1744+ # private store (release_store); gc frees the trees it left in .forge/store.17421745 (17431746 export GIT_DIR="$RDIR"17441747 OWNER=${1%%/*} NAME=${1#*/}@@ -1785,6 +1788,8 @@ sc_public() {
17851788 esac17861789 touch "$RDIR/git-daemon-export-ok"17871790 rm -rf "${FORGE_PRIVATE_WWW:?}/$1"1791+ # Release downloads are linked from .forge/store again (release_store);1792+ # gc frees the repo's trees in the private store.17881793 (17891794 export GIT_DIR="$RDIR"17901795 OWNER=${1%%/*} NAME=${1#*/}@@ -2218,14 +2223,15 @@ push_site() {
22182223 fi22192224 }22202225 2221-# store_tree <commit>: extract its tree once into the content store,2222-# .forge/store/<tree>/, and print where. Sites link to it (deploy_site),2223-# releases hard-link its files (release_link). git archive stamps every file2226+# store_tree <commit> [<store>]: extract its tree once into the content2227+# store, .forge/store/<tree>/, or into <store>/<tree>/, and print where.2228+# Sites link to it (deploy_site), releases hard-link its files from the2229+# store release_store names (release_link). git archive stamps every file22242230 # with the commit's time, so a tree extracted again (after a restore) gets22252231 # the same mtimes. A reused tree counts as new, so a concurrent forge gc,22262232 # which spares what changed in its last --age minutes, leaves it alone.22272233 store_tree() {2228- st_dir=$FORGE_STATE/store/$(git rev-parse "$1^{tree}")2234+ st_dir=${2:-$FORGE_STATE/store}/$(git rev-parse "$1^{tree}")22292235 if [ ! -d "$st_dir" ]; then22302236 rm -rf "$st_dir.tmp.$$"22312237 mkdir -p "$st_dir.tmp.$$"@@ -2238,17 +2244,18 @@ store_tree() {
22382244 22392245 # release_link <root> <tag> <commit|ZERO>: make22402246 # <root>/<owner>/<repo>/releases/download/<tag>/ hard links to the files of2241-# a release, extracted once into the store, or remove it. Hard links, not a2242-# symlink like a site's: rclone (forge mirror) skips symlinks, and the files2243-# keep the store's mtime. Runs in a repo (GIT_DIR) with OWNER and NAME set;2244-# <root> is the repo's pages root (pages_root) or a build's output.2247+# a release, extracted once into the repo's store (release_store), or remove2248+# it. Hard links, not a symlink like a site's: rclone (forge mirror) skips2249+# symlinks, and the files keep the store's mtime. Runs in a repo (GIT_DIR)2250+# with OWNER and NAME set; <root> is the repo's pages root (pages_root) or a2251+# build's output.22452252 release_link() {22462253 rl=$1/$OWNER/$NAME/releases/download/$(enc_ref "$2")22472254 if [ "$3" = "$ZERO" ]; then22482255 rm -rf "$rl"22492256 return 022502257 fi2251- rs=$(store_tree "$3")2258+ rs=$(store_tree "$3" "$(release_store)")22522259 rm -rf "$rl.tmp.$$"22532260 link_files "$rl.tmp.$$" "$rs"/*22542261 swap "$rl.tmp.$$" "$rl"@@ -2292,6 +2299,24 @@ private_www() {
22922299 chmod 700 "$(dirname "$FORGE_PRIVATE_WWW")"22932300 }22942301 2302+# release_store: where the repo's release trees are extracted, to be2303+# hard-linked into its pages. A public repo's go in the content store, which2304+# the caddy user can read, since sites are served from it. A private repo's2305+# go in FORGE_PRIVATE_STORE, beside the private root under the directory2306+# only the forge can read: nothing serves a store, but those files are a2307+# private repo's all the same. Both are under .forge, so the links stay on2308+# one filesystem. Like pages_root it follows the repo's flag, so a flip2309+# links the downloads from the store of the side the repo goes to,2310+# extracting them there. Runs in a repo (GIT_DIR).2311+release_store() {2312+ if [ -f "$GIT_DIR/git-daemon-export-ok" ]; then2313+ printf '%s' "$FORGE_STATE/store"2314+ else2315+ private_www2316+ printf '%s' "$FORGE_PRIVATE_STORE"2317+ fi2318+}2319+22952320 # pages_root <git-dir>: where the repo's pages and release downloads live:22962321 # www for a public repo, the private root for a private one.22972322 pages_root() {@@ -2447,7 +2472,11 @@ build_all() {
24472472 # hashed assets, so the build's assets go into www/assets as any build's24482473 # do; they are templates' CSS and the forge's scripts, nothing of the repo.24492474 # The rest of what the build writes (index and owner pages) is dropped.2450-# Runs in a subshell of cmd_build, for its EXIT trap.2475+# It builds in a scratch directory beside the private root, under the2476+# directory only the forge can read, so the pages and release links are2477+# never outside it, even mid-build, and the swaps stay renames (the assets'2478+# too: www is under .forge as well). Runs in a subshell of cmd_build, for2479+# its EXIT trap.24512480 build_private() {24522481 only=$124532482 lock=$FORGE_STATE/build.lock@@ -2457,7 +2486,7 @@ build_private() {
24572486 [ "$n" -lt 600 ] || die "build lock held: $lock"24582487 sleep 124592488 done2460- out=$FORGE_STATE/build.$$2489+ out=$(dirname "$FORGE_PRIVATE_WWW")/build.$$24612490 trap 'rm -rf "$out"; rmdir "$lock" 2>/dev/null' EXIT24622491 rm -rf "$out"24632492 t0=$(date +%s)@@ -2613,7 +2642,8 @@ was_review() { grep -qxF "branch $2" "$1"/forge/reviews/* 2>/dev/null; }
26132642 # branches that are gone (deleted outside the hook, or before it removed26142643 # previews), hostnames whose deploy is gone, downloads of deleted releases,26152644 # and store trees that no site or preview links to (hostnames point at those2616-# links, never at the store) and no release ref names.2645+# links, never at the store) and no release ref names, in the content store2646+# and the private store (release_store), each keeping its own side's.26172647 # A deploy extracts its tree before linking it, so entries changed in the26182648 # last --age minutes stay. Then repacks every repo (repack_repo).26192649 cmd_gc() {@@ -2677,27 +2707,38 @@ cmd_gc() {
26772707 # Candidates before links: a deploy that links a tree after this looked26782708 # has touched it first, so it can't be a candidate.26792709 n=0 kb=02680- if [ -d "$FORGE_STATE/store" ]; then2681- set --2682- [ "$age" = 0 ] || set -- -mmin +"$age"2683- old=$(find "$FORGE_STATE/store" -mindepth 1 -maxdepth 1 "$@")2684- used=$(for l in "$FORGE_WWW"/sites/*/*/* "$FORGE_WWW"/previews/*/*/*/*; do if [ -L "$l" ]; then basename "$(readlink "$l")"; fi; done2685- # Release files are hard links, which don't name their tree: the refs do.2686- for d in $repos; do git --git-dir="$FORGE_ROOT/$d" for-each-ref --format='%(tree)' refs/releases/; done)2710+ set --2711+ [ "$age" = 0 ] || set -- -mmin +"$age"2712+ for st in "$FORGE_STATE/store" "$FORGE_PRIVATE_STORE"; do2713+ [ -d "$st" ] || continue2714+ pub=1 what=store2715+ [ "$st" = "$FORGE_STATE/store" ] || pub=0 what='private store'2716+ old=$(find "$st" -mindepth 1 -maxdepth 1 "$@")2717+ used=$(if [ "$pub" = 1 ]; then for l in "$FORGE_WWW"/sites/*/*/* "$FORGE_WWW"/previews/*/*/*/*; do if [ -L "$l" ]; then basename "$(readlink "$l")"; fi; done; fi2718+ # Release files are hard links, which don't name their tree: the2719+ # refs do, of the repos on this store's side. A repo that flipped2720+ # links from the other store now (release_store), so its trees2721+ # here go.2722+ for d in $repos; do2723+ p=02724+ [ ! -f "$FORGE_ROOT/$d/git-daemon-export-ok" ] || p=12725+ [ "$p" = "$pub" ] || continue2726+ git --git-dir="$FORGE_ROOT/$d" for-each-ref --format='%(tree)' refs/releases/2727+ done)26872728 for s in $old; do26882729 printf '%s\n' "$used" | grep -qxF "${s##*/}" && continue26892730 k=$(du -sk "$s" | cut -f1)26902731 [ "$dry" = 1 ] || rm -rf "$s"2691- echo "$did store ${s##*/} (${k}K)"2732+ echo "$did $what ${s##*/} (${k}K)"26922733 n=$((n + 1)) kb=$((kb + k))26932734 done2694- fi2735+ done26952736 if [ "$dry" = 1 ]; then2696- echo "gc: would remove $n from the store (${kb}K); repacking skipped"2737+ echo "gc: would remove $n from the stores (${kb}K); repacking skipped"26972738 return26982739 fi26992740 for d in $repos; do repack_repo "$FORGE_ROOT/$d" || say "forge: repacking $d failed"; done2700- echo "gc: removed $n from the store (${kb}K)"2741+ echo "gc: removed $n from the stores (${kb}K)"27012742 }27022743 27032744 # Geometric repacking merges packs only until each is at least twice the@@ -2863,10 +2904,11 @@ stage_repo() {
28632904 }28642905 28652906 # .forge entries a backup leaves out: rebuilt (www, store, cache,2866-# allowed_signers, private pages), regenerated at boot or by commands2867-# (secrets, litestream.yml), or logs, locks and scratch. The lake is staged2868-# by itself (stage_lake), and identity.db by stage_identity; its WAL and2869-# litestream's state are never copied.2907+# allowed_signers, private: private pages and the private store),2908+# regenerated at boot or by commands (secrets, litestream.yml), or logs,2909+# locks and scratch. The lake is staged by itself (stage_lake), and2910+# identity.db by stage_identity; its WAL and litestream's state are never2911+# copied.28702912 derived_state() {28712913 case $1 in28722914 www | store | cache | backup | secrets | allowed_signers | config.restored | restore.tmp | lake | private | \@@ -3165,9 +3207,10 @@ restore_finish() {
31653207 awk '$2 == "commit" { print "update refs/forge/keep/" $1 " " $1 }' | git update-ref --stdin31663208 # A private repo's sites stay unserved (sites_public), as before the backup.31673209 if sites_public "$GIT_DIR"; then redeploy_sites; fi3168- # Releases are refs too: a public repo's go to www, a private repo's to3169- # the private root. The files get their commit's time again, so the3170- # mirror sees nothing new.3210+ # Releases are refs too: a public repo's go to www, linked from3211+ # .forge/store, and a private repo's to the private root, extracted3212+ # into the private store again (release_store). The files get their3213+ # commit's time again, so the mirror sees nothing new.31713214 releases_into "$(pages_root "$GIT_DIR")"31723215 git update-server-info31733216 unset GIT_DIR@@ -0,0 +1,46 @@
1+# A private repo's build writes its pages, and links its release files, in2+# a scratch directory under .forge/private, which only the forge can read,3+# never in .forge/build.*, so nothing of the repo is outside it even4+# mid-build; the swap into the private root keeps the links hard links.5+# A stand-in node first on PATH (11ty runs through env node) notes where6+# each build writes, and how its parent is kept, while it runs.7+. "$(dirname "$0")/../lib.sh"8+user dami9+repo --private dami/secret10+repo dami/open11+clone dami dami/secret "$T/w/d"12+at "$T/w/d" commit "first"13+at "$T/w/d" git tag v114+ok push_as dami "$T/w/d" origin main15+mkdir "$T/f"16+printf 'secret tool\n' >"$T/f/tool"17+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool"18+19+mkdir "$T/bin"20+cat >"$T/bin/node" <<EOF21+#!/bin/sh22+case \${1:-} in *eleventy*) printf '%s %s %s\n' "\${FORGE_PRIVATE:-0}" "\$FORGE_WWW" "\$(ls -ld "\$(dirname "\$FORGE_WWW")" | cut -c1-10)" >>"$T/builds" ;; esac23+exec "$(command -v node)" "\$@"24+EOF25+chmod +x "$T/bin/node"26+PATH=$T/bin:$PATH build dami/secret27+PATH=$T/bin:$PATH build dami/open28+29+eq "$(grep -c . "$T/builds")" 230+set -- $(sed -n 1p "$T/builds")31+eq "$1" 132+case $2 in "$FORGE_ROOT/.forge/private/build."[0-9]*) ;; *) fail "the private build wrote in $2" ;; esac33+eq "$3" drwx------34+missing "$2"35+set -- $(sed -n 2p "$T/builds")36+eq "$1" 037+case $2 in "$FORGE_ROOT/.forge/build."[0-9]*) ;; *) fail "the public build wrote in $2" ;; esac38+if find "$FORGE_ROOT/.forge" -maxdepth 2 -name 'build.[0-9]*' | grep -q .; then fail "a build left its scratch directory"; fi39+40+p=$FORGE_ROOT/.forge/private/www/dami/secret41+exists "$p/index.html"42+tree=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')43+inode() { ls -i "$1" | awk '{ print $1 }'; }44+eq "$(inode "$p/releases/download/v1/tool")" "$(inode "$FORGE_ROOT/.forge/private/store/$tree/tool")"45+missing "$FORGE_WWW/dami/secret"46+exists "$FORGE_WWW/assets"@@ -1,6 +1,7 @@
1-# forge gc keeps the store tree of a private repo's release, whose files2-# are hard links in the private root, and drops the downloads of a release3-# whose ref went behind the hooks' back there as in www, freeing its tree.1+# forge gc keeps the private store tree of a private repo's release, whose2+# files are hard links in the private root, and drops the downloads of a3+# release whose ref went behind the hooks' back there as in www, freeing4+# its tree.45 . "$(dirname "$0")/../lib.sh"56 user dami67 repo --private dami/secret@@ -19,21 +20,21 @@ for v in 1 2; do
1920 done2021 unset FORGE_USER2122 build22-store=$FORGE_ROOT/.forge/store23+store=$FORGE_ROOT/.forge/private/store2324 dl=$FORGE_ROOT/.forge/private/www/dami/secret/releases/download2425 exists "$store/$tree1/tool"2526 exists "$store/$tree2/tool"2627 has "$dl/latest/tool" 'secret 2'2728 2829 ok forge gc --age 029-lacks "$T_LOG" 'removed (release|store)'30+lacks "$T_LOG" 'removed (release|store|private store)'3031 exists "$store/$tree1/tool"3132 exists "$store/$tree2/tool"3233 3334 git --git-dir="$g" update-ref -d refs/releases/v23435 ok forge gc --age 03536 has "$T_LOG" '^removed release dami/secret v2$'36-has "$T_LOG" "^removed store $tree2 "37+has "$T_LOG" "^removed private store $tree2 "3738 lacks "$T_LOG" "$tree1"3839 missing "$dl/v2"3940 has "$dl/v1/tool" 'secret 1'@@ -0,0 +1,40 @@
1+# A backup leaves .forge/private/store out, like .forge/store, since the2+# release refs it is extracted from are in the backup; a restore extracts a3+# private repo's releases into it again and links its downloads from there,4+# and a public repo's from .forge/store.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo dami/open8+repo --private dami/secret9+mkdir "$T/f"10+printf 'tool\n' >"$T/f/tool"11+export FORGE_USER=dami12+for r in open secret; do13+ clone dami "dami/$r" "$T/w/$r"14+ at "$T/w/$r" commit "first $r"15+ at "$T/w/$r" git tag v116+ ok push_as dami "$T/w/$r" origin main17+ ok at "$T/w/$r" "$FORGE" release publish v1 "$T/f/tool"18+done19+unset FORGE_USER20+open=$(git --git-dir="$FORGE_ROOT/dami/open.git" rev-parse 'refs/releases/v1^{tree}')21+secret=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')22+exists "$FORGE_ROOT/.forge/private/store/$secret/tool"23+24+ok forge backup --dry-run25+st=$FORGE_ROOT/.forge/backup/stage26+missing "$st/.forge/private"27+missing "$st/.forge/store"28+exists "$st/dami/secret.git/HEAD"29+30+r=$T/restored31+ok env FORGE_ROOT="$r" FORGE_WWW="$r/.forge/www" FORGE_CACHE="$T/cache2" "$FORGE" restore --from "$st"32+inode() { ls -i "$1" | awk '{ print $1 }'; }33+dl=dami/secret/releases/download34+ok cmp "$T/f/tool" "$r/.forge/private/store/$secret/tool"35+eq "$(inode "$r/.forge/private/www/$dl/v1/tool")" "$(inode "$r/.forge/private/store/$secret/tool")"36+eq "$(inode "$r/.forge/private/www/$dl/latest/tool")" "$(inode "$r/.forge/private/store/$secret/tool")"37+missing "$r/.forge/store/$secret"38+eq "$(ls -ld "$r/.forge/private" | cut -c1-10)" drwx------39+eq "$(inode "$r/.forge/www/dami/open/releases/download/v1/tool")" "$(inode "$r/.forge/store/$open/tool")"40+missing "$r/.forge/private/store/$open"@@ -0,0 +1,47 @@
1+# A flip moves a repo's release trees with its downloads: going public2+# extracts them into .forge/store and links www's from there, going private3+# into .forge/private/store and links the private root's from there. No4+# download is left a link into the other side's store, and gc frees the5+# trees the flip left behind there.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo --private dami/greet9+clone dami dami/greet "$T/w/d"10+at "$T/w/d" commit "first"11+at "$T/w/d" git tag v112+ok push_as dami "$T/w/d" origin main13+mkdir "$T/f"14+printf 'greet 1\n' >"$T/f/greet"15+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/greet"16+tree=$(git --git-dir="$FORGE_ROOT/dami/greet.git" rev-parse 'refs/releases/v1^{tree}')17+pub=$FORGE_ROOT/.forge/store/$tree18+priv=$FORGE_ROOT/.forge/private/store/$tree19+w=$FORGE_WWW/dami/greet/releases/download20+p=$FORGE_ROOT/.forge/private/www/dami/greet/releases/download21+inode() { ls -i "$1" | awk '{ print $1 }'; }22+missing "$pub"23+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"24+25+ok ssh_as dami public dami/greet26+missing "$p"27+eq "$(inode "$w/v1/greet")" "$(inode "$pub/greet")"28+eq "$(inode "$w/latest/greet")" "$(inode "$pub/greet")"29+build dami/greet30+eq "$(inode "$w/v1/greet")" "$(inode "$pub/greet")"31+ok forge gc --age 032+has "$T_LOG" "^removed private store $tree "33+lacks "$T_LOG" "^removed store "34+missing "$priv"35+has "$w/v1/greet" 'greet 1'36+37+ok ssh_as dami private dami/greet38+missing "$FORGE_WWW/dami/greet"39+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"40+eq "$(inode "$p/latest/greet")" "$(inode "$priv/greet")"41+build dami/greet42+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"43+ok forge gc --age 044+has "$T_LOG" "^removed store $tree "45+lacks "$T_LOG" "^removed private store "46+missing "$pub"47+has "$p/v1/greet" 'greet 1'@@ -0,0 +1,68 @@
1+# forge gc frees unused trees in both stores: .forge/store keeps what sites2+# link to and public repos' releases name, .forge/private/store what private3+# repos' releases name. Fresh trees wait out the grace (--age), --dry-run4+# only lists, and a removed private repo's trees go with it while its5+# trash is purged on time.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo dami/open9+repo --private dami/secret10+mkdir "$T/f"11+export FORGE_USER=dami12+for r in open secret; do13+ clone dami "dami/$r" "$T/w/$r"14+ for v in 1 2; do15+ tick 360016+ at "$T/w/$r" commit "v$v"17+ at "$T/w/$r" git tag "v$v"18+ ok push_as dami "$T/w/$r" origin main19+ printf '%s %s\n' "$r" "$v" >"$T/f/tool"20+ ok at "$T/w/$r" "$FORGE" release publish "v$v" "$T/f/tool"21+ eval "${r}$v=$(git --git-dir="$FORGE_ROOT/dami/$r.git" rev-parse "refs/releases/v$v^{tree}")"22+ done23+done24+ok push_as dami "$T/w/open" origin "$(site_commit "$T/w/open" docs):refs/sites/docs"25+unset FORGE_USER26+site=$(basename "$(readlink "$FORGE_WWW/sites/dami/open/docs")")27+store=$FORGE_ROOT/.forge/store28+priv=$FORGE_ROOT/.forge/private/store29+30+# Everything is in use.31+ok forge gc --age 032+lacks "$T_LOG" 'removed (release|store|private store)'33+for t in "$open1" "$open2" "$site"; do exists "$store/$t"; done34+for t in "$secret1" "$secret2"; do exists "$priv/$t"; done35+36+# Dead trees in each: kept while fresh, listed by --dry-run, then freed.37+git --git-dir="$FORGE_ROOT/dami/open.git" update-ref -d refs/releases/v138+git --git-dir="$FORGE_ROOT/dami/secret.git" update-ref -d refs/releases/v139+ok forge gc40+lacks "$T_LOG" 'removed (store|private store)'41+exists "$store/$open1"42+exists "$priv/$secret1"43+ok forge gc --age 0 --dry-run44+has "$T_LOG" "^would remove store $open1 "45+has "$T_LOG" "^would remove private store $secret1 "46+exists "$store/$open1"47+exists "$priv/$secret1"48+ok forge gc --age 049+has "$T_LOG" "^removed store $open1 "50+has "$T_LOG" "^removed private store $secret1 "51+has "$T_LOG" '^gc: removed 2 from the stores'52+missing "$store/$open1"53+missing "$priv/$secret1"54+for t in "$open2" "$site"; do exists "$store/$t"; done55+exists "$priv/$secret2"56+57+# A removed private repo: its downloads go at once, its tree at gc, its58+# trash after FORGE_TRASH_DAYS.59+ok forge repo rm dami/secret60+missing "$FORGE_ROOT/.forge/private/www/dami/secret"61+ok forge gc --age 062+has "$T_LOG" "^removed private store $secret2 "63+lacks "$T_LOG" '^removed trash'64+missing "$priv/$secret2"65+ok env FORGE_TRASH_DAYS=0 "$FORGE" gc --age 066+has "$T_LOG" '^removed trash dami/secret\.git\.'67+eq "$(find "$FORGE_ROOT/.forge/trash" -mindepth 2 | wc -l | tr -d ' ')" 068+for t in "$open2" "$site"; do exists "$store/$t"; done@@ -0,0 +1,30 @@
1+# A private repo's release tree is extracted into .forge/private/store, under2+# the directory only the forge can read, never into .forge/store, which the3+# caddy user can read; its downloads in the private root, latest/ too, are4+# hard links into it, the push's and the build's alike.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo --private dami/secret8+clone dami dami/secret "$T/w/d"9+at "$T/w/d" commit "first"10+at "$T/w/d" git tag v111+ok push_as dami "$T/w/d" origin main12+mkdir "$T/f"13+printf 'secret tool\n' >"$T/f/tool"14+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool"15+tree=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')16+ps=$FORGE_ROOT/.forge/private/store/$tree17+dl=$FORGE_ROOT/.forge/private/www/dami/secret/releases/download18+inode() { ls -i "$1" | awk '{ print $1 }'; }19+ok cmp "$T/f/tool" "$ps/tool"20+exists "$ps/SHA256SUMS.sig"21+missing "$FORGE_ROOT/.forge/store/$tree"22+eq "$(inode "$dl/v1/tool")" "$(inode "$ps/tool")"23+eq "$(inode "$dl/latest/tool")" "$(inode "$ps/tool")"24+eq "$(ls -ld "$FORGE_ROOT/.forge/private" | cut -c1-10)" drwx------25+26+build dami/secret27+eq "$(inode "$dl/v1/tool")" "$(inode "$ps/tool")"28+eq "$(inode "$dl/latest/tool")" "$(inode "$ps/tool")"29+missing "$FORGE_ROOT/.forge/store/$tree"30+if find "$FORGE_ROOT/.forge/store" "$FORGE_WWW" -name tool | grep -q .; then fail "a private release file is in .forge/store or www"; fi