A private repo's release files are extracted into .forge/private/store (mode 700 parent) instead of .forge/store; flips, gc, backups and restore follow.
revisions2r2: behavior change
| r2 | cat | d6847cf | what changed since r1 behavior change | |
| r1 | cat | 42f0525 |
No comments yet: forge comment 4 adds one.
1/4 Start no background lake in CI's restored forge
The restore's build kicks forge lake to run 10 seconds later, in the background, in the restored root. CI removes that root once its checks and the restic round trip are done, and on a busy machine those took about as long: the lake wrote into the root while rm -rf was emptying it, rm failed with "Directory not empty", and so did CI. The restore now runs with FORGE_LAKE=0, which starts none; nothing there checks the restored forge's lake.
| M | .forge/ | 6 | ++++-- |
@@ -80,10 +80,12 @@ test -f "$w/sites/dami/forge/guide/index.html"
8080 # Backups: stage the demo forge the way forge backup does, restore that into8181 # an empty root, and check the forge comes back whole: hooks, keys, refs,8282 # review logs and comments, sites, pages. The restic round trip needs83-# restic, which the container image has.83+# restic, which the container image has. The restore's build would start a84+# forge lake in the background 10 seconds on, writing into $r while it is85+# removed at the end; FORGE_LAKE=0 starts none.8486 r=$(mktemp -d "${TMPDIR:-/tmp}/forge-ci-restore.XXXXXX")8587 FORGE_ROOT=$PWD/demo/root sh bin/forge backup --dry-run 2>/dev/null86-FORGE_ROOT=$r sh bin/forge restore --from demo/root/.forge/backup/stage >/dev/null 2>&188+FORGE_LAKE=0 FORGE_ROOT=$r sh bin/forge restore --from demo/root/.forge/backup/stage >/dev/null 2>&18789 test -x "$r/dami/greet.git/hooks/pre-receive"8890 grep -q 'shell ana"' "$r/.ssh/authorized_keys"8991 diff -r demo/root/dami/greet.git/forge "$r/dami/greet.git/forge"2/4 Keep private repos' release trees in a store only the forge reads
A private repo's release files were extracted into .forge/store and hard-linked from there into the private root. The caddy user can read .forge/store, since sites are served from it; nothing serves a release tree from it, but a private repo's files shouldn't sit there. They are now extracted into .forge/private/store, under the directory only the forge can read, beside the private root, so the links stay on one filesystem. Which store follows the repo's flag, as the pages root does, so going private links the downloads from the private store and going public from .forge/store, extracting them there. gc frees trees in both, each keeping what its own side's release refs name (and .forge/store what sites link to), so a flipped repo's old trees go.
| M | README.md | 45 | +++++++++++----- |
| M | bin/ | 81 | ++++++++++++++++++++++-------- |
| M | test/ | 13 | +++-- |
| A | test/ | 47 | +++++++++++++++++ |
| A | test/ | 68 | +++++++++++++++++++++++++ |
| A | test/ | 30 | +++++++++++ |
@@ -193,7 +193,8 @@ ssh git@git.example.com ls # yours, and the private o
193193 The owner and collaborators flip a repo, the way Forgejo does. `private`194194 removes the flag, the repo's public pages, its entries on the index, owner195195 pages and `repos.json`, and its served sites, previews and their196-hostnames, moves its release downloads into the private root, and builds196+hostnames, moves its release downloads into the private root (and their197+files into the private store, see [Releases](#releases)), and builds197198 its pages again there. Then it starts198199 a mirror sync, which deletes them from the bucket. Clones, caches and copies199200 people already have can't be recalled. `public` first runs the secret check@@ -484,7 +485,8 @@ The secret check skips releases. It reads branches and tags, and a
484485 release's source is its tag.485486 486487 `post-receive` extracts the tree once into `.forge/store/<tree>/`, as for a487-site, and hard-links its files into488+site (a private repo's into a store of its own, below), and hard-links its489+files into488490 `/<owner>/<repo>/releases/download/<tag>/` (a `/` in the tag becomes `~`).489491 `latest/` next to them holds the files of the newest tag that has a490492 release, so `/dami/greet/releases/download/latest/greet-linux-amd64` keeps@@ -508,10 +510,22 @@ doesn't see them and `forge.publicsites` doesn't change that. The private
508510 server gives them to its readers, signed in or showing a token (see509511 [Private pages](#private-pages)) as Caddy gives a public repo's510512 (attachments, uncompressed, byte ranges) but `Cache-Control: private,511-no-cache`, and anyone else gets the 404 of a file that isn't there. A repo512-that goes private takes its downloads there at once, and one that goes513-public brings them back to `www`. Readers can also get them over ssh, and to514-anyone else these commands answer as for a missing repo:513+no-cache`, and anyone else gets the 404 of a file that isn't there.514+515+Their files aren't extracted into `.forge/store` either, but into516+`.forge/private/store/<tree>/`, and linked from there. `.forge/store` is517+readable by the caddy user, which serves sites from it; nothing serves a518+release tree from a store, but a private repo's files shouldn't sit where519+anything but the forge can read them, so they live under `.forge/private`,520+which only git can read (mode 700). Both stores are under `.forge`, so the521+downloads stay hard links on one filesystem. A repo that goes private takes522+its downloads to the private root at once, linked from the private store523+(extracted there), and one that goes public brings them back to `www`,524+linked from `.forge/store` (extracted there again); `forge gc` frees the525+trees left in the store it no longer uses.526+527+Readers can also get them over ssh, and to anyone else these commands528+answer as for a missing repo:515529 516530 ```sh517531 curl -fO -u ana:<token> https://git.example.com/dami/notes/releases/download/v1.0/tool # token --repo dami/notes@@ -800,14 +814,17 @@ push (log: `.forge/build.log`); `forge build` rebuilds everything.
800814 801815 ### Cleaning up802816 803-Every distinct deploy is extracted once into `.forge/store/<tree>/`, and the804-old tree stays when a newer deploy takes its link. `forge gc` removes store805-trees that no `www/sites` or `www/previews` link points at (hostnames point806-at those links) and no `refs/releases/*` ref names, a private repo's too807-(release files are hard links, which name no tree), hostnames whose deploy808-is gone, previews of review branches that no longer exist, and the809-downloads of releases whose ref was deleted without the hooks, in `www` or810-the private root. A deploy extracts its tree before it links it, and817+Every distinct deploy is extracted once into `.forge/store/<tree>/` (a818+private repo's releases into `.forge/private/store/<tree>/`), and the old819+tree stays when a newer deploy takes its link. `forge gc` removes trees in820+`.forge/store` that no `www/sites` or `www/previews` link points at821+(hostnames point at those links) and no public repo's `refs/releases/*` ref822+names, and trees in `.forge/private/store` that no private repo's release823+ref names (release files are hard links, which name no tree), so a repo824+that flipped leaves nothing in the store it left. It also removes hostnames825+whose deploy is gone, previews of review branches that no longer exist,826+and the downloads of releases whose ref was deleted without the hooks, in827+`www` or the private root. A deploy extracts its tree before it links it, and811828 touches a tree it reuses, so store entries changed in the last 10 minutes812829 are kept (`--age <minutes>`), which makes gc safe to run while people push. `--dry-run` lists813830 what it would remove. A release published again leaves its old files in the@@ -962,6 +962,8 @@ load_config() {
962962 # Private repos' pages: the same layout as www, served only to their963963 # readers (lib/private-server.ts).964964 FORGE_PRIVATE_WWW=${FORGE_PRIVATE_WWW:-$FORGE_STATE/private/www}965+ # Their release trees, beside it (release_store).966+ FORGE_PRIVATE_STORE=$(dirname "$FORGE_PRIVATE_WWW")/store965967 FORGE_AUTHORIZED_KEYS=${FORGE_AUTHORIZED_KEYS:-$FORGE_ROOT/.ssh/authorized_keys}966968 export FORGE_ROOT FORGE_STATE FORGE_WWW FORGE_PRIVATE_WWW967969 }@@ -1738,7 +1740,8 @@ sc_private() {
17381740 rm -f "$RDIR/git-daemon-export-ok"17391741 rm -rf "${FORGE_WWW:?}/$1"17401742 # Its release downloads go with its pages, and are in the private root1741- # for its readers now rather than after the build.1743+ # for its readers now rather than after the build, linked from the1744+ # private store (release_store); gc frees the trees it left in .forge/store.17421745 (17431746 export GIT_DIR="$RDIR"17441747 OWNER=${1%%/*} NAME=${1#*/}@@ -1785,6 +1788,8 @@ sc_public() {
17851788 esac17861789 touch "$RDIR/git-daemon-export-ok"17871790 rm -rf "${FORGE_PRIVATE_WWW:?}/$1"1791+ # Release downloads are linked from .forge/store again (release_store);1792+ # gc frees the repo's trees in the private store.17881793 (17891794 export GIT_DIR="$RDIR"17901795 OWNER=${1%%/*} NAME=${1#*/}@@ -2218,14 +2223,15 @@ push_site() {
22182223 fi22192224 }22202225 2221-# store_tree <commit>: extract its tree once into the content store,2222-# .forge/store/<tree>/, and print where. Sites link to it (deploy_site),2223-# releases hard-link its files (release_link). git archive stamps every file2226+# store_tree <commit> [<store>]: extract its tree once into the content2227+# store, .forge/store/<tree>/, or into <store>/<tree>/, and print where.2228+# Sites link to it (deploy_site), releases hard-link its files from the2229+# store release_store names (release_link). git archive stamps every file22242230 # with the commit's time, so a tree extracted again (after a restore) gets22252231 # the same mtimes. A reused tree counts as new, so a concurrent forge gc,22262232 # which spares what changed in its last --age minutes, leaves it alone.22272233 store_tree() {2228- st_dir=$FORGE_STATE/store/$(git rev-parse "$1^{tree}")2234+ st_dir=${2:-$FORGE_STATE/store}/$(git rev-parse "$1^{tree}")22292235 if [ ! -d "$st_dir" ]; then22302236 rm -rf "$st_dir.tmp.$$"22312237 mkdir -p "$st_dir.tmp.$$"@@ -2238,17 +2244,18 @@ store_tree() {
22382244 22392245 # release_link <root> <tag> <commit|ZERO>: make22402246 # <root>/<owner>/<repo>/releases/download/<tag>/ hard links to the files of2241-# a release, extracted once into the store, or remove it. Hard links, not a2242-# symlink like a site's: rclone (forge mirror) skips symlinks, and the files2243-# keep the store's mtime. Runs in a repo (GIT_DIR) with OWNER and NAME set;2244-# <root> is the repo's pages root (pages_root) or a build's output.2247+# a release, extracted once into the repo's store (release_store), or remove2248+# it. Hard links, not a symlink like a site's: rclone (forge mirror) skips2249+# symlinks, and the files keep the store's mtime. Runs in a repo (GIT_DIR)2250+# with OWNER and NAME set; <root> is the repo's pages root (pages_root) or a2251+# build's output.22452252 release_link() {22462253 rl=$1/$OWNER/$NAME/releases/download/$(enc_ref "$2")22472254 if [ "$3" = "$ZERO" ]; then22482255 rm -rf "$rl"22492256 return 022502257 fi2251- rs=$(store_tree "$3")2258+ rs=$(store_tree "$3" "$(release_store)")22522259 rm -rf "$rl.tmp.$$"22532260 link_files "$rl.tmp.$$" "$rs"/*22542261 swap "$rl.tmp.$$" "$rl"@@ -2292,6 +2299,24 @@ private_www() {
22922299 chmod 700 "$(dirname "$FORGE_PRIVATE_WWW")"22932300 }22942301 2302+# release_store: where the repo's release trees are extracted, to be2303+# hard-linked into its pages. A public repo's go in the content store, which2304+# the caddy user can read, since sites are served from it. A private repo's2305+# go in FORGE_PRIVATE_STORE, beside the private root under the directory2306+# only the forge can read: nothing serves a store, but those files are a2307+# private repo's all the same. Both are under .forge, so the links stay on2308+# one filesystem. Like pages_root it follows the repo's flag, so a flip2309+# links the downloads from the store of the side the repo goes to,2310+# extracting them there. Runs in a repo (GIT_DIR).2311+release_store() {2312+ if [ -f "$GIT_DIR/git-daemon-export-ok" ]; then2313+ printf '%s' "$FORGE_STATE/store"2314+ else2315+ private_www2316+ printf '%s' "$FORGE_PRIVATE_STORE"2317+ fi2318+}2319+22952320 # pages_root <git-dir>: where the repo's pages and release downloads live:22962321 # www for a public repo, the private root for a private one.22972322 pages_root() {@@ -2613,7 +2638,8 @@ was_review() { grep -qxF "branch $2" "$1"/forge/reviews/* 2>/dev/null; }
26132638 # branches that are gone (deleted outside the hook, or before it removed26142639 # previews), hostnames whose deploy is gone, downloads of deleted releases,26152640 # and store trees that no site or preview links to (hostnames point at those2616-# links, never at the store) and no release ref names.2641+# links, never at the store) and no release ref names, in the content store2642+# and the private store (release_store), each keeping its own side's.26172643 # A deploy extracts its tree before linking it, so entries changed in the26182644 # last --age minutes stay. Then repacks every repo (repack_repo).26192645 cmd_gc() {@@ -2677,27 +2703,38 @@ cmd_gc() {
26772703 # Candidates before links: a deploy that links a tree after this looked26782704 # has touched it first, so it can't be a candidate.26792705 n=0 kb=02680- if [ -d "$FORGE_STATE/store" ]; then2681- set --2682- [ "$age" = 0 ] || set -- -mmin +"$age"2683- old=$(find "$FORGE_STATE/store" -mindepth 1 -maxdepth 1 "$@")2684- used=$(for l in "$FORGE_WWW"/sites/*/*/* "$FORGE_WWW"/previews/*/*/*/*; do if [ -L "$l" ]; then basename "$(readlink "$l")"; fi; done2685- # Release files are hard links, which don't name their tree: the refs do.2686- for d in $repos; do git --git-dir="$FORGE_ROOT/$d" for-each-ref --format='%(tree)' refs/releases/; done)2706+ set --2707+ [ "$age" = 0 ] || set -- -mmin +"$age"2708+ for st in "$FORGE_STATE/store" "$FORGE_PRIVATE_STORE"; do2709+ [ -d "$st" ] || continue2710+ pub=1 what=store2711+ [ "$st" = "$FORGE_STATE/store" ] || pub=0 what='private store'2712+ old=$(find "$st" -mindepth 1 -maxdepth 1 "$@")2713+ used=$(if [ "$pub" = 1 ]; then for l in "$FORGE_WWW"/sites/*/*/* "$FORGE_WWW"/previews/*/*/*/*; do if [ -L "$l" ]; then basename "$(readlink "$l")"; fi; done; fi2714+ # Release files are hard links, which don't name their tree: the2715+ # refs do, of the repos on this store's side. A repo that flipped2716+ # links from the other store now (release_store), so its trees2717+ # here go.2718+ for d in $repos; do2719+ p=02720+ [ ! -f "$FORGE_ROOT/$d/git-daemon-export-ok" ] || p=12721+ [ "$p" = "$pub" ] || continue2722+ git --git-dir="$FORGE_ROOT/$d" for-each-ref --format='%(tree)' refs/releases/2723+ done)26872724 for s in $old; do26882725 printf '%s\n' "$used" | grep -qxF "${s##*/}" && continue26892726 k=$(du -sk "$s" | cut -f1)26902727 [ "$dry" = 1 ] || rm -rf "$s"2691- echo "$did store ${s##*/} (${k}K)"2728+ echo "$did $what ${s##*/} (${k}K)"26922729 n=$((n + 1)) kb=$((kb + k))26932730 done2694- fi2731+ done26952732 if [ "$dry" = 1 ]; then2696- echo "gc: would remove $n from the store (${kb}K); repacking skipped"2733+ echo "gc: would remove $n from the stores (${kb}K); repacking skipped"26972734 return26982735 fi26992736 for d in $repos; do repack_repo "$FORGE_ROOT/$d" || say "forge: repacking $d failed"; done2700- echo "gc: removed $n from the store (${kb}K)"2737+ echo "gc: removed $n from the stores (${kb}K)"27012738 }27022739 27032740 # Geometric repacking merges packs only until each is at least twice the@@ -1,6 +1,7 @@
1-# forge gc keeps the store tree of a private repo's release, whose files2-# are hard links in the private root, and drops the downloads of a release3-# whose ref went behind the hooks' back there as in www, freeing its tree.1+# forge gc keeps the private store tree of a private repo's release, whose2+# files are hard links in the private root, and drops the downloads of a3+# release whose ref went behind the hooks' back there as in www, freeing4+# its tree.45 . "$(dirname "$0")/../lib.sh"56 user dami67 repo --private dami/secret@@ -19,21 +20,21 @@ for v in 1 2; do
1920 done2021 unset FORGE_USER2122 build22-store=$FORGE_ROOT/.forge/store23+store=$FORGE_ROOT/.forge/private/store2324 dl=$FORGE_ROOT/.forge/private/www/dami/secret/releases/download2425 exists "$store/$tree1/tool"2526 exists "$store/$tree2/tool"2627 has "$dl/latest/tool" 'secret 2'2728 2829 ok forge gc --age 029-lacks "$T_LOG" 'removed (release|store)'30+lacks "$T_LOG" 'removed (release|store|private store)'3031 exists "$store/$tree1/tool"3132 exists "$store/$tree2/tool"3233 3334 git --git-dir="$g" update-ref -d refs/releases/v23435 ok forge gc --age 03536 has "$T_LOG" '^removed release dami/secret v2$'36-has "$T_LOG" "^removed store $tree2 "37+has "$T_LOG" "^removed private store $tree2 "3738 lacks "$T_LOG" "$tree1"3839 missing "$dl/v2"3940 has "$dl/v1/tool" 'secret 1'@@ -0,0 +1,47 @@
1+# A flip moves a repo's release trees with its downloads: going public2+# extracts them into .forge/store and links www's from there, going private3+# into .forge/private/store and links the private root's from there. No4+# download is left a link into the other side's store, and gc frees the5+# trees the flip left behind there.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo --private dami/greet9+clone dami dami/greet "$T/w/d"10+at "$T/w/d" commit "first"11+at "$T/w/d" git tag v112+ok push_as dami "$T/w/d" origin main13+mkdir "$T/f"14+printf 'greet 1\n' >"$T/f/greet"15+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/greet"16+tree=$(git --git-dir="$FORGE_ROOT/dami/greet.git" rev-parse 'refs/releases/v1^{tree}')17+pub=$FORGE_ROOT/.forge/store/$tree18+priv=$FORGE_ROOT/.forge/private/store/$tree19+w=$FORGE_WWW/dami/greet/releases/download20+p=$FORGE_ROOT/.forge/private/www/dami/greet/releases/download21+inode() { ls -i "$1" | awk '{ print $1 }'; }22+missing "$pub"23+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"24+25+ok ssh_as dami public dami/greet26+missing "$p"27+eq "$(inode "$w/v1/greet")" "$(inode "$pub/greet")"28+eq "$(inode "$w/latest/greet")" "$(inode "$pub/greet")"29+build dami/greet30+eq "$(inode "$w/v1/greet")" "$(inode "$pub/greet")"31+ok forge gc --age 032+has "$T_LOG" "^removed private store $tree "33+lacks "$T_LOG" "^removed store "34+missing "$priv"35+has "$w/v1/greet" 'greet 1'36+37+ok ssh_as dami private dami/greet38+missing "$FORGE_WWW/dami/greet"39+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"40+eq "$(inode "$p/latest/greet")" "$(inode "$priv/greet")"41+build dami/greet42+eq "$(inode "$p/v1/greet")" "$(inode "$priv/greet")"43+ok forge gc --age 044+has "$T_LOG" "^removed store $tree "45+lacks "$T_LOG" "^removed private store "46+missing "$pub"47+has "$p/v1/greet" 'greet 1'@@ -0,0 +1,68 @@
1+# forge gc frees unused trees in both stores: .forge/store keeps what sites2+# link to and public repos' releases name, .forge/private/store what private3+# repos' releases name. Fresh trees wait out the grace (--age), --dry-run4+# only lists, and a removed private repo's trees go with it while its5+# trash is purged on time.6+. "$(dirname "$0")/../lib.sh"7+user dami8+repo dami/open9+repo --private dami/secret10+mkdir "$T/f"11+export FORGE_USER=dami12+for r in open secret; do13+ clone dami "dami/$r" "$T/w/$r"14+ for v in 1 2; do15+ tick 360016+ at "$T/w/$r" commit "v$v"17+ at "$T/w/$r" git tag "v$v"18+ ok push_as dami "$T/w/$r" origin main19+ printf '%s %s\n' "$r" "$v" >"$T/f/tool"20+ ok at "$T/w/$r" "$FORGE" release publish "v$v" "$T/f/tool"21+ eval "${r}$v=$(git --git-dir="$FORGE_ROOT/dami/$r.git" rev-parse "refs/releases/v$v^{tree}")"22+ done23+done24+ok push_as dami "$T/w/open" origin "$(site_commit "$T/w/open" docs):refs/sites/docs"25+unset FORGE_USER26+site=$(basename "$(readlink "$FORGE_WWW/sites/dami/open/docs")")27+store=$FORGE_ROOT/.forge/store28+priv=$FORGE_ROOT/.forge/private/store29+30+# Everything is in use.31+ok forge gc --age 032+lacks "$T_LOG" 'removed (release|store|private store)'33+for t in "$open1" "$open2" "$site"; do exists "$store/$t"; done34+for t in "$secret1" "$secret2"; do exists "$priv/$t"; done35+36+# Dead trees in each: kept while fresh, listed by --dry-run, then freed.37+git --git-dir="$FORGE_ROOT/dami/open.git" update-ref -d refs/releases/v138+git --git-dir="$FORGE_ROOT/dami/secret.git" update-ref -d refs/releases/v139+ok forge gc40+lacks "$T_LOG" 'removed (store|private store)'41+exists "$store/$open1"42+exists "$priv/$secret1"43+ok forge gc --age 0 --dry-run44+has "$T_LOG" "^would remove store $open1 "45+has "$T_LOG" "^would remove private store $secret1 "46+exists "$store/$open1"47+exists "$priv/$secret1"48+ok forge gc --age 049+has "$T_LOG" "^removed store $open1 "50+has "$T_LOG" "^removed private store $secret1 "51+has "$T_LOG" '^gc: removed 2 from the stores'52+missing "$store/$open1"53+missing "$priv/$secret1"54+for t in "$open2" "$site"; do exists "$store/$t"; done55+exists "$priv/$secret2"56+57+# A removed private repo: its downloads go at once, its tree at gc, its58+# trash after FORGE_TRASH_DAYS.59+ok forge repo rm dami/secret60+missing "$FORGE_ROOT/.forge/private/www/dami/secret"61+ok forge gc --age 062+has "$T_LOG" "^removed private store $secret2 "63+lacks "$T_LOG" '^removed trash'64+missing "$priv/$secret2"65+ok env FORGE_TRASH_DAYS=0 "$FORGE" gc --age 066+has "$T_LOG" '^removed trash dami/secret\.git\.'67+eq "$(find "$FORGE_ROOT/.forge/trash" -mindepth 2 | wc -l | tr -d ' ')" 068+for t in "$open2" "$site"; do exists "$store/$t"; done@@ -0,0 +1,30 @@
1+# A private repo's release tree is extracted into .forge/private/store, under2+# the directory only the forge can read, never into .forge/store, which the3+# caddy user can read; its downloads in the private root, latest/ too, are4+# hard links into it, the push's and the build's alike.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo --private dami/secret8+clone dami dami/secret "$T/w/d"9+at "$T/w/d" commit "first"10+at "$T/w/d" git tag v111+ok push_as dami "$T/w/d" origin main12+mkdir "$T/f"13+printf 'secret tool\n' >"$T/f/tool"14+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool"15+tree=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')16+ps=$FORGE_ROOT/.forge/private/store/$tree17+dl=$FORGE_ROOT/.forge/private/www/dami/secret/releases/download18+inode() { ls -i "$1" | awk '{ print $1 }'; }19+ok cmp "$T/f/tool" "$ps/tool"20+exists "$ps/SHA256SUMS.sig"21+missing "$FORGE_ROOT/.forge/store/$tree"22+eq "$(inode "$dl/v1/tool")" "$(inode "$ps/tool")"23+eq "$(inode "$dl/latest/tool")" "$(inode "$ps/tool")"24+eq "$(ls -ld "$FORGE_ROOT/.forge/private" | cut -c1-10)" drwx------25+26+build dami/secret27+eq "$(inode "$dl/v1/tool")" "$(inode "$ps/tool")"28+eq "$(inode "$dl/latest/tool")" "$(inode "$ps/tool")"29+missing "$FORGE_ROOT/.forge/store/$tree"30+if find "$FORGE_ROOT/.forge/store" "$FORGE_WWW" -name tool | grep -q .; then fail "a private release file is in .forge/store or www"; fi3/4 Leave the private store out of backups, and rebuild it on restore
The private store holds files extracted from release refs, which a backup has, so like .forge/store it is left out (it is under .forge/private, which derived_state leaves out already) and a restore rebuilds it: restore_finish links a private repo's downloads into the private root through release_store, which extracts them into the private store again. A scenario pins both, and the README says so.
| M | README.md | 6 | +++-- |
| M | bin/ | 16 | +++++++----- |
| A | test/ | 40 | ++++++++++++++++++++++++++++++ |
@@ -936,7 +936,8 @@ Everything in `/srv/git` that can't be rebuilt:
936936 937937 Left out, because a restore rebuilds it or every boot rewrites it: the pages in938938 `.forge/www` and `.forge/private`, the extracted sites and releases in939-`.forge/store` (both are git refs),939+`.forge/store` and private repos' releases in `.forge/private/store` (all940+are git refs),940941 `.forge/cache`, `authorized_keys`, `allowed_signers`, `git-shell-commands/`,941942 hooks, `.forge/secrets` and `.forge/backup.env` (both written from `forge.env`942943 at boot), and logs, locks and scratch files.@@ -1074,7 +1075,8 @@ What you set up yourself, once:
10741075 `/srv/git`. It restores the files, rebuilds what a backup leaves out (what10751076 `forge server init` and `forge keys sync` write, hooks in every repo, pinned10761077 review revisions, extracted sites, release downloads, a private repo's in1077-the private root, with the mtimes they had, so the mirror uploads none1078+the private root, extracted into `.forge/private/store` again, with the1079+mtimes they had, so the mirror uploads none10781080 again) and runs `forge10791081 build`. identity.db is the backup's copy or litestream's replica, whichever10801082 changed last (see [above](#identitydb-backed-up-as-it-changes)). It refuses@@ -2900,10 +2900,11 @@ stage_repo() {
29002900 }29012901 29022902 # .forge entries a backup leaves out: rebuilt (www, store, cache,2903-# allowed_signers, private pages), regenerated at boot or by commands2904-# (secrets, litestream.yml), or logs, locks and scratch. The lake is staged2905-# by itself (stage_lake), and identity.db by stage_identity; its WAL and2906-# litestream's state are never copied.2903+# allowed_signers, private: private pages and the private store),2904+# regenerated at boot or by commands (secrets, litestream.yml), or logs,2905+# locks and scratch. The lake is staged by itself (stage_lake), and2906+# identity.db by stage_identity; its WAL and litestream's state are never2907+# copied.29072908 derived_state() {29082909 case $1 in29092910 www | store | cache | backup | secrets | allowed_signers | config.restored | restore.tmp | lake | private | \@@ -3202,9 +3203,10 @@ restore_finish() {
32023203 awk '$2 == "commit" { print "update refs/forge/keep/" $1 " " $1 }' | git update-ref --stdin32033204 # A private repo's sites stay unserved (sites_public), as before the backup.32043205 if sites_public "$GIT_DIR"; then redeploy_sites; fi3205- # Releases are refs too: a public repo's go to www, a private repo's to3206- # the private root. The files get their commit's time again, so the3207- # mirror sees nothing new.3206+ # Releases are refs too: a public repo's go to www, linked from3207+ # .forge/store, and a private repo's to the private root, extracted3208+ # into the private store again (release_store). The files get their3209+ # commit's time again, so the mirror sees nothing new.32083210 releases_into "$(pages_root "$GIT_DIR")"32093211 git update-server-info32103212 unset GIT_DIR@@ -0,0 +1,40 @@
1+# A backup leaves .forge/private/store out, like .forge/store, since the2+# release refs it is extracted from are in the backup; a restore extracts a3+# private repo's releases into it again and links its downloads from there,4+# and a public repo's from .forge/store.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo dami/open8+repo --private dami/secret9+mkdir "$T/f"10+printf 'tool\n' >"$T/f/tool"11+export FORGE_USER=dami12+for r in open secret; do13+ clone dami "dami/$r" "$T/w/$r"14+ at "$T/w/$r" commit "first $r"15+ at "$T/w/$r" git tag v116+ ok push_as dami "$T/w/$r" origin main17+ ok at "$T/w/$r" "$FORGE" release publish v1 "$T/f/tool"18+done19+unset FORGE_USER20+open=$(git --git-dir="$FORGE_ROOT/dami/open.git" rev-parse 'refs/releases/v1^{tree}')21+secret=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')22+exists "$FORGE_ROOT/.forge/private/store/$secret/tool"23+24+ok forge backup --dry-run25+st=$FORGE_ROOT/.forge/backup/stage26+missing "$st/.forge/private"27+missing "$st/.forge/store"28+exists "$st/dami/secret.git/HEAD"29+30+r=$T/restored31+ok env FORGE_ROOT="$r" FORGE_WWW="$r/.forge/www" FORGE_CACHE="$T/cache2" "$FORGE" restore --from "$st"32+inode() { ls -i "$1" | awk '{ print $1 }'; }33+dl=dami/secret/releases/download34+ok cmp "$T/f/tool" "$r/.forge/private/store/$secret/tool"35+eq "$(inode "$r/.forge/private/www/$dl/v1/tool")" "$(inode "$r/.forge/private/store/$secret/tool")"36+eq "$(inode "$r/.forge/private/www/$dl/latest/tool")" "$(inode "$r/.forge/private/store/$secret/tool")"37+missing "$r/.forge/store/$secret"38+eq "$(ls -ld "$r/.forge/private" | cut -c1-10)" drwx------39+eq "$(inode "$r/.forge/www/dami/open/releases/download/v1/tool")" "$(inode "$r/.forge/store/$open/tool")"40+missing "$r/.forge/private/store/$open"4/4 Build private pages in a scratch directory under .forge/private
A private repo's build wrote its pages, and hard-linked its release files, in .forge/build.<pid> before swapping them into the private root, so while it ran they sat outside the directory only the forge can read. It now builds in .forge/private/build.<pid>, beside the private root: nothing of the repo is outside .forge/private even mid-build, and the swaps stay renames on one filesystem (the assets go to www/assets, under .forge too). A scenario records where each build writes through a stand-in node, which 11ty runs through env, and checks the private one is under .forge/private while the public one is not.
| M | README.md | 5 | +++- |
| M | bin/ | 8 | ++++- |
| A | test/ | 46 | ++++++++++++++++++++++++++++++ |
@@ -211,7 +211,10 @@ any left in the wrong place.
211211 212212 A private repo's pages are built like any repo's, into213213 `.forge/private/www` instead of `www`, with the same layout: pages, search214-index, `code.db`, release downloads, insights exports. They link the same hashed assets, which the build puts in214+index, `code.db`, release downloads, insights exports. The build writes215+them in `.forge/private/build.<pid>` and swaps them in from there, so216+they are never outside `.forge/private`, which only git can read, even217+mid-build. They link the same hashed assets, which the build puts in215218 `www/assets` (they are the forge's scripts and styles, nothing of the216219 repo), and their header marks the repo private. Caddy serves a file of217220 `www` when there is one and hands every other path to@@ -2472,7 +2472,11 @@ build_all() {
24722472 # hashed assets, so the build's assets go into www/assets as any build's24732473 # do; they are templates' CSS and the forge's scripts, nothing of the repo.24742474 # The rest of what the build writes (index and owner pages) is dropped.2475-# Runs in a subshell of cmd_build, for its EXIT trap.2475+# It builds in a scratch directory beside the private root, under the2476+# directory only the forge can read, so the pages and release links are2477+# never outside it, even mid-build, and the swaps stay renames (the assets'2478+# too: www is under .forge as well). Runs in a subshell of cmd_build, for2479+# its EXIT trap.24762480 build_private() {24772481 only=$124782482 lock=$FORGE_STATE/build.lock@@ -2482,7 +2486,7 @@ build_private() {
24822486 [ "$n" -lt 600 ] || die "build lock held: $lock"24832487 sleep 124842488 done2485- out=$FORGE_STATE/build.$$2489+ out=$(dirname "$FORGE_PRIVATE_WWW")/build.$$24862490 trap 'rm -rf "$out"; rmdir "$lock" 2>/dev/null' EXIT24872491 rm -rf "$out"24882492 t0=$(date +%s)@@ -0,0 +1,46 @@
1+# A private repo's build writes its pages, and links its release files, in2+# a scratch directory under .forge/private, which only the forge can read,3+# never in .forge/build.*, so nothing of the repo is outside it even4+# mid-build; the swap into the private root keeps the links hard links.5+# A stand-in node first on PATH (11ty runs through env node) notes where6+# each build writes, and how its parent is kept, while it runs.7+. "$(dirname "$0")/../lib.sh"8+user dami9+repo --private dami/secret10+repo dami/open11+clone dami dami/secret "$T/w/d"12+at "$T/w/d" commit "first"13+at "$T/w/d" git tag v114+ok push_as dami "$T/w/d" origin main15+mkdir "$T/f"16+printf 'secret tool\n' >"$T/f/tool"17+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool"18+19+mkdir "$T/bin"20+cat >"$T/bin/node" <<EOF21+#!/bin/sh22+case \${1:-} in *eleventy*) printf '%s %s %s\n' "\${FORGE_PRIVATE:-0}" "\$FORGE_WWW" "\$(ls -ld "\$(dirname "\$FORGE_WWW")" | cut -c1-10)" >>"$T/builds" ;; esac23+exec "$(command -v node)" "\$@"24+EOF25+chmod +x "$T/bin/node"26+PATH=$T/bin:$PATH build dami/secret27+PATH=$T/bin:$PATH build dami/open28+29+eq "$(grep -c . "$T/builds")" 230+set -- $(sed -n 1p "$T/builds")31+eq "$1" 132+case $2 in "$FORGE_ROOT/.forge/private/build."[0-9]*) ;; *) fail "the private build wrote in $2" ;; esac33+eq "$3" drwx------34+missing "$2"35+set -- $(sed -n 2p "$T/builds")36+eq "$1" 037+case $2 in "$FORGE_ROOT/.forge/build."[0-9]*) ;; *) fail "the public build wrote in $2" ;; esac38+if find "$FORGE_ROOT/.forge" -maxdepth 2 -name 'build.[0-9]*' | grep -q .; then fail "a build left its scratch directory"; fi39+40+p=$FORGE_ROOT/.forge/private/www/dami/secret41+exists "$p/index.html"42+tree=$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse 'refs/releases/v1^{tree}')43+inode() { ls -i "$1" | awk '{ print $1 }'; }44+eq "$(inode "$p/releases/download/v1/tool")" "$(inode "$FORGE_ROOT/.forge/private/store/$tree/tool")"45+missing "$FORGE_WWW/dami/secret"46+exists "$FORGE_WWW/assets"