A private repo's release files were only to be had over ssh (release get). They now go where its pages are, the private root, with the same layout as www: post-receive links them there, the private build links them into the pages it swaps in as the public build does, and a restore puts them back there. Going private moves the downloads out of www into the private root at once, going public moves them back, so neither waits for the build or leaves a copy behind. gc drops the downloads of releases deleted behind the hooks' back in either root, and already kept the store trees that any repo's release refs name. The private server sends them as Caddy and demo/serve.ts send a public repo's: Content-Disposition: attachment and X-Content-Type-Options: nosniff, with ETag and Range, and Cache-Control: private, no-cache. The pattern and headers live in lib/static.ts for both servers. Anyone who may not read the repo gets the 404 of a missing file, so a token for the repo works with curl and nothing else does. The mirror syncs www only, so it never sees them. New scenarios: release-private-web (token and passkey session, attachment headers, ranges, the same 404 for anonymous users, non-readers and a token for another repo, not in www, not mirrored), release-private-flip (downloads move both ways and stay put through builds) and release-private-gc. release-private and release-restore follow. The README and the guide say what private repos' releases get now.
- commit
- c86d66765824bc8ea053e9460518308ff2769f5a
- parent
- 278dd7e23ea3bca6062e33b208671a8d6f324b52
- author
- cat <admin@dev.site>
- kind
- feat risk score 2.0 of 3, Jev's estimate
| M | README.md | 47 | ++++++++---- |
| M | bin/ | 115 | +++++++++++++++++++++--------- |
| M | demo/ | 6 | +- |
| M | guide/ | 2 | +- |
| M | lib/ | 8 | ++- |
| M | lib/ | 5 | + |
| A | test/ | 49 | +++++++++++++ |
| A | test/ | 41 | +++++++++++ |
| A | test/ | 82 | +++++++++++++++++++++ |
| M | test/ | 13 | ++-- |
| M | test/ | 5 | +- |
@@ -165,8 +165,8 @@ same answer, byte for byte, as for a repo that doesn't exist, however the
165165 path is spelled, and so does every page of it. It has no entry on the166166 index, owner pages or `repos.json`, its pages and live events go to its167167 readers only, and the pages mirror never sees it. Its sites and previews168-aren't served unless `git config forge.publicsites true`, and its releases169-aren't served publicly; readers get them over ssh (see168+aren't served unless `git config forge.publicsites true`. Its releases169+are, to its readers only, next to its pages, and over ssh (see170170 [Releases](#releases)). Pushes record reviews and print their links, which171171 work for its readers, and `forge reviews` reads them from the review logs172172 (without CI). The secret check still runs, since a secret stays in the@@ -183,14 +183,16 @@ ssh git@git.example.com ls # yours, and the private o
183183 The owner and collaborators flip a repo, the way Forgejo does. `private`184184 removes the flag, the repo's public pages, its entries on the index, owner185185 pages and `repos.json`, and its served sites, previews and their186-hostnames, and builds its pages again into the private root. Then it starts186+hostnames, moves its release downloads into the private root, and builds187+its pages again there. Then it starts187188 a mirror sync, which deletes them from the bucket. Clones, caches and copies188189 people already have can't be recalled. `public` first runs the secret check189190 over the whole history: every commit on a branch or tag, and the old review190191 revisions that review pages show, not only what a push adds. If it finds191192 something, the repo stays private unless you pass `--force`. Then it adds192-the flag, removes its private pages, links the sites and previews again193-from their refs, and rebuilds the pages into `www`. A flag added or removed193+the flag, removes its private pages, links the sites, previews and194+release downloads again from their refs, and rebuilds the pages into195+`www`. A flag added or removed194196 by hand moves the pages at the repo's next build, and `forge build` drops195197 any left in the wrong place.196198 @@ -207,7 +209,8 @@ serves a private repo's page to someone who may read it (the rule
207209 `can_read` in `bin/forge` applies, kept in step with `lib/access.ts` by a208210 scenario) and is still a user, signed in with a passkey or showing a token,209211 with the headers Caddy gives pages (ETag, 304s, byte ranges, which code210-search reads `code.db` with) except `Cache-Control: private, no-cache`.212+search reads `code.db` with) except `Cache-Control: private, no-cache`,213+and its release downloads as attachments, as Caddy sends a public repo's.211214 Not signed in, not allowed, or a page that isn't there: the same 404, byte212215 for byte, which every missing page of the forge gets, public or not.213216 `/private/` lists the private repos you can read.@@ -489,11 +492,19 @@ replace and `rm` delete them. Blobs over 16M are stored without deltas
489492 size and sha256, whether `SHA256SUMS.sig` verifies and whose key signed it,490493 and the CI result of the tag's commit.491494 492-A private repo's releases are served nowhere, even with493-`forge.publicsites`. Its readers get them over ssh, and to anyone else these494-commands answer as for a missing repo:495+A private repo's releases go where its pages are, the private root496+(`.forge/private/www`), with the same layout, never `www`, so the mirror497+doesn't see them and `forge.publicsites` doesn't change that. The private498+server gives them to its readers, signed in or showing a token (see499+[Private pages](#private-pages)) as Caddy gives a public repo's500+(attachments, uncompressed, byte ranges) but `Cache-Control: private,501+no-cache`, and anyone else gets the 404 of a file that isn't there. A repo502+that goes private takes its downloads there at once, and one that goes503+public brings them back to `www`. Readers can also get them over ssh, and to504+anyone else these commands answer as for a missing repo:495505 496506 ```sh507+curl -fO -u ana:<token> https://git.example.com/dami/notes/releases/download/v1.0/tool # token --repo dami/notes497508 ssh git@git.example.com release ls dami/notes498509 ssh git@git.example.com release get dami/notes v1.0 tool > tool # also SHA256SUMS, SHA256SUMS.sig499510 forge release verify --repo dami/notes --tag v1.0 tool@@ -766,12 +777,13 @@ push (log: `.forge/build.log`); `forge build` rebuilds everything.
766777 Every distinct deploy is extracted once into `.forge/store/<tree>/`, and the767778 old tree stays when a newer deploy takes its link. `forge gc` removes store768779 trees that no `www/sites` or `www/previews` link points at (hostnames point769-at those links) and no `refs/releases/*` ref names (release files are hard770-links, which name no tree), hostnames whose deploy is gone, previews of771-review branches that no longer exist, and the downloads of releases whose772-ref was deleted without the hooks. A deploy extracts its tree before it773-links it, so store entries changed in the last 10 minutes are kept (`--age774-<minutes>`), which makes gc safe to run while people push. `--dry-run` lists780+at those links) and no `refs/releases/*` ref names, a private repo's too781+(release files are hard links, which name no tree), hostnames whose deploy782+is gone, previews of review branches that no longer exist, and the783+downloads of releases whose ref was deleted without the hooks, in `www` or784+the private root. A deploy extracts its tree before it links it, and785+touches a tree it reuses, so store entries changed in the last 10 minutes786+are kept (`--age <minutes>`), which makes gc safe to run while people push. `--dry-run` lists775787 what it would remove. A release published again leaves its old files in the776788 repo's packs, unreachable: gc prunes nothing.777789 @@ -1017,8 +1029,9 @@ What you set up yourself, once:
10171029 `forge restore [<snapshot>|latest]` brings a forge back into an empty10181030 `/srv/git`. It restores the files, rebuilds what a backup leaves out (what10191031 `forge server init` and `forge keys sync` write, hooks in every repo, pinned1020-review revisions, extracted sites, public repos' release downloads, with1021-the mtimes they had, so the mirror uploads none again) and runs `forge1032+review revisions, extracted sites, release downloads, a private repo's in1033+the private root, with the mtimes they had, so the mirror uploads none1034+again) and runs `forge10221035 build`. identity.db is the backup's copy or litestream's replica, whichever10231036 changed last (see [above](#identitydb-backed-up-as-it-changes)). It refuses10241037 a forge that already has repos. The host's own `.forge/config` wins over the backup's,@@ -1106,9 +1106,10 @@ EOF
11061106 # they download rather than open: they are anyone's files, on this origin.11071107 #11081108 # A path with no file (or directory) in www goes to the private server:1109- # private repos' pages for their readers, which live outside www, and1110- # the 404 every other missing page gets (lib/private-server.ts). So1111- # Caddy needs no list of private repos.1109+ # private repos' pages for their readers, which live outside www, with1110+ # their release downloads (sent as attachments by that server, and left1111+ # uncompressed by @compress), and the 404 every other missing page gets1112+ # (lib/private-server.ts). So Caddy needs no list of private repos.11121113 hashed='^/assets/([^/]+\.)?[0-9A-Za-z_-]{10}\.[a-z]+$'11131114 cat <<EOF11141115 @public file {@@ -1665,14 +1666,22 @@ redeploy_sites() {
16651666 # from now on. Its pages go, the index, owner pages and repos.json are16661667 # rebuilt without it, its sites, previews and their hostnames go unless16671668 # forge.publicsites, and the mirror's next sync deletes them from the bucket.1668-# Its pages are built again into the private root, for its readers. The1669-# events server streams it only to them from now on (it checks the flag).1669+# Its pages are built again into the private root, for its readers, and its1670+# release downloads move there. The events server streams it only to them1671+# from now on (it checks the flag).16701672 sc_private() {16711673 [ $# -eq 1 ] || die "usage: private <owner>/<repo>"16721674 open_admin "$1"16731675 [ -f "$RDIR/git-daemon-export-ok" ] || { echo "$1 is already private"; return 0; }16741676 rm -f "$RDIR/git-daemon-export-ok"16751677 rm -rf "${FORGE_WWW:?}/$1"1678+ # Its release downloads go with its pages, and are in the private root1679+ # for its readers now rather than after the build.1680+ (1681+ export GIT_DIR="$RDIR"1682+ OWNER=${1%%/*} NAME=${1#*/}1683+ releases_into "$(pages_root "$RDIR")"1684+ ) || say "forge: linking the release files of $1 failed"16761685 if ! sites_public "$RDIR"; then16771686 for l in "$FORGE_WWW/sites/$1"/* "$FORGE_WWW/previews/$1"/*/*; do16781687 if [ -L "$l" ]; then deploy_site "${l#"$FORGE_WWW"/}" "$ZERO"; fi@@ -1690,9 +1699,10 @@ sc_private() {
16901699 }16911700 16921701 # public <owner>/<repo> [--force]: anyone can clone it and its pages come1693-# back, built into www; its private pages go. All of its history becomes public at once, so the secret check reads1694-# every commit on its branches and tags first, not only what a push adds,1695-# and refuses unless --force.1702+# back, built into www; its private pages go, and its release downloads1703+# move back to www. All of its history becomes public at once, so the1704+# secret check reads every commit on its branches and tags first, not only1705+# what a push adds, and refuses unless --force.16961706 sc_public() {16971707 force=016981708 case $#:${2:-} in 1:) ;; 2:--force) force=1 ;; *) die "usage: public <owner>/<repo> [--force]" ;; esac@@ -1717,6 +1727,7 @@ sc_public() {
17171727 export GIT_DIR="$RDIR"17181728 OWNER=${1%%/*} NAME=${1#*/}17191729 redeploy_sites1730+ releases_into "$FORGE_WWW"17201731 )17211732 rebuild "$1" public17221733 echo "$1 is public: anyone can clone it, and its pages and sites are back once the build finishes."@@ -1753,8 +1764,9 @@ sc_readers() {
17531764 }17541765 17551766 # release ls <owner>/<repo> | release get <owner>/<repo> <tag> <file>: for1756-# anyone who may read the repo, so a private repo's releases, which nothing1757-# serves, are there for its readers; to anyone else it is a missing repo.1767+# anyone who may read the repo, so a private repo's releases, which the web1768+# gives its readers only once they sign in or show a token, are there for1769+# them over ssh too; to anyone else it is a missing repo.17581770 # ls prints tag, file, size and sha256 (from SHA256SUMS), newest first.17591771 sc_release() {17601772 case ${1:-}:$# in@@ -2151,7 +2163,7 @@ store_tree() {
21512163 # a release, extracted once into the store, or remove it. Hard links, not a21522164 # symlink like a site's: rclone (forge mirror) skips symlinks, and the files21532165 # keep the store's mtime. Runs in a repo (GIT_DIR) with OWNER and NAME set;2154-# <root> is FORGE_WWW for a public repo.2166+# <root> is the repo's pages root (pages_root) or a build's output.21552167 release_link() {21562168 rl=$1/$OWNER/$NAME/releases/download/$(enc_ref "$2")21572169 if [ "$3" = "$ZERO" ]; then@@ -2195,17 +2207,39 @@ releases_into() {
21952207 release_latest "$1"21962208 }21972209 2210+# private_www: the private root, made if need be, with its parent for git2211+# alone; only the private server reads it.2212+private_www() {2213+ mkdir -p "$FORGE_PRIVATE_WWW"2214+ chmod 700 "$(dirname "$FORGE_PRIVATE_WWW")"2215+}2216+2217+# pages_root <git-dir>: where the repo's pages and release downloads live:2218+# www for a public repo, the private root for a private one.2219+pages_root() {2220+ if [ -f "$1/git-daemon-export-ok" ]; then2221+ printf '%s' "$FORGE_WWW"2222+ else2223+ private_www2224+ printf '%s' "$FORGE_PRIVATE_WWW"2225+ fi2226+}2227+21982228 # push_release <tag> <commit|ZERO>: after a push, serve the release's files,2199-# or stop. A private repo's are served nowhere; its readers get them over2200-# ssh (sc_release).2229+# or stop. A public repo's are anyone's, from www; a private repo's go to2230+# the private root, where the private server gives them to its readers,2231+# signed in or with a token, and readers can get them over ssh too2232+# (sc_release).22012233 push_release() {2202- if [ ! -f "$GIT_DIR/git-daemon-export-ok" ]; then2203- [ "$2" = "$ZERO" ] || say "release $1: not served, since $OWNER/$NAME is private; its readers get the files with ssh ${FORGE_SSH:-git@host} release get $OWNER/$NAME $1 <file>"2204- return 02205- fi2206- release_link "$FORGE_WWW" "$1" "$2"2234+ release_link "$(pages_root "$GIT_DIR")" "$1" "$2"2235+ [ "$2" != "$ZERO" ] || return 022072236 u=${FORGE_BASE_URL:-}2208- [ "$2" = "$ZERO" ] || say "release $1: ${u%/}/$OWNER/$NAME/releases/download/$(enc_ref "$1")/"2237+ at=${u%/}/$OWNER/$NAME/releases/download/$(enc_ref "$1")/2238+ if [ -f "$GIT_DIR/git-daemon-export-ok" ]; then2239+ say "release $1: $at"2240+ else2241+ say "release $1: $at for its readers, signed in or with a token (ssh ${FORGE_SSH:-git@host} release get $OWNER/$NAME $1 <file> works too)"2242+ fi22092243 }22102244 22112245 hook_post_receive() {@@ -2262,7 +2296,7 @@ noreview
22622296 $updates22632297 EOF22642298 # A new tag can be the newest with a release, or a deleted one can stop being it.2265- if [ "$rel" = 1 ] && [ -f "$GIT_DIR/git-daemon-export-ok" ]; then release_latest "$FORGE_WWW"; fi2299+ if [ "$rel" = 1 ]; then release_latest "$(pages_root "$GIT_DIR")"; fi22662300 git update-server-info22672301 refs=$(printf '%s' "$PUSHED" | tr -d '"\\')22682302 printf '{"t":%s000,"repo":"%s","type":"push","user":"%s","refs":"%s"%s}\n' "${FORGE_NOW:-$(date +%s)}" "$OWNER/$NAME" "${USER_:-admin}" "$refs" "$(ev_private "$GIT_DIR")" >>"$FORGE_STATE/events.jsonl"@@ -2350,13 +2384,18 @@ build_private() {
23502384 rm -rf "$out"23512385 t0=$(date +%s)23522386 write_graph "$FORGE_ROOT/$only.git"2353- mkdir -p "$FORGE_PRIVATE_WWW"2354- chmod 700 "$(dirname "$FORGE_PRIVATE_WWW")"2387+ private_www23552388 (cd "$FORGE_HOME" && FORGE_PRIVATE=1 FORGE_ONLY=$only FORGE_WWW=$out FORGE_LIVE_WWW=$FORGE_PRIVATE_WWW FORGE_PUBLIC_WWW=$FORGE_WWW \23562389 FORGE_BUILD_STARTED=$((t0 * 1000)) FORGE_BUILD_REASON=${FORGE_BUILD_REASON:-manual} \23572390 "$FORGE_HOME/node_modules/.bin/eleventy" --config=eleventy.config.ts --quiet --output="$out")23582391 mkdir -p "$FORGE_WWW"23592392 swap "$out/assets" "$FORGE_WWW/assets"2393+ # Its release downloads, as a public build links them (cmd_build), for2394+ # the private server to give its readers.2395+ if [ -d "$out/$only" ]; then2396+ (trap - EXIT && export GIT_DIR="$FORGE_ROOT/$only.git" && OWNER=${only%%/*} NAME=${only#*/} && releases_into "$out") ||2397+ say "forge: linking the release files of $only failed"2398+ fi23602399 # Made public while this was building: no private pages after all.23612400 if [ -d "$out/$only" ] && [ ! -f "$FORGE_ROOT/$only.git/git-daemon-export-ok" ]; then swap "$out/$only" "$FORGE_PRIVATE_WWW/$only"23622401 else rm -rf "${FORGE_PRIVATE_WWW:?}/$only"; fi@@ -2521,17 +2560,22 @@ cmd_gc() {
25212560 echo "$did preview $OWNER/$NAME ${p#refs/site-previews/}"25222561 done25232562 # Downloads of releases whose ref went outside the hooks (hard2524- # links, so their store tree's space comes back only with them).2525- gone=02526- for w in "$FORGE_WWW/$OWNER/$NAME/releases/download"/*/; do2527- [ -d "$w" ] || continue2528- t=$(basename "$w" | tr '~' /)2529- [ "$t" != latest ] && ! git rev-parse -q --verify "refs/releases/$t" >/dev/null || continue2530- [ "$dry" = 1 ] || rm -rf "$w"2531- echo "$did release $OWNER/$NAME $t"2532- gone=12563+ # links, so their store tree's space comes back only with them),2564+ # in www or the private root; latest/ follows in the repo's own.2565+ own=$FORGE_WWW2566+ [ -f "$GIT_DIR/git-daemon-export-ok" ] || own=$FORGE_PRIVATE_WWW2567+ for root in "$FORGE_WWW" "$FORGE_PRIVATE_WWW"; do2568+ gone=02569+ for w in "$root/$OWNER/$NAME/releases/download"/*/; do2570+ [ -d "$w" ] || continue2571+ t=$(basename "$w" | tr '~' /)2572+ [ "$t" != latest ] && ! git rev-parse -q --verify "refs/releases/$t" >/dev/null || continue2573+ [ "$dry" = 1 ] || rm -rf "$w"2574+ echo "$did release $OWNER/$NAME $t"2575+ gone=12576+ done2577+ if [ "$gone" = 1 ] && [ "$dry" = 0 ] && [ "$root" = "$own" ]; then release_latest "$root"; fi25332578 done2534- if [ "$gone" = 1 ] && [ "$dry" = 0 ]; then release_latest "$FORGE_WWW"; fi25352579 unset GIT_DIR25362580 done25372581 if [ -d "$FORGE_WWW/hosts" ]; then@@ -3028,9 +3072,10 @@ restore_finish() {
30283072 awk '$2 == "commit" { print "update refs/forge/keep/" $1 " " $1 }' | git update-ref --stdin30293073 # A private repo's sites stay unserved (sites_public), as before the backup.30303074 if sites_public "$GIT_DIR"; then redeploy_sites; fi3031- # Releases are refs too; a private repo's are served nowhere. The files3032- # get their commit's time again, so the mirror sees nothing new.3033- if [ -f "$GIT_DIR/git-daemon-export-ok" ]; then releases_into "$FORGE_WWW"; fi3075+ # Releases are refs too: a public repo's go to www, a private repo's to3076+ # the private root. The files get their commit's time again, so the3077+ # mirror sees nothing new.3078+ releases_into "$(pages_root "$GIT_DIR")"30343079 git update-server-info30353080 unset GIT_DIR30363081 done@@ -11,7 +11,7 @@ import path from 'node:path';
1111 import { fileURLToPath } from 'node:url';1212 import { serveEvents } from '../lib/events-server.ts';1313 import { privateHandler } from '../lib/private-server.ts';14-import { HASHED, resolve, sendFile } from '../lib/static.ts';14+import { DOWNLOAD, DOWNLOAD_HEADERS, HASHED, resolve, sendFile } from '../lib/static.ts';1515 1616 const here = path.dirname(fileURLToPath(import.meta.url));1717 const state = process.env.FORGE_STATE || path.join(here, 'root/.forge');@@ -29,9 +29,7 @@ const misses = privateHandler({
2929 const cacheControl = (p: string): Record<string, string> =>3030 HASHED.test(p) ? { 'Cache-Control': 'public, max-age=31536000, immutable' } : /^\/(sites|previews)\//.test(p) ? {} : { 'Cache-Control': 'no-cache' };3131 // Release downloads are saved, not opened, on the forge's origin.32-const DOWNLOAD = /^\/[^/]+\/[^/]+\/releases\/download\/./;33-const headersFor = (p: string): Record<string, string> =>34- DOWNLOAD.test(p) ? { ...cacheControl(p), 'Content-Disposition': 'attachment', 'X-Content-Type-Options': 'nosniff' } : cacheControl(p);32+const headersFor = (p: string): Record<string, string> => (DOWNLOAD.test(p) ? { ...cacheControl(p), ...DOWNLOAD_HEADERS } : cacheControl(p));3533 3634 http.createServer((req, res) => {3735 const url = new URL(req.url ?? '/', 'http://x');@@ -215,7 +215,7 @@ log sha256:6cc79019... 4 lines
215215 <p>The pages server serves them the way Cloudflare Pages does, so a site built for there works unchanged: <code>_headers</code> and <code>_redirects</code> apply, <code>/about</code> finds <code>about.html</code> or <code>about/index.html</code>, a missing page gets the nearest <code>404.html</code> (or <code>index.html</code>, for a single-page app), and previews are sent with <code>noindex</code>. Old <code>/sites/...</code> paths redirect to the hostname. Without a pages domain, sites are plain files under <code>/sites/<owner>/<repo>/<name>/</code> on the forge itself.</p>216216 <p>Deleting a branch deletes its previews: the ref, the served link and the hostname, so the sites and review pages stop listing them. Previews of merged branches stay until their branch is deleted. The extracted files stay in the store until <code>forge gc</code>, which removes store trees nothing links to, hostnames whose deploy is gone and leftover previews, skipping store entries changed in the last ten minutes, so it can run while people push.</p>217217 <h3>Releases are the same idea</h3>218-<p><code>forge release publish v1.0 <file>...</code> commits a tag's files to <code>refs/releases/v1.0</code> with a <code>SHA256SUMS</code> signed by your key for that repo and tag (<code>ssh-keygen -n forge-release</code>), so it can't pass for another release. <code>pre-receive</code> checks every sum, the signature (an owner's or collaborator's key), the tag and a size cap, and with <code>forge.releaserequireci</code> a verified pass for the tag's tree. <code>post-receive</code> extracts the tree into the store and hard-links the files into <code>/dami/greet/releases/download/v1.0/</code>, plus <code>latest/</code>; hard links, so the mirror copies them. Clones and fetches never see <code>refs/releases</code>. A private repo's releases aren't served; its readers get them with <code>ssh git@host release get</code>.</p>218+<p><code>forge release publish v1.0 <file>...</code> commits a tag's files to <code>refs/releases/v1.0</code> with a <code>SHA256SUMS</code> signed by your key for that repo and tag (<code>ssh-keygen -n forge-release</code>), so it can't pass for another release. <code>pre-receive</code> checks every sum, the signature (an owner's or collaborator's key), the tag and a size cap, and with <code>forge.releaserequireci</code> a verified pass for the tag's tree. <code>post-receive</code> extracts the tree into the store and hard-links the files into <code>/dami/greet/releases/download/v1.0/</code>, plus <code>latest/</code>; hard links, so the mirror copies them. Clones and fetches never see <code>refs/releases</code>. A private repo's go to the private root next to its pages, never the mirror, and its readers download them signed in or with a token, or get them with <code>ssh git@host release get</code>.</p>219219 </div>220220 </section>221221 @@ -5,6 +5,8 @@
55 // build_private), with the same layout and the same headers as Caddy gives66 // pages (ETag, 304s, byte ranges for code.db) except Cache-Control, which77 // is "private, no-cache": the same revalidation, kept out of shared caches.8+// Its release downloads (/<owner>/<repo>/releases/download/...) are sent9+// as attachments, as Caddy sends a public repo's.810 //911 // A reader is the repo's owner, a collaborator or a name in its readers1012 // (lib/access.ts canRead, the rule bin/forge applies) who is still a user of@@ -28,7 +30,7 @@ import path from 'node:path';
2830 import { fileURLToPath } from 'node:url';2931 import { Identity, SESSION_TTL, TOKEN_PREFIX, cookieValue, csrfToken, sameSecret, sessionCookie, type Session, type Token } from './identity.ts';3032 import { canRead, isPublic, readable, repoDir, users } from './access.ts';31-import { resolve, sendFile } from './static.ts';33+import { DOWNLOAD, DOWNLOAD_HEADERS, resolve, sendFile } from './static.ts';3234 import { Login, LoginError, agentName, readBody, relyingParty } from './login.ts';3335 import { pageRenderer, type Render } from './render-page.ts';3436 import { assets } from './assets.ts';@@ -177,7 +179,9 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
177179 res.writeHead(308, { Location: url.pathname + '/' + url.search, ...cookie(who) }).end();178180 return;179181 }180- sendFile(req, res, file, { 'Cache-Control': 'private, no-cache', ...cookie(who) });182+ // Release downloads as Caddy sends a public repo's: attachments, never183+ // sniffed, in byte ranges if asked.184+ sendFile(req, res, file, { 'Cache-Control': 'private, no-cache', ...(DOWNLOAD.test(url.pathname) ? DOWNLOAD_HEADERS : {}), ...cookie(who) });181185 }182186 183187 // An HTTPS clone or fetch of a repo without git-daemon-export-ok, asked by@@ -16,6 +16,11 @@ export const TYPES: Record<string, string> = {
1616 // Hashed assets never change; pages and the indexes they fetch revalidate.1717 export const HASHED = /^\/assets\/([^/]+\.)?[0-9A-Za-z_-]{10}\.[a-z]+$/;1818 19+// Release downloads are saved, not opened, on the forge's origin: they are20+// anyone's files, so they never render as a page of it (Caddy sends the same).21+export const DOWNLOAD = /^\/[^/]+\/[^/]+\/releases\/download\/./;22+export const DOWNLOAD_HEADERS = { 'Content-Disposition': 'attachment', 'X-Content-Type-Options': 'nosniff' };23+1924 // One byte range, as code search reads code.db: [start, end] inclusive, or2025 // null for a malformed or unsatisfiable one. Several ranges get the whole file.2126 export function byteRange(h: string, size: number): [number, number] | null | undefined {@@ -0,0 +1,49 @@
1+# Release downloads follow the repo's pages when it flips: private moves2+# them out of www into the private root at once, public moves them back,3+# and neither leaves a copy behind, nor does a build afterwards.4+. "$(dirname "$0")/../lib.sh"5+user dami6+repo dami/greet7+clone dami dami/greet "$T/w/d"8+at "$T/w/d" commit "first"9+at "$T/w/d" git tag v110+ok push_as dami "$T/w/d" origin main11+mkdir "$T/f"12+printf 'greet 1\n' >"$T/f/greet"13+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/greet"14+build15+w=$FORGE_WWW/dami/greet/releases/download16+p=$FORGE_ROOT/.forge/private/www/dami/greet/releases/download17+exists "$w/v1/greet"18+exists "$w/latest/greet"19+missing "$p"20+21+ok ssh_as dami private dami/greet22+missing "$FORGE_WWW/dami/greet"23+ok cmp "$T/f/greet" "$p/v1/greet"24+ok cmp "$T/f/greet" "$p/latest/greet"25+build26+missing "$FORGE_WWW/dami/greet"27+exists "$p/v1/greet"28+exists "$p/../index.html"29+30+# A release published while private goes to the private root only.31+tick 360032+at "$T/w/d" commit "second"33+at "$T/w/d" git tag v234+ok push_as dami "$T/w/d" origin main35+printf 'greet 2\n' >"$T/f/greet"36+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v2 "$T/f/greet"37+has "$p/v2/greet" 'greet 2'38+has "$p/latest/greet" 'greet 2'39+missing "$FORGE_WWW/dami/greet"40+41+ok ssh_as dami public dami/greet42+missing "$FORGE_ROOT/.forge/private/www/dami/greet"43+has "$w/v1/greet" 'greet 1'44+has "$w/v2/greet" 'greet 2'45+has "$w/latest/greet" 'greet 2'46+build47+missing "$FORGE_ROOT/.forge/private/www/dami/greet"48+has "$w/latest/greet" 'greet 2'49+exists "$w/../index.html"@@ -0,0 +1,41 @@
1+# forge gc keeps the store tree of a private repo's release, whose files2+# are hard links in the private root, and drops the downloads of a release3+# whose ref went behind the hooks' back there as in www, freeing its tree.4+. "$(dirname "$0")/../lib.sh"5+user dami6+repo --private dami/secret7+clone dami dami/secret "$T/w/d"8+mkdir "$T/f"9+export FORGE_USER=dami10+g=$FORGE_ROOT/dami/secret.git11+for v in 1 2; do12+ tick 360013+ at "$T/w/d" commit "v$v"14+ at "$T/w/d" git tag "v$v"15+ ok push_as dami "$T/w/d" origin main16+ printf 'secret %s\n' "$v" >"$T/f/tool"17+ ok at "$T/w/d" "$FORGE" release publish "v$v" "$T/f/tool"18+ eval "tree$v=$(git --git-dir="$g" rev-parse "refs/releases/v$v^{tree}")"19+done20+unset FORGE_USER21+build22+store=$FORGE_ROOT/.forge/store23+dl=$FORGE_ROOT/.forge/private/www/dami/secret/releases/download24+exists "$store/$tree1/tool"25+exists "$store/$tree2/tool"26+has "$dl/latest/tool" 'secret 2'27+28+ok forge gc --age 029+lacks "$T_LOG" 'removed (release|store)'30+exists "$store/$tree1/tool"31+exists "$store/$tree2/tool"32+33+git --git-dir="$g" update-ref -d refs/releases/v234+ok forge gc --age 035+has "$T_LOG" '^removed release dami/secret v2$'36+has "$T_LOG" "^removed store $tree2 "37+lacks "$T_LOG" "$tree1"38+missing "$dl/v2"39+has "$dl/v1/tool" 'secret 1'40+has "$dl/latest/tool" 'secret 1'41+missing "$FORGE_WWW/dami/secret"@@ -0,0 +1,82 @@
1+# A private repo's release files are in the private root, and the private2+# server gives them to its readers, with a token (curl, a token for that3+# repo) or signed in, as Caddy gives a public repo's: attachments, not4+# sniffed, revalidated, in byte ranges. Anyone else gets the 404 of a file5+# that isn't there, byte for byte. They are not in www, so the mirror,6+# which syncs www, never sees them.7+. "$(dirname "$0")/../lib.sh"8+user dami9+user ana10+user cat11+repo --private dami/secret12+repo dami/open13+ok ssh_as dami readers dami/secret add ana14+clone dami dami/secret "$T/w/d"15+at "$T/w/d" commit "first"16+at "$T/w/d" git tag v117+ok push_as dami "$T/w/d" origin main18+mkdir "$T/f"19+printf 'secret tool, 0123456789\n' >"$T/f/tool.html"20+FORGE_USER=dami ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool.html"21+build22+p=$FORGE_ROOT/.forge/private/www/dami/secret23+ok cmp "$T/f/tool.html" "$p/releases/download/v1/tool.html"24+ok cmp "$T/f/tool.html" "$p/releases/download/latest/tool.html"25+has "$p/releases/index.html" 'href="/dami/secret/releases/download/v1/tool.html"'26+missing "$FORGE_WWW/dami/secret"27+if find "$FORGE_WWW" -name 'tool.html' | grep -q .; then fail "a private release file is in www"; fi28+29+export FORGE_BASE_URL=https://forge.test30+private_server31+u=$PRIV/dami/secret/releases/download/v1/tool.html32+tok=$(ssh_as ana token --repo dami/secret 2>/dev/null)33+curl -s -D "$T/h" -o "$T/got" -u "ana:$tok" "$u"34+has "$T/h" '^HTTP/1.1 200'35+has "$T/h" '^Content-Disposition: attachment'36+has "$T/h" '^X-Content-Type-Options: nosniff'37+has "$T/h" '^Cache-Control: private, no-cache'38+has "$T/h" '^Accept-Ranges: bytes'39+ok cmp "$T/f/tool.html" "$T/got"40+curl -s -D "$T/h" -o "$T/part" -H "Authorization: Bearer $tok" -H 'Range: bytes=7-10' "$u"41+has "$T/h" '^HTTP/1.1 206'42+has "$T/h" "^Content-Range: bytes 7-10/$(wc -c <"$T/f/tool.html" | tr -d ' ')"43+has "$T/h" '^Content-Disposition: attachment'44+eq "$(cat "$T/part")" "tool"45+etag=$(sed -n 's/^ETag: //p' "$T/h" | tr -d '\r')46+eq "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $tok" -H "If-None-Match: $etag" "$u")" 30447+eq "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $tok" "$PRIV/dami/secret/releases/download/v1/SHA256SUMS.sig")" 20048+49+# Signed in with a passkey.50+code=$(ssh_as ana passkey | sed -n 's/^ .*#enroll=//p')51+ana=$(node "$T_HOME/test/authenticator.ts" enroll "$PRIV" "$code" "$T/ana.key") || fail "ana can't sign in"52+curl -s -D "$T/h" -o "$T/got" -H "Cookie: __Host-forge=$ana" -H 'Range: bytes=0-5' "$u"53+has "$T/h" '^HTTP/1.1 206'54+has "$T/h" '^Content-Disposition: attachment'55+has "$T/h" '^X-Content-Type-Options: nosniff'56+eq "$(cat "$T/got")" "secret"57+58+# Anyone else: the missing file's 404.59+response "$PRIV/dami/secret/releases/download/v1/nope" -H "Authorization: Bearer $tok" >"$T/missing"60+has "$T/missing" '^HTTP/1.1 404'61+cat_tok=$(ssh_as cat token 2>/dev/null)62+code=$(ssh_as cat passkey | sed -n 's/^ .*#enroll=//p')63+cat=$(node "$T_HOME/test/authenticator.ts" enroll "$PRIV" "$code" "$T/cat.key") || fail "cat can't sign in"64+elsewhere=$(ssh_as ana token --repo dami/open 2>/dev/null)65+same() {66+ response "$u" "$@" >"$T/got"67+ cmp -s "$T/missing" "$T/got" || { diff "$T/missing" "$T/got" >&2 || true; fail "not the missing file's 404: $*"; }68+}69+same70+same -H 'Range: bytes=0-5'71+same -H "Authorization: Bearer $cat_tok"72+same -u "cat:$cat_tok"73+same -H "Cookie: __Host-forge=$cat"74+same -H "Authorization: Bearer $elsewhere"75+76+# The mirror (a stand-in rclone that lists what it was given) sees none of it.77+mkdir "$T/bin"78+printf '#!/bin/sh\nfind "$2" >>"%s/rclone.saw"\n' "$T" >"$T/bin/rclone"79+chmod +x "$T/bin/rclone"80+PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages ok forge mirror81+has "$T/rclone.saw" '/index\.html$'82+lacks "$T/rclone.saw" 'secret|tool'@@ -1,7 +1,8 @@
1-# A private repo's releases are served nowhere, but its readers get them2-# over ssh (release ls, release get); to anyone else those commands say what3-# they say about a repo that isn't there. A repo made private loses its4-# public downloads.1+# A private repo's releases aren't in www: they go to the private root, for2+# its readers on the web (release-private-web), and its readers get them3+# over ssh as well (release ls, release get); to anyone else those commands4+# say what they say about a repo that isn't there. A repo made private5+# loses its public downloads.56 . "$(dirname "$0")/../lib.sh"67 user dami78 user ana@@ -16,10 +17,10 @@ mkdir "$T/f"
1617 printf 'secret\000binary\n' >"$T/f/tool"1718 export FORGE_USER=dami1819 ok at "$T/w/d" "$FORGE" release publish v1 "$T/f/tool"19-has "$T_LOG" 'release v1: not served, since dami/secret is private; its readers get the files with ssh git@forge.test release get dami/secret v1 <file>'20+has "$T_LOG" 'release v1: /dami/secret/releases/download/v1/ for its readers, signed in or with a token \(ssh git@forge.test release get dami/secret v1 <file> works too\)'2021 unset FORGE_USER2122 missing "$FORGE_WWW/dami/secret"22-eq "$(ls "$FORGE_ROOT/.forge/store" | wc -l | tr -d ' ')" 023+ok cmp "$T/f/tool" "$FORGE_ROOT/.forge/private/www/dami/secret/releases/download/v1/tool"2324 2425 ssh_as ana release get dami/secret v1 tool >"$T/got" 2>"$T_LOG" || fail "a reader can't get it"2526 ok cmp "$T/f/tool" "$T/got"@@ -1,6 +1,7 @@
11 # Releases are refs, so a backup holds them with nothing more, and a22 # restore links a public repo's downloads again, the same bytes with the3-# same mtimes (so the mirror uploads nothing); a private repo's stay unserved.3+# same mtimes (so the mirror uploads nothing), and a private repo's into4+# the private root, for its readers, never www.45 . "$(dirname "$0")/../lib.sh"56 user dami67 repo dami/open@@ -31,4 +32,6 @@ exists "$r/.forge/www/dami/open/releases/download/latest/tool"
3132 exists "$r/.forge/www/dami/open/releases/index.html"3233 eq "$(mtime "$r/.forge/www/dami/open/$dl/tool")" "$was"3334 missing "$r/.forge/www/dami/secret"35+ok cmp "$T/f/tool" "$r/.forge/private/www/dami/secret/$dl/tool"36+exists "$r/.forge/private/www/dami/secret/releases/download/latest/tool"3437 eq "$(git --git-dir="$r/dami/secret.git" rev-parse refs/releases/v1)" "$(git --git-dir="$FORGE_ROOT/dami/secret.git" rev-parse refs/releases/v1)"