A Releases section after Sites says what a release is, what publish and verify do, what pre-receive checks, where the files are served and how, what clones see, how a private repo's readers get them, and what a verified download proves and doesn't. Cleaning up, the repo settings, the pages mirror's two passes, what a backup holds and what a restore rebuilds mention releases; so do the URL table, the intro and the demo list. The guide gets a paragraph next to sites, the refs table row and a cheat sheet entry.
- commit
- ccae97705e83ea43f50be202780cf21959dc271e
- parent
- 2e295bf7e3d59582e85d79f5e4f160c6ecf2ee95
- author
- cat <admin@dev.site>
- kind
- docs risk score 0.2 of 3, Jev's estimate
| M | README.md | 148 | ++++++++++++++++++++++++++---- |
| M | guide/ | 15 | ++- |
@@ -32,6 +32,9 @@ It adds what stagit doesn't have, borrowing from [args.io](https://args.io):
3232 - **Sites ship with repos.** Docs, Docusaurus builds, or anything static is3333 committed to `refs/sites/<name>` and served from the forge. Each review3434 branch gets its own preview. The wiki is markdown in `wiki/`.35+- **Releases are signed files.** `forge release publish v1.0 <file>...`36+ puts a tag's binaries on `refs/releases/v1.0` with a `SHA256SUMS` signed37+ by your key; the forge checks both and serves the files.3538 - **Keyboard navigation and clean URLs**, e.g. `/dami/greet/tree/main/greet.sh/`.3639 - **Search and syntax highlighting.** `/` searches a repo's commits, reviews,3740 wiki and file paths from a static index, and file contents from a SQLite@@ -58,6 +61,7 @@ The demo creates two users (`dami`, `ana`) and a repo with:
5861 turns away because nobody approved it;5962 - a draft that grows, then is rebased onto a newer `main` and reworked;6063 - an abandoned branch, a wiki, and a docs site with a preview;64+- a release of `v0.1`: the script and a source tarball;6165 - a standalone site that `ana` owns;6266 - code in JavaScript, Go, CSS, Python and markdown fences, and a review of a6367 1200-line file with long lines.@@ -74,7 +78,8 @@ git push ─────────── ssh, forced command ────> for
7478 forge attest -> refs/notes/ci (signed) pre-receive: who may push what; CI and7579 approval gates on main7680 forge site deploy -> refs/sites/<name> post-receive: record review revisions,77- refs/site-previews/... extract sites, rebuild pages81+ refs/site-previews/... extract sites and releases, rebuild pages82+forge release publish -> refs/releases/<tag>7883 11ty build -> /srv/git/.forge/www (static)7984 events.jsonl -> live page updates8085 forge comment, approve ─── ssh ──────────────> comment, approve: next to the review log@@ -99,7 +104,9 @@ config for either setup. It serves assets, whose names carry a hash of their
99104 contents, with `Cache-Control: public, max-age=31536000, immutable`, and100105 pages and the JSON and text indexes they fetch with `no-cache`, so browsers101106 revalidate them with the ETag and Last-Modified Caddy sends and get a 304102-when nothing changed. `demo/serve.ts` sends the same headers.107+when nothing changed. Release downloads revalidate the same way, aren't108+compressed, and are sent as attachments (see [Releases](#releases)).109+`demo/serve.ts` sends the same headers.103110 104111 ### Identity and permissions105112 @@ -108,8 +115,9 @@ source of identity. `forge keys sync` generates two files from it:
108115 109116 - `authorized_keys`, which forces every key through `forge shell <name>`, so110117 hooks know who pushed;111-- `allowed_signers`, so the same keys verify CI attestations and signed site112- deploys.118+- `allowed_signers`, so the same keys verify CI attestations, signed site119+ deploys and release checksums (namespaces `forge-ci`, `forge-release` and120+ `git`).113121 114122 Repos live at `/srv/git/<owner>/<repo>.git`, and `<owner>` owns them. Anyone115123 can read a public repo; a private one, only the people below. The owner and@@ -145,7 +153,8 @@ clone, fetch, archive, push or ssh command gets the same answer, byte for
145153 byte, as for a repo that doesn't exist, however the path is spelled. It has146154 no pages, no entry on the index, owner pages or `repos.json`, and no live147155 events. Its sites and previews aren't served unless `git config148-forge.publicsites true`. Pushes still record reviews, and `forge reviews`156+forge.publicsites true`, and its releases aren't served at all; readers get157+them over ssh (see [Releases](#releases)). Pushes still record reviews, and `forge reviews`149158 reads them from the review logs (without CI). The secret check still runs,150159 since a secret stays in the history when the repo goes public later.151160 @@ -322,6 +331,81 @@ so the sites and review pages stop listing it. Previews of merged branches
322331 stay until their branch is deleted. Their extracted files stay in the store323332 until `forge gc` (below).324333 334+### Releases335+336+A release is a commit on `refs/releases/<tag>` whose tree is the release's337+files plus `SHA256SUMS` and `SHA256SUMS.sig`, the way a site deploy is a338+commit whose tree is the built output.339+340+```sh341+git tag v1.0342+forge release publish v1.0 dist/greet-linux-amd64 dist/greet-1.0.tar.gz343+forge release verify ~/Downloads/greet-linux-amd64 # SHA256SUMS and its .sig next to it344+forge release ls # tag, file, size, sha256345+forge release rm v1.0346+```347+348+`publish` writes `SHA256SUMS` in `sha256sum`'s format, signs it with349+`ssh-keygen -Y sign -n forge-release` using the key that signs CI results,350+and makes a commit with trailers `Tag`, `Source-Commit` (the tag's commit)351+and `Built-By`, signed like a site deploy. It pushes the commit, the tag and352+your CI notes in one atomic push. The commit has no parent, so publishing353+under the tag again replaces the files and the old ones aren't kept354+reachable. `verify` checks files against `SHA256SUMS` and its signature355+against the forge's keys: `--signers <file>`, or `ssh <host> signers` with356+`--host`, `forge.allowedsigners`, `forge.host` or the remote.357+358+`pre-receive` takes a release from the owner and collaborators only, like a359+tag, and only if:360+361+- the tag exists once the push lands (it may come in the same push), and the362+ trailers name it and its commit;363+- the tree is plain files at the top level, named with letters, digits and364+ `. _ + -`, none over `git config forge.releasemaxsize` (256m unless set;365+ git's k, m and g work);366+- `SHA256SUMS` lists every file once with its sha256, and nothing else;367+- `SHA256SUMS.sig` is a `forge-release` signature of it by any key in the368+ forge's keys file;369+- with `git config forge.releaserequireci true`, the tag's tree has a370+ verified passing attestation, as `forge.requireci` asks of `main`371+ (`forge attest v1.0` before publishing).372+373+The secret check skips releases. It reads branches and tags, and a374+release's source is its tag.375+376+`post-receive` extracts the tree once into `.forge/store/<tree>/`, as for a377+site, and hard-links its files into378+`/<owner>/<repo>/releases/download/<tag>/` (a `/` in the tag becomes `~`).379+`latest/` next to them holds the files of the newest tag that has a380+release, so `/dami/greet/releases/download/latest/greet-linux-amd64` keeps381+working from one release to the next. Hard links, not a symlink as for a382+site: rclone skips symlinks, and the [mirror](#the-pages-mirror) copies383+these. The page build links them into the repo's new pages before it swaps384+them in. Caddy serves them with `no-cache`, since a tag can be published385+again, uncompressed, and with `Content-Disposition: attachment`, since they386+are anyone's files on the forge's own origin. Deleting the ref removes them.387+388+Clones and fetches don't see `refs/releases` (`uploadpack.hideRefs`), so389+nobody downloads binaries with the source; pushes do, so `publish` can390+replace and `rm` delete them. Blobs over 16M are stored without deltas391+(`core.bigFileThreshold`). The releases page lists each tag's files, with392+size and sha256, whether `SHA256SUMS.sig` verifies and whose key signed it,393+and the CI result of the tag's commit.394+395+A private repo's releases are served nowhere, even with396+`forge.publicsites`. Its readers get them over ssh, and to anyone else these397+commands answer as for a missing repo:398+399+```sh400+ssh git@git.example.com release ls dami/notes401+ssh git@git.example.com release get dami/notes v1.0 tool > tool # also SHA256SUMS, SHA256SUMS.sig402+```403+404+A verified download proves what a verified attestation does, about these405+bytes: the holder of a registered key vouched for this `SHA256SUMS`. It406+doesn't say for which tag; an older release's files verify with their own407+`SHA256SUMS` too.408+325409 ### Search and highlighting326410 327411 `/`, or the search button in the header, opens a search over the repo's@@ -481,11 +565,14 @@ push (log: `.forge/build.log`); `forge build` rebuilds everything.
481565 Every distinct deploy is extracted once into `.forge/store/<tree>/`, and the482566 old tree stays when a newer deploy takes its link. `forge gc` removes store483567 trees that no `www/sites` or `www/previews` link points at (hostnames point484-at those links), hostnames whose deploy is gone, and previews of review485-branches that no longer exist. A deploy extracts its tree before it links it,486-so store entries changed in the last 10 minutes are kept (`--age <minutes>`),487-which makes gc safe to run while people push. `--dry-run` lists what it would488-remove.568+at those links) and no `refs/releases/*` ref names (release files are hard569+links, which name no tree), hostnames whose deploy is gone, previews of570+review branches that no longer exist, and the downloads of releases whose571+ref was deleted without the hooks. A deploy extracts its tree before it572+links it, so store entries changed in the last 10 minutes are kept (`--age573+<minutes>`), which makes gc safe to run while people push. `--dry-run` lists574+what it would remove. A release published again leaves its old files in the575+repo's packs, unreachable: gc prunes nothing.489576 490577 Then it maintains every repo:491578 @@ -520,9 +607,12 @@ Settings forge gives every repo, when it is created and again on
520607 `receive.advertisePushOptions true` for review metadata,521608 `transfer.hideRefs refs/forge`, so clones, fetches and pushes don't see the522609 pinned revisions (the build reads them directly, and pushes to them are523-refused), and `commitGraph.changedPathsVersion 2`, Bloom filters that hash524-non-ASCII paths the same on every platform. The dumb protocol's `info/refs`525-still lists `refs/forge`; git clients use the smart protocol.610+refused), `uploadpack.hideRefs refs/releases`, so clones and fetches don't611+download release binaries but pushes still reach them, `core.bigFileThreshold612+16m`, so big blobs are stored without deltas, and613+`commitGraph.changedPathsVersion 2`, Bloom filters that hash non-ASCII paths614+the same on every platform. The dumb protocol's `info/refs` still lists615+`refs/forge` and `refs/releases`; git clients use the smart protocol.526616 527617 ### Moving repos over528618 @@ -549,8 +639,8 @@ credentials.
549639 Then point existing clones at the forge:550640 `git remote set-url origin git@git.example.com:dami/<repo>.git`.551641 552-A public repo is public everywhere: pages, sites, HTTPS clones and the pages553-mirror. A private one is only read over ssh for now; pages for it, behind a642+A public repo is public everywhere: pages, sites, releases, HTTPS clones and643+the pages mirror. A private one is only read over ssh for now; pages for it, behind a554644 login, come later (see [Private repos](#private-repos)).555645 556646 ## Backups and moving hosts@@ -566,7 +656,9 @@ hasn't run, so a box that was off catches up; a failed one is retried hourly.
566656 567657 Everything in `/srv/git` that can't be rebuilt:568658 569-- every bare repo, `<owner>/<repo>.git`, with its config, its659+- every bare repo, `<owner>/<repo>.git`, with its refs (releases too, and so660+ their files, which restic stores once however many backups hold them),661+ its config, its570662 `git-daemon-export-ok` if it has one (so a private repo comes back571663 private), and its whole `forge/` directory: review logs, comments,572664 collaborators, readers, and anything else kept there;@@ -578,7 +670,8 @@ Everything in `/srv/git` that can't be rebuilt:
578670 below, so state added later is backed up by default.579671 580672 Left out, because a restore rebuilds it or every boot rewrites it: the pages in581-`.forge/www`, the extracted sites in `.forge/store` (sites are git refs),673+`.forge/www`, the extracted sites and releases in `.forge/store` (both are674+git refs),582675 `.forge/cache`, `authorized_keys`, `allowed_signers`, `git-shell-commands/`,583676 hooks, `.forge/secrets` and `.forge/backup.env` (both written from `forge.env`584677 at boot), and logs, locks and scratch files.@@ -634,8 +727,18 @@ waits `FORGE_MIRROR_DELAY` seconds (30) after each build and only the newest
634727 one syncs, so a burst of pushes makes one sync. It never replaces the copy635728 with an empty forge. The edge serves this copy while slate is down (see636729 [above](#home-machine-behind-a-public-edge-how-srcrest-runs)). It holds637-public content only, the pages the forge serves anyway. When a repo goes638-private, the next sync deletes its pages there.730+public content only, the pages the forge serves anyway, and public repos'731+release downloads. When a repo goes private, the next sync deletes its732+pages and downloads there.733+734+A sync is two `rclone sync` passes. The pages go by checksum, since a735+rebuilt page that came out the same has a new mtime and shouldn't be736+uploaded again; that reads every file. So the downloads, which can be big737+and never change in place, are left out of it (`/*/*/releases/download/**`)738+and synced in a second pass by size and mtime, rclone's default. Their739+mtime is their release commit's, so an unchanged binary isn't read. Each740+pass leaves the other's files in the bucket alone. The edge serves741+downloads from the copy like any file, as attachments.639742 640743 ### Setting it up641744 @@ -678,7 +781,9 @@ What you set up yourself, once:
678781 `forge restore [<snapshot>|latest]` brings a forge back into an empty679782 `/srv/git`. It restores the files, rebuilds what a backup leaves out (what680783 `forge server init` and `forge keys sync` write, hooks in every repo, pinned681-review revisions, extracted sites) and runs `forge build`. It refuses a forge784+review revisions, extracted sites, public repos' release downloads, with785+the mtimes they had, so the mirror uploads none again) and runs `forge786+build`. It refuses a forge682787 that already has repos. The host's own `.forge/config` wins over the backup's,683788 since it is written for the domain that host serves, and users from both are684789 kept. `forge restore <snapshot> --to <dir>` only unpacks the files, to look at@@ -736,6 +841,7 @@ forge attest --stack # CI for every commit on top of main
736841 forge push -t "Title" # branch + notes; becomes a review737842 forge land # CI and approval checks, then fast-forward main738843 forge land --server # or let the forge land it, with its gates only844+git tag v1.0 && forge release publish v1.0 dist/* # files for a tag, signed739845 ```740846 741847 For a stack, `git rebase -i --update-refs` keeps the layers in sync, and@@ -856,6 +962,8 @@ may be missing needs an explicit check or `or ''`.
856962 | `/<owner>/<repo>/reviews.json` | the review list as data, for `forge reviews` |857963 | `/<owner>/<repo>/code.db` | the default branch's code index, read by code search in byte ranges |858964 | `/<owner>/<repo>/sites/` | deploys and previews |965+| `/<owner>/<repo>/releases/` | release notes per tag, and each release's files |966+| `/<owner>/<repo>/releases/download/<tag>/<file>` | a release file; `latest/` for the newest release's |859967 | `/sites/<owner>/<repo>/<name>/` | a deployed site |860968 861969 ## Not yet@@ -84,7 +84,7 @@
8484 <li><span class="where">sshd</span><div><p>Your key matches a line in <code>authorized_keys</code> that forces <code>forge shell <name></code>. It records who you are and hands the command to <code>git-shell</code>, which runs git and a few commands: <code>new</code>, <code>ls</code>, <code>whoami</code>, <code>signers</code>, <code>site-url</code>, <code>help</code>.</p><p class="src">bin/forge: cmd_shell</p></div></li>8585 <li><span class="where">pre-receive</span><div><p>Decides who may push which refs, refuses to delete <code>main</code> or force-push it (unless the repo allows that), and, when the repo asks for it, refuses a new <code>main</code> without a verified passing CI attestation.</p><p class="src">bin/forge: hook_pre_receive</p></div></li>8686 <li><span class="where">git</span><div><p>The refs move.</p></div></li>87-<li><span class="where">post-receive</span><div><p>Goes through the updated refs:</p><ul><li>a branch other than <code>main</code> gets a revision in its review log, and its tip is pinned;</li><li>a site or preview deploy is extracted and linked into the web root;</li><li>a deleted branch takes its previews with it;</li><li>the push is appended to <code>.forge/events.jsonl</code>.</li></ul><p class="src">bin/forge: hook_post_receive, record_review, deploy_site</p></div></li>87+<li><span class="where">post-receive</span><div><p>Goes through the updated refs:</p><ul><li>a branch other than <code>main</code> gets a revision in its review log, and its tip is pinned;</li><li>a site or preview deploy is extracted and linked into the web root, and so are a release's files;</li><li>a deleted branch takes its previews with it;</li><li>the push is appended to <code>.forge/events.jsonl</code>.</li></ul><p class="src">bin/forge: hook_post_receive, record_review, deploy_site</p></div></li>8888 <li class="returns"><span>your git push returns here; the rest runs in the background</span></li>8989 <li class="bg"><span class="where">forge build</span><div><p>11ty rebuilds that repo's pages into a scratch directory, hard-linking every page that didn't change from the live site, swaps them in, and appends a <code>built</code> event. Builds take turns under a lock.</p><p class="src">bin/forge: cmd_build, eleventy.config.ts, lib/forge.ts</p></div></li>9090 <li class="bg"><span class="where">your browser</span><div><p>Open pages hold a server-sent events stream from <code>/events/<owner>/<repo>/</code>. The push shows up as <q>dami pushed dami/flags, updating</q>; when the build lands, the page fetches itself and swaps in the new content, keeping your scroll position.</p><p class="src">lib/events-server.ts, site/assets/live.ts</p></div></li>@@ -214,6 +214,8 @@ log sha256:6cc79019... 4 lines
214214 <div class="prose">215215 <p>The pages server serves them the way Cloudflare Pages does, so a site built for there works unchanged: <code>_headers</code> and <code>_redirects</code> apply, <code>/about</code> finds <code>about.html</code> or <code>about/index.html</code>, a missing page gets the nearest <code>404.html</code> (or <code>index.html</code>, for a single-page app), and previews are sent with <code>noindex</code>. Old <code>/sites/...</code> paths redirect to the hostname. Without a pages domain, sites are plain files under <code>/sites/<owner>/<repo>/<name>/</code> on the forge itself.</p>216216 <p>Deleting a branch deletes its previews: the ref, the served link and the hostname, so the sites and review pages stop listing them. Previews of merged branches stay until their branch is deleted. The extracted files stay in the store until <code>forge gc</code>, which removes store trees nothing links to, hostnames whose deploy is gone and leftover previews, skipping store entries changed in the last ten minutes, so it can run while people push.</p>217+<h3>Releases are the same idea</h3>218+<p><code>forge release publish v1.0 <file>...</code> commits a tag's files to <code>refs/releases/v1.0</code> with a <code>SHA256SUMS</code> signed by your key (<code>ssh-keygen -n forge-release</code>). <code>pre-receive</code> checks every sum, the signature, the tag and a size cap, and with <code>forge.releaserequireci</code> a verified pass for the tag's tree. <code>post-receive</code> extracts the tree into the store and hard-links the files into <code>/dami/greet/releases/download/v1.0/</code>, plus <code>latest/</code>; hard links, so the mirror copies them. Clones and fetches never see <code>refs/releases</code>. A private repo's releases aren't served; its readers get them with <code>ssh git@host release get</code>.</p>217219 </div>218220 </section>219221 @@ -225,7 +227,7 @@ log sha256:6cc79019... 4 lines
225227 <div class="by">forge keys sync</div>226228 <div class="outs">227229 <div class="box"><span class="file">.ssh/authorized_keys</span><code>restrict,command="/opt/forge/bin/forge shell ana" ssh-ed25519 AAAAC3Nz... ana@laptop</code><p>Who is pushing: hooks see <code>FORGE_USER=ana</code>.</p></div>228-<div class="box"><span class="file">.forge/allowed_signers</span><code>ana namespaces="forge-ci,git" ssh-ed25519 AAAAC3Nz... ana@laptop</code><p>Who signed a CI result or a deploy. <code>ssh git@host signers</code> prints it for <code>forge verify</code>.</p></div>230+<div class="box"><span class="file">.forge/allowed_signers</span><code>ana namespaces="forge-ci,forge-release,git" ssh-ed25519 AAAAC3Nz... ana@laptop</code><p>Who signed a CI result, a deploy or a release's checksums. <code>ssh git@host signers</code> prints it for <code>forge verify</code>.</p></div>229231 </div>230232 <figcaption><b>fig. 7</b>one line in, two generated files out</figcaption>231233 </figure>@@ -238,7 +240,7 @@ log sha256:6cc79019... 4 lines
238240 <tr><th scope="row"><code>refs/heads/<wbr>ana/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="y" data-l="anyone else">yes, her own prefix</td></tr>239241 <tr><th scope="row">other branches</th><td class="y" data-l="owner, collaborators">yes</td><td class="n" data-l="anyone else">no</td></tr>240242 <tr><th scope="row"><code>refs/<wbr>site-previews/<wbr>*/ana/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="y" data-l="anyone else">yes</td></tr>241-<tr><th scope="row"><code>refs/<wbr>tags/*</code>, <code>refs/<wbr>sites/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="n" data-l="anyone else">no</td></tr>243+<tr><th scope="row"><code>refs/<wbr>tags/*</code>, <code>refs/<wbr>sites/*</code>, <code>refs/<wbr>releases/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="n" data-l="anyone else">no</td></tr>242244 <tr><th scope="row"><code>refs/<wbr>notes/ci</code></th><td class="y" data-l="owner, collaborators">yes, and may rewrite</td><td class="y" data-l="anyone else">add only: fast-forward, no delete</td></tr>243245 <tr><th scope="row"><code>refs/<wbr>forge/*</code></th><td class="n" data-l="owner, collaborators">internal</td><td class="n" data-l="anyone else">internal</td></tr>244246 </tbody>@@ -303,11 +305,16 @@ log sha256:6cc79019... 4 lines
303305 <div><dt>forge site push <dir> <name></dt><dd>Publish a directory you built yourself.</dd></div>304306 <div><dt>forge site url <name> [branch]</dt><dd>Where a deploy is served.</dd></div>305307 </dl></section>308+<section aria-labelledby="cs-releases"><h3 id="cs-releases">Releases</h3><dl>309+<div><dt>forge release publish <tag> <file>...</dt><dd>The tag's files with a signed <code>SHA256SUMS</code>, pushed with the tag and your CI notes; again to replace them.</dd></div>310+<div><dt>forge release verify <file>...</dt><dd>Check downloads against <code>SHA256SUMS</code> and its signature against the forge's keys.</dd></div>311+<div><dt>forge release ls, rm <tag></dt><dd>This repo's releases; delete one.</dd></div>312+</dl></section>306313 <section aria-labelledby="cs-server"><h3 id="cs-server">On the server, as root</h3><dl>307314 <div><dt>forge-admin user add ana 'ssh-ed25519 AAAA...'</dt><dd>Add a user and key; <code>user rm ana</code> revokes their keys.</dd></div>308315 <div><dt>forge-admin repo new dami/greet "description"</dt><dd>Or <code>repo import dami/greet <url></code> to fetch one server side.</dd></div>309316 <div><dt>forge-admin build [owner/repo]</dt><dd>Rebuild pages; <code>metrics</code> shows where builds spend their time.</dd></div>310-<div><dt>forge-admin gc [--dry-run]</dt><dd>Remove unused site files, hostnames and leftover previews, then repack each repo geometrically, with a multi-pack bitmap and a fresh commit-graph. Daily from cron.</dd></div>317+<div><dt>forge-admin gc [--dry-run]</dt><dd>Remove unused site and release files, hostnames and leftover previews, then repack each repo geometrically, with a multi-pack bitmap and a fresh commit-graph. Daily from cron.</dd></div>311318 <div><dt>forge-admin backups</dt><dd>How the last backup went, then the snapshots. <code>backup</code> takes one now; the container takes one daily.</dd></div>312319 <div><dt>forge-admin restore latest</dt><dd>Bring the forge back on an empty box, from the newest snapshot or a named one.</dd></div>313320 </dl></section>