clone

a personal forge on plain git: static pages, stacked-diff reviews, signed local ci, repo sites and releases

main
forge / guide / internals.html
internals.html 46.7K
1<!DOCTYPE html>2<html lang="en">3<head>4<meta charset="utf-8">5<meta name="viewport" content="width=device-width, initial-scale=1">6<title>forge internals: where things are and how they fit</title>7<meta name="description" content="For whoever maintains forge next: a push through the server, the source map, state on disk, identity, what runs and where, how a build works, URLs, settings and upkeep.">8<meta name="theme-color" content="#e9e9e9" media="(prefers-color-scheme: light)">9<meta name="theme-color" content="#08090b" media="(prefers-color-scheme: dark)">10<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke-width='2' stroke-linecap='round' stroke-linejoin='round'%3E%3Cstyle%3Ecircle%7Bstroke:%231c2127%7Dpath%7Bstroke:%233148f6%7D@media (prefers-color-scheme:dark)%7Bcircle%7Bstroke:%23e2e4e7%7D%7D%3C/style%3E%3Ccircle cx='12' cy='12' r='9'/%3E%3Cpath d='M16 15 12 7 8 15Zm-1 0L11.5 8'/%3E%3C/svg%3E">11<link rel="preload" href="fonts/open-sans.woff2" as="font" type="font/woff2" crossorigin>12<link rel="stylesheet" href="guide.css">13<script src="guide.js" defer></script>14</head>15<body>16<a class="skip" href="#main">skip to the internals</a>17<header class="bar">18<div class="wrap">19<a class="brand" href="./"><svg class="mark" viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="12" cy="12" r="9"/><path d="M16 15 12 7 8 15Zm-1 0L11.5 8"/></svg>forge <span>guide</span></a>20<nav aria-label="guide"><a href="./">guide</a><a href="internals.html" aria-current="page">internals</a></nav>21</div>22</header>2324<section class="hero short" aria-labelledby="top">25<div class="wrap">26<div class="hero-text">27<p class="eyebrow">for whoever maintains it next</p>28<h1 id="top">internals</h1>29<p class="lede">How a push becomes pages, where the code is, what it keeps on disk, who may push what, what runs where and how to keep it running. Read <a href="./">the guide</a> first for what forge does; this page is about how.</p>30</div>31</div>32</section>3334<div class="wrap page">35<nav class="toc" aria-label="on this page">36<div>37<h2>on this page</h2>38<ol>39<li><a href="#push"><span>01</span>a push</a></li>40<li><a href="#map"><span>02</span>source</a></li>41<li><a href="#disk"><span>03</span>on disk</a></li>42<li><a href="#identity"><span>04</span>identity</a></li>43<li><a href="#deploys"><span>05</span>signed refs</a></li>44<li><a href="#run"><span>06</span>what runs</a></li>45<li><a href="#hosting"><span>07</span>hosting</a></li>46<li><a href="#build"><span>08</span>the build</a></li>47<li><a href="#urls"><span>09</span>urls</a></li>48<li><a href="#knobs"><span>10</span>settings</a></li>49<li><a href="#server"><span>11</span>on the server</a></li>50<li><a href="#upkeep"><span>12</span>upkeep</a></li>51<li><a href="#hacking"><span>13</span>hacking</a></li>52</ol>53</div>54</nav>5556<main id="main" class="log">5758<section class="commit" id="push" aria-labelledby="push-h">59<header><p class="ref"><b>01</b> a push</p><h2 id="push-h"><a href="#push">What happens when you push</a></h2></header>60<p class="prose">A push is how almost everything changes. Here is one, followed through the server.</p>61<figure class="plate">62<ol class="flow">63<li><span class="where">your machine</span><div><p><code>git push</code>, or <code>forge push</code>, which also merges the server's CI notes with yours, pushes them along and passes options such as a review title.</p></div></li>64<li><span class="where">sshd</span><div><p>Your key matches a line in <code>authorized_keys</code> that forces <code>forge shell &lt;name&gt;</code>. It records who you are, creates a repo of yours that a push names but isn't there yet, and hands the command to <code>git-shell</code>, which runs git or one of the commands in <code>git-shell-commands/</code> (<code>new</code>, <code>ls</code>, <code>reviews</code>, <code>land</code>, <code>comment</code>, <code>who</code>, <code>passkey</code>, <code>token</code> and the rest; <code>help</code> lists them). An admin's session also finds the admins' own, <code>invite</code>, <code>invites</code>, <code>users</code> and <code>motd</code>.</p><p class="src">bin/forge: cmd_shell</p></div></li>65<li><span class="where">pre-receive</span><div><p>Decides who may push which refs, refuses to delete <code>main</code> or rewrite it (unless the repo allows that), turns away a push that adds a credential, checks a release's sums and signature, and, when the repo asks for it, refuses a new <code>main</code> without a verified passing CI attestation or another person's approval.</p><p class="src">bin/forge: hook_pre_receive, scan_secrets, check_release</p></div></li>66<li><span class="where">git</span><div><p>The refs move.</p></div></li>67<li><span class="where">post-receive</span><div><p>Goes through the updated refs:</p><ul><li>a branch other than <code>main</code> gets a revision in its review log, and its tip is pinned;</li><li>a site or preview deploy is extracted and linked into the web root, and so are a release's files;</li><li>a deleted branch takes its previews with it;</li><li>the push is appended to <code>.forge/events.jsonl</code>.</li></ul><p class="src">bin/forge: hook_post_receive, record_review, deploy_site</p></div></li>68<li class="returns"><span>your git push returns here; the rest runs in the background</span></li>69<li class="bg"><span class="where">forge build</span><div><p>11ty rebuilds that repo's pages into a scratch directory, hard-linking every page that didn't change from the live site, swaps them in and appends a <code>built</code> event. Builds take turns under a lock (<a href="#build">the build</a>).</p><p class="src">bin/forge: cmd_build, eleventy.config.ts, lib/forge.ts</p></div></li>70<li class="bg"><span class="where">your browser</span><div><p>Open pages hold a server-sent events stream from <code>/events/&lt;owner&gt;/&lt;repo&gt;/</code>. The push shows up as <q>cat pushed cat/flags, updating</q>; when the build lands, the page fetches itself and swaps in the new content, keeping your scroll position.</p><p class="src">lib/events-server.ts, site/assets/live.ts</p></div></li>71</ol>72<figcaption><b>fig. 1</b>one push, from your terminal to every open page</figcaption>73</figure>74<p class="prose">Building after the push rather than on each request means public pages cost nothing to serve, can be cached anywhere and keep working if the small Node servers stop. Those only add what files can't: live updates, private pages for their readers, comments and approvals from the web, and sites on their own hostnames.</p>75</section>7677<section class="commit" id="map" aria-labelledby="map-h">78<header><p class="ref"><b>02</b> source</p><h2 id="map-h"><a href="#map">Where the code is</a></h2></header>79<p class="prose">Four kinds of code, kept small: one shell script for everything git and ssh touch, a TypeScript data layer that Node 22 runs as it is (types are stripped, never compiled), Nunjucks templates with a base stylesheet, and container and deploy files.</p>80<div class="table">81<table class="rows">82<thead><tr><th scope="col">path</th><th scope="col">what it is</th></tr></thead>83<tbody>84<tr><th scope="row"><code>bin/forge</code></th><td>The whole CLI in POSIX sh. Developer commands (<code>attest</code>, <code>verify</code>, <code>push</code>, <code>site</code>, <code>import</code>) run in a clone; server commands (<code>server init</code>, <code>user</code>, <code>keys sync</code>, <code>repo</code>, <code>build</code>, <code>gc</code>, <code>shell</code>, <code>hook</code>) run as the git user. Both hooks are this script.</td></tr>85<tr><th scope="row"><code>lib/forge.ts</code></th><td>Finds repos and flattens them into one list per page type for 11ty. Fingerprints every page so unchanged ones are skipped, and holds the filters that render heavy HTML when a page is written.</td></tr>86<tr><th scope="row"><code>lib/repo.ts</code></th><td>Loads one bare repo: refs, commit metadata, trees, CI notes, reviews, sites. Diffs and file contents are read later, only for pages that will render.</td></tr>87<tr><th scope="row"><code>lib/reviews.ts</code></th><td>Review logs plus live refs become stacks, layers, statuses and range-diffs.</td></tr>88<tr><th scope="row"><code>lib/attest.ts</code></th><td>Parses attestation notes and verifies their signatures with <code>ssh-keygen</code>, remembering each result.</td></tr>89<tr><th scope="row"><code>lib/sites.ts</code>, <code>lib/hosts.ts</code></th><td>Deploy history and signatures for the sites page; per-site hostnames, with <code>www/hosts/</code> as the registry. <code>hosts.ts</code> is also a small CLI the hooks call.</td></tr>90<tr><th scope="row"><code>lib/events-server.ts</code>, <code>lib/private-server.ts</code>, <code>lib/pages-server.ts</code></th><td>The long-running servers: live updates over server-sent events; private pages, sign-in, and comments and approvals from the web; sites on their own hostnames.</td></tr>91<tr><th scope="row"><code>lib/identity.ts</code>, <code>lib/access.ts</code>, <code>lib/login.ts</code></th><td>Web credentials in <code>identity.db</code> (passkeys, sessions, tokens, invites), who may read what, and passkey sign-in.</td></tr>92<tr><th scope="row"><code>lib/</code>, the rest</th><td><code>git.ts</code> wraps the git CLI, <code>diff.ts</code> parses patches, <code>render.ts</code> writes diff and code HTML, <code>markdown.ts</code> renders and sanitizes READMEs and wiki pages, <code>urls.ts</code> is the URL scheme, <code>cache.ts</code> the per-repo build cache, <code>assets.ts</code> the hashed asset names, <code>render-page.ts</code> the Nunjucks setup 11ty shares and pages rendered without 11ty, <code>metrics.ts</code> build timings, <code>types.ts</code> the model.</td></tr>93<tr><th scope="row"><code>site/</code>, <code>eleventy.config.ts</code></th><td>Templates, each extending <code>_includes/base.njk</code> and carrying its own CSS in <code>{% css %}</code> blocks; <code>assets/style.css</code> for what every page needs; the small client scripts in <code>assets/*.ts</code>, whose types are stripped when the site is built.</td></tr>94<tr><th scope="row"><code>server/</code></th><td>The container's <code>start.sh</code>, <code>sshd.conf</code> and <code>forge-admin</code>; <code>provision.sh</code> for a plain VPS; an example Caddyfile.</td></tr>95<tr><th scope="row"><code>deploy/</code></th><td><code>edge/</code>, the Fly relay; <code>slate/</code>, the compose file and the rsync deploy for the home box.</td></tr>96<tr><th scope="row"><code>demo/</code>, <code>.forge/</code>, <code>guide/</code></th><td>A throwaway forge built with the real hooks, and a local server for it; forge's own CI script and site list; this guide.</td></tr>97</tbody>98</table>99</div>100</section>101102<section class="commit" id="disk" aria-labelledby="disk-h">103<header><p class="ref"><b>03</b> on disk</p><h2 id="disk-h"><a href="#disk">State on disk</a></h2></header>104<p class="prose">Everything lives under the git user's home, <code>FORGE_ROOT</code>: <code>/srv/git</code> on a server, <code>demo/root</code> in the demo. Back up this one directory and you have the forge; <code>forge backup</code> does, leaving out what a restore rebuilds (<a href="#upkeep">upkeep</a>).</p>105<figure>106<pre class="tree"><span class="d">/srv/git/</span>107├── .ssh/authorized_keys <span class="c">generated from .forge/keys108</span>├── git-shell-commands/ <span class="c">new, ls, reviews, land, comment, approve, help...109</span>├── <span class="d">cat/greet.git/</span> <span class="c">a bare repo; the directory names the owner110</span>│ ├── hooks/ <span class="c">pre-receive, post-receive: exec forge hook ...111</span>│ ├── forge/reviews/&lt;id&gt; <span class="c">one log per review112</span>│ ├── forge/collaborators <span class="c">names that may push anything113</span>│ ├── refs/forge/keep/&lt;sha&gt; <span class="c">every revision's tip, hidden from clients114</span>│ ├── refs/notes/ci <span class="c">signed attestations115</span>│ ├── refs/sites/&lt;name&gt; <span class="c">deploys116</span>│ └── refs/site-previews/&lt;name&gt;/&lt;branch&gt;117└── <span class="d">.forge/</span>118 ├── keys <span class="c">name and key per line: the only identity119</span> ├── allowed_signers <span class="c">generated from keys120</span> ├── config <span class="c">shell variables, sourced by forge121</span> ├── <span class="d">www/</span> <span class="c">the pages, served as plain files122</span> │ ├── sites/, previews/ <span class="c">symlinks into store/123</span> │ └── hosts/&lt;label&gt; <span class="c">symlinks to sites/ or previews/124</span> ├── store/&lt;tree&gt;/ <span class="c">each deployed tree, extracted once125</span> ├── cache/ <span class="c">facts keyed by object id, page fingerprints126</span> ├── events.jsonl <span class="c">pushes and builds, for live pages127</span> ├── backup.env <span class="c">backup and mirror keys, written from forge.env128</span> ├── backup.status, backup.log <span class="c">the last backup's health, for forge backups129</span> ├── backup/ <span class="c">staging, restic's cache, the restore drill130</span> └── build.log, metrics.jsonl</pre>131<figcaption><b>fig. 2</b>the forge's state, all of it</figcaption>132</figure>133<div class="prose">134<p>A review log is a few lines appended by <code>post-receive</code>:</p>135</div>136<pre class="code">branch cat/help137created 1788276612 cat138opt title=Usage and --help139rev 1788276612 ef1d881b67e0... cat 895bc48ac1d9... <span class="dim"># time, tip, who, merge-base with main</span>140rev 1788282012 6771779c76bb... cat 895bc48ac1d9...141deleted 1788292812 cat <span class="dim"># only once the branch is gone</span></pre>142<p class="prose">The container adds <code>.hostkeys/</code>, so clients see the same host key across deploys, and <code>.tailcat/</code>, whose server key keeps the tunnel address stable. Both are root's, and both are in every backup, so a restored box keeps its identity.</p>143</section>144145<section class="commit" id="identity" aria-labelledby="identity-h">146<header><p class="ref"><b>04</b> identity</p><h2 id="identity-h"><a href="#identity">Identity is one file of keys</a></h2></header>147<p class="prose"><code>.forge/keys</code> holds <code>name key</code> lines and is the root of identity; passkeys, sessions and tokens in <code>identity.db</code> each belong to a name in it. <code>forge keys sync</code> (run by <code>forge user add</code>, <code>user rm</code> and a taken-up invite) writes two files from it, so the key you push with is also the key that signs your CI results, deploys and releases.</p>148<figure class="plate fan">149<div class="box"><span class="file">.forge/keys</span><code>fox <wbr>ssh-ed25519 <wbr>AAAAC3NzaC1lZDI1NTE5... <wbr>fox@laptop</code></div>150<div class="by">forge keys sync</div>151<div class="outs">152<div class="box"><span class="file">.ssh/authorized_keys</span><code>restrict,command="/<wbr>opt/<wbr>forge/<wbr>bin/<wbr>forge <wbr>shell <wbr>fox" <wbr>ssh-ed25519 <wbr>AAAAC3Nz... <wbr>fox@laptop</code><p>Who is pushing: hooks see <code>FORGE_USER=fox</code>.</p></div>153<div class="box"><span class="file">.forge/allowed_signers</span><code>fox <wbr>namespaces="forge-ci,forge-release,forge-review,git" <wbr>ssh-ed25519 <wbr>AAAAC3Nz... <wbr>fox@laptop</code><p>Who signed a CI result, a deploy, a release's checksums or an agent review. <code>ssh git@host signers</code> prints it for <code>forge verify</code>.</p></div>154</div>155<figcaption><b>fig. 3</b>one line in, two generated files out</figcaption>156</figure>157<p class="prose">Repos live at <code>/srv/git/&lt;owner&gt;/&lt;repo&gt;.git</code>, and the directory names the owner. The owner, and anyone in <code>&lt;repo&gt;.git/forge/collaborators</code>, can push anything; everyone else who can read the repo works under their own name. This is what <code>pre-receive</code> enforces, ref by ref:</p>158<div class="table">159<table class="rows">160<thead><tr><th scope="col">ref</th><th scope="col">owner, collaborators</th><th scope="col">anyone else who can read it, e.g. fox</th></tr></thead>161<tbody>162<tr><th scope="row"><code>main</code> (the default branch)</th><td class="y" data-l="owner, collaborators">fast-forward only, unless <code>forge.allowforce</code>; never deleted</td><td class="n" data-l="anyone else">no</td></tr>163<tr><th scope="row"><code>refs/heads/<wbr>fox/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="y" data-l="anyone else">yes, their own prefix</td></tr>164<tr><th scope="row">other branches</th><td class="y" data-l="owner, collaborators">yes</td><td class="n" data-l="anyone else">no</td></tr>165<tr><th scope="row"><code>refs/<wbr>site-previews/<wbr>*/fox/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="y" data-l="anyone else">yes</td></tr>166<tr><th scope="row"><code>refs/<wbr>tags/*</code>, <code>refs/<wbr>sites/*</code>, <code>refs/<wbr>releases/*</code></th><td class="y" data-l="owner, collaborators">yes</td><td class="n" data-l="anyone else">no</td></tr>167<tr><th scope="row"><code>refs/<wbr>notes/ci</code></th><td class="y" data-l="owner, collaborators">yes, and may rewrite</td><td class="y" data-l="anyone else">add only: fast-forward, no delete</td></tr>168<tr><th scope="row"><code>refs/<wbr>forge/*</code></th><td class="n" data-l="owner, collaborators">internal</td><td class="n" data-l="anyone else">internal</td></tr>169</tbody>170</table>171</div>172<p class="prose">Removing someone's line revokes their pushes, their web credentials and their signatures together: their old attestations stop verifying. A private repo is one without <code>git-daemon-export-ok</code>, git's own flag; <code>forge shell</code> turns away anyone who may not read it with the same answer as for a missing repo, and so does the private server for its pages.</p>173</section>174175<section class="commit" id="deploys" aria-labelledby="deploys-h">176<header><p class="ref"><b>05</b> signed refs</p><h2 id="deploys-h"><a href="#deploys">CI notes, sites and releases are refs</a></h2></header>177<div class="prose">178<p><strong>CI.</strong> <code>forge attest</code> checks each commit out into a worktree (one per repo, reused so build caches stay warm), runs <code>.forge/ci</code> in it (inside <code>nix develop</code> when there is a <code>flake.nix</code> and Nix is installed), signs the statement with <code>ssh-keygen -Y sign -n forge-ci</code> and appends it to <code>refs/notes/ci</code>, keyed by commit. <code>pre-receive</code> verifies against the server's own <code>allowed_signers</code> when <code>forge.requireci</code> asks it to; anyone who can push may add notes, as long as <code>refs/notes/ci</code> only fast-forwards.</p>179<p><strong>Reviews.</strong> A review's parent is the nearest other review whose tip lies inside its branch; its layer is the commits between that parent (or <code>main</code>) and its tip. Status and revisions come from the review log and the refs, so nothing else is stored.</p>180<p><strong>Sites.</strong> A deploy is a commit on <code>refs/sites/&lt;name&gt;</code> whose tree is the built output, so git provides content-addressed storage, dedup across deploys, history and revert by pushing an older deploy. Trailers record the source commit, the build command and who built it, and the commit carries an SSH signature when your key can sign. A preview is the same on <code>refs/<wbr>site-previews/<wbr>&lt;name&gt;/<wbr>&lt;branch&gt;</code>. The build gets <code>FORGE_SITE_BASE</code>, the path it will be served from (<code>/</code> on its own hostname).</p>181</div>182<figure class="plate">183<ol class="chain">184<li><span class="k">ref</span><span class="v">refs/sites/docs<small>a deploy commit, signed</small></span></li>185<li class="hop"><span>post-receive extracts its tree, once per tree</span></li>186<li><span class="k">store</span><span class="v">.forge/store/339f775e.../</span></li>187<li class="hop up"><span>symlink</span></li>188<li><span class="k">served</span><span class="v">www/<wbr>sites/<wbr>cat/<wbr>greet/<wbr>docs</span></li>189<li class="hop up"><span>symlink, with a pages domain</span></li>190<li><span class="k">hostname</span><span class="v">www/hosts/greet-docs</span></li>191<li class="hop up"><span>the pages server looks up the Host header</span></li>192<li><span class="k">browser</span><span class="v">https://<wbr>greet-docs<wbr>.pages.example.com/</span></li>193</ol>194<figcaption><b>fig. 4</b>a deploy, from ref to hostname</figcaption>195</figure>196<div class="prose">197<p>The <code>www/hosts/</code> directory of symlinks is the whole hostname registry. Without a pages domain, sites are plain files under <code>/sites/&lt;owner&gt;/&lt;repo&gt;/&lt;name&gt;/</code> on the forge itself, and old <code>/sites/...</code> paths redirect to the hostname when there is one. Only the owner and collaborators deploy sites; anyone may deploy previews of their own branches.</p>198<p><strong>Releases.</strong> <code>forge release publish</code> commits a tag's files to <code>refs/releases/&lt;tag&gt;</code> with a <code>SHA256SUMS</code> signed with <code>ssh-keygen -n forge-release</code> over the repo and tag too. <code>pre-receive</code> checks every sum, the signature (an owner's or collaborator's key), the tag and a size cap, and with <code>forge.releaserequireci</code> a verified pass for the tag's tree. <code>post-receive</code> extracts the tree into the store and hard-links the files into <code>/&lt;owner&gt;/<wbr>&lt;repo&gt;/<wbr>releases/<wbr>download/<wbr>&lt;tag&gt;/</code>, plus <code>latest/</code>; hard links, so the mirror copies them. Clones and fetches never see <code>refs/releases</code>. A private repo's go to the private root next to its pages, never the mirror.</p>199</div>200</section>201202<section class="commit" id="run" aria-labelledby="run-h">203<header><p class="ref"><b>06</b> what runs</p><h2 id="run-h"><a href="#run">What runs and why</a></h2></header>204<p class="prose">The container's entrypoint is <code>tini</code>, which also reaps the background builds, and then <code>server/container/start.sh</code>. It creates host keys on first boot, runs <code>forge server init</code>, writes <code>.forge/config</code> from the environment, adds the owner's key, syncs keys, writes the Caddyfile with <code>forge server caddyfile</code> and starts:</p>205<div class="table">206<table class="rows">207<thead><tr><th scope="col">process</th><th scope="col">listens</th><th scope="col">as</th><th scope="col">why</th></tr></thead>208<tbody>209<tr><th scope="row">sshd</th><td data-l="listens"><code>:2222</code></td><td data-l="as">root</td><td>git over ssh, only for the git user, keys only.</td></tr>210<tr><th scope="row">fcgiwrap</th><td data-l="listens">unix socket</td><td data-l="as">git</td><td>Runs <code>git http-backend</code> for HTTPS clones; Caddy routes only <code>info/refs</code> and <code>git-upload-pack</code> to it.</td></tr>211<tr><th scope="row">tailcat</th><td data-l="listens">the tunnel</td><td data-l="as">root</td><td>Only with <code>FORGE_TAILCAT_ALLOW</code>: carries :2222 and :8080 to the edge.</td></tr>212<tr><th scope="row">events server</th><td data-l="listens"><code>127.0.0.1:8091</code></td><td data-l="as">git</td><td>Tails <code>events.jsonl</code>, streams it at <code>/events/</code>.</td></tr>213<tr><th scope="row">private server</th><td data-l="listens"><code>127.0.0.1:8093</code></td><td data-l="as">git</td><td>Every path <code>www</code> has no file for: private pages for their readers, <code>/login/</code>, comment and approve posts, and the one 404.</td></tr>214<tr><th scope="row">litestream</th><td data-l="listens"></td><td data-l="as">git</td><td>Only with <code>FORGE_BACKUP_REPOSITORY</code>: copies <code>identity.db</code> to the backup bucket as it changes.</td></tr>215<tr><th scope="row">pages server</th><td data-l="listens"><code>127.0.0.1:8092</code></td><td data-l="as">caddy</td><td>Only with <code>FORGE_PAGES_DOMAIN</code>: sites on their own hostnames. Runs as caddy because deploys are untrusted.</td></tr>216<tr><th scope="row"><code>forge build</code></th><td data-l="listens"></td><td data-l="as">git</td><td>Once at boot, in the background, since templates may have changed with the image; after that, after each push.</td></tr>217<tr><th scope="row">backup loop</th><td data-l="listens"></td><td data-l="as">root</td><td>Only with <code>FORGE_BACKUP_REPOSITORY</code>: runs <code>forge backup</code> once a day at <code>FORGE_BACKUP_AT</code>, catching up after downtime, and prints its health to the container log.</td></tr>218<tr><th scope="row"><code>forge mirror</code></th><td data-l="listens"></td><td data-l="as">git</td><td>Only with <code>FORGE_MIRROR_S3</code>: after builds, at most hourly, syncs the forge's pages to the bucket the edge falls back to.</td></tr>219<tr><th scope="row">Caddy</th><td data-l="listens"><code>:8080</code></td><td data-l="as">caddy</td><td>In the foreground. Plain HTTP; TLS ends at the edge. Pages, <code>.git</code> files, <code>go get</code> meta tags and the two proxies.</td></tr>220</tbody>221</table>222</div>223<p class="prose">The container takes <code>FORGE_DOMAIN</code>, <code>FORGE_PAGES_DOMAIN</code>, <code>FORGE_OWNER</code>, <code>FORGE_OWNER_KEY</code>, <code>FORGE_NAME</code> and <code>FORGE_TAILCAT_ALLOW</code>, plus <code>FORGE_BACKUP_*</code> and <code>FORGE_MIRROR_*</code>, which it writes to <code>.forge/backup.env</code> for backups and the mirror. On a VPS, <code>provision.sh</code> installs the same pieces natively and runs the Node servers as systemd units, <code>forge-events</code>, <code>forge-private</code> and, with a pages domain, <code>forge-pages</code>, with Caddy on :443 getting its own certificates.</p>224</section>225226<section class="commit" id="hosting" aria-labelledby="hosting-h">227<header><p class="ref"><b>07</b> hosting</p><h2 id="hosting-h"><a href="#hosting">Where it runs</a></h2></header>228<p class="prose">The forge is one container: sshd for git, Caddy for pages, sites and read-only HTTPS clones (<code>git http-backend</code> through fcgiwrap, and <code>go get</code> works), the events server, the private server, the pages server when there is a pages domain and optionally tailcat. All state is in <code>/srv/git</code>, a volume. <code>forge backup</code> snapshots it every day with restic into an R2 bucket and test-restores each snapshot, and <code>forge restore</code> brings the forge back on a new box, host keys and tunnel address included.</p>229<div class="hosts">230<figure class="plate">231<ol class="hops">232<li class="node"><b>internet</b><span>ssh on :22, https on :443. DNS for the forge and pages domains points at the edge, DNS-only, since ssh can't go through Cloudflare's proxy.</span></li>233<li class="wire">raw TCP for :22, TLS ends here for :443</li>234<li class="node"><b>Fly edge</b><span><code>deploy/edge</code>. Stateless, with a dedicated IPv4 so raw TCP on :22 works. Its key and the container's address are Fly secrets. While the home box is down, its Caddy serves the forge's pages from a copy in R2.</span></li>235<li class="wire tunnel">tailcat: point-to-point WireGuard on Tailscale's data plane, with no tailnet, accounts or control plane</li>236<li class="node home"><b>forge container on a home box</b><span><code>deploy/slate</code>. Publishes no ports. Its tailcat server keeps its key on the volume and accepts only the edge's client key.</span><ul><li>sshd :2222</li><li>Caddy :8080</li><li>events :8091</li><li>pages :8092</li><li>private :8093</li></ul></li>237</ol>238<figcaption><b>fig. 5</b>how src.rest runs</figcaption>239</figure>240<figure class="plate">241<ol class="hops">242<li class="node"><b>internet</b><span>ssh on :22, https on :443</span></li>243<li class="wire">straight in</li>244<li class="node home"><b>a Debian or Ubuntu VPS</b><span><code>server/provision.sh</code> installs git, sshd and Caddy, which gets certificates itself, and runs the Node servers under systemd. It needs Node 22.18 or newer.</span><ul><li>sshd :22</li><li>Caddy :443</li><li>forge-events</li><li>forge-private</li><li>forge-pages</li></ul></li>245</ol>246<figcaption><b>fig. 6</b>or a plain VPS</figcaption>247</figure>248</div>249<p class="prose">Either way, Caddy serves the pages directory as plain files and hands on only a few things: clones to <code>git http-backend</code>, <code>/events/</code> to the events server, comment and approve posts and every path the pages directory has no file for to the private server and, with a pages domain, sites to the pages server. The pages server runs as the caddy user, because deploys are untrusted content. A private repo's pages are built into a root of their own and served by the private server to readers signed in with a passkey or showing a token; to anyone else every page of it is the 404 of a page that isn't there. Passkeys, sessions and tokens live, as hashes, in one SQLite file that litestream copies to the backup bucket as it changes.</p>250</section>251252<section class="commit" id="build" aria-labelledby="build-h">253<header><p class="ref"><b>08</b> the build</p><h2 id="build-h"><a href="#build">How a push becomes pages</a></h2></header>254<figure class="plate">255<ol class="flow">256<li><span class="where">post-receive</span><div><p>Starts <code>forge build &lt;owner&gt;/&lt;repo&gt;</code> with <code>nohup</code>, logging to <code>build.log</code>. <code>FORGE_SYNC_BUILD=1</code> runs it in the foreground instead; <code>FORGE_NO_BUILD=1</code> skips it.</p></div></li>257<li><span class="where">lock</span><div><p><code>mkdir .forge/build.lock</code>. Builds wait their turn, for up to 10 minutes.</p></div></li>258<li><span class="where">graph</span><div><p><code>git <wbr>commit-graph <wbr>write <wbr>--reachable <wbr>--changed-paths <wbr>--split</code> adds a small layer for the new commits, with Bloom filters, so <code>git log -- &lt;path&gt;</code> and blame skip commits that don't touch the path. Here rather than in the hook, so the pusher never waits on it.</p></div></li>259<li><span class="where">data</span><div><p>11ty loads <code>lib/forge.ts</code> with <code>FORGE_ONLY</code> set, so only that repo is read. Index and owner pages use cached summaries of the others. Commit metadata, range-diffs and signature checks come from <code>.forge/cache/</code>, keyed by object ids that never change.</p></div></li>260<li><span class="where">skip</span><div><p>Every page has a fingerprint of what it shows, plus a hash of the forge's own code. A page whose fingerprint matches the last build, and whose file still exists, isn't rendered: its file is hard-linked from the live site.</p></div></li>261<li><span class="where">render</span><div><p>The rest render into <code>.forge/build.&lt;pid&gt;/</code>. Diffs and file views are written by filters as each page is, so memory stays bounded by the largest page, not the repo. Each page links one CSS bundle of the blocks its templates carry.</p></div></li>262<li><span class="where">assets</span><div><p>Stylesheet, scripts and bundles get names from a hash of their contents, so browsers cache them for good. Pages from older code are still live (the other repos mid-rebuild, or a repo whose build failed), so the files of the last five code versions are carried over from the live site; <code>assets/versions.json</code> lists them.</p></div></li>263<li><span class="where">swap</span><div><p>The repo's directory is moved into <code>www/</code> in place of the old one, and the owner's index page is copied over. <code>sites/</code>, <code>previews/</code> and <code>hosts/</code> belong to the hooks and are never touched.</p></div></li>264<li><span class="where">after</span><div><p>A line in <code>metrics.jsonl</code> (<code>forge metrics</code> reads it) and a <code>built</code> event that open pages act on.</p></div></li>265</ol>266<figcaption><b>fig. 7</b>a single-repo build</figcaption>267</figure>268<p class="prose"><code>forge build</code> with no argument rebuilds every repo, one at a time so memory stays bounded by the largest, then the index pages, then removes pages of repos that no longer exist and prunes dangling hostnames.</p>269</section>270271<section class="commit" id="urls" aria-labelledby="urls-h">272<header><p class="ref"><b>09</b> urls</p><h2 id="urls-h"><a href="#urls">URLs</a></h2></header>273<p class="prose">Every page is a directory with an <code>index.html</code>, so URLs read like paths. Branch names may contain <code>/</code>, which would make <code>/tree/&lt;branch&gt;/&lt;path&gt;</code> ambiguous, so it is written <code>~</code>, which git forbids in ref names: <code>cat/flags</code> becomes <code>cat~flags</code>.</p>274<div class="table">275<table class="rows">276<thead><tr><th scope="col">url</th><th scope="col">page</th></tr></thead>277<tbody>278<tr><td><code>/&lt;owner&gt;/&lt;repo&gt;/</code></td><td>open reviews, recent commits, README</td></tr>279<tr><td><code>.../log/&lt;branch&gt;/</code></td><td>log for a branch</td></tr>280<tr><td><code>.../tree/&lt;branch&gt;/&lt;path&gt;/</code></td><td>directory</td></tr>281<tr><td><code>.../blob/&lt;branch&gt;/&lt;path&gt;/</code></td><td>file; markdown renders, with its source at <code>source/</code></td></tr>282<tr><td><code>.../commit/&lt;sha&gt;/</code></td><td>commit</td></tr>283<tr><td><code>.../reviews/&lt;id&gt;/</code></td><td>stacked diff; <code>diff/</code> combined, <code>r&lt;n&gt;/</code> range-diff</td></tr>284<tr><td><code>.../wiki/&lt;page&gt;/</code>, <code>.../sites/</code>, <code>.../atom.xml</code></td><td>wiki, deploys and previews, feed</td></tr>285<tr><td><code>/&lt;owner&gt;/&lt;repo&gt;.git</code></td><td>HTTPS clone</td></tr>286<tr><td><code>/sites/&lt;owner&gt;/&lt;repo&gt;/&lt;name&gt;/</code></td><td>a site, or a redirect to its hostname</td></tr>287<tr><td><code>/previews/.../&lt;name&gt;/&lt;branch&gt;/</code></td><td>a preview, likewise</td></tr>288<tr><td><code>/events/&lt;owner&gt;/&lt;repo&gt;/</code></td><td>the live event stream</td></tr>289</tbody>290</table>291</div>292</section>293294<section class="commit" id="knobs" aria-labelledby="knobs-h">295<header><p class="ref"><b>10</b> settings</p><h2 id="knobs-h"><a href="#knobs">Settings, wherever they live</a></h2></header>296<div class="cheat keys">297<section aria-labelledby="k-config"><h3 id="k-config">.forge/config, on the server</h3><dl>298<div><dt>FORGE_NAME, FORGE_BASE_URL, FORGE_SSH</dt><dd>The forge's name, web address and ssh address, for pages and clone URLs.</dd></div>299<div><dt>FORGE_PAGES_DOMAIN, FORGE_OWNER</dt><dd>Turn on per-site hostnames; the owner's sites get the short ones.</dd></div>300<div><dt>FORGE_MAX_COMMITS, FORGE_MAX_FILES, FORGE_MAX_BLOB, FORGE_MAX_DIFF_LINES</dt><dd>Limits per branch log, per tree, per file shown and per diff shown. The defaults are 1,000 commits, 5,000 files, 1 MiB and 20,000 lines.</dd></div>301</dl></section>302<section aria-labelledby="k-repo"><h3 id="k-repo">a bare repo's git config</h3><dl>303<div><dt>forge.requireci tip | all</dt><dd>A verified pass before <code>main</code> moves, for the new tip or every new commit.</dd></div>304<div><dt>forge.allowforce true</dt><dd>Let admins force-push <code>main</code>.</dd></div>305<div><dt>transfer.hideRefs refs/forge</dt><dd>Set by forge on every repo, with <code>receive.advertisePushOptions <wbr>true</code> and <code>commitGraph.changedPathsVersion <wbr>2</code>; see <a href="#upkeep">upkeep</a>.</dd></div>306</dl></section>307<section aria-labelledby="k-dev"><h3 id="k-dev">your git config</h3><dl>308<div><dt>forge.signingkey, forge.allowedsigners</dt><dd>Sign with this key; verify against this file.</dd></div>309<div><dt>forge.host, forge.remote, forge.base</dt><dd>Where <code>forge import</code> goes; which remote and base branch the other commands use.</dd></div>310<div><dt>forge.cidir, forge.nix</dt><dd>Where CI worktrees live; <code>forge.nix false</code> skips <code>nix develop</code>.</dd></div>311</dl></section>312<section aria-labelledby="k-backup"><h3 id="k-backup">forge.env, for backups</h3><dl>313<div><dt>FORGE_BACKUP_REPOSITORY, _PASSWORD, _KEY_ID, _SECRET</dt><dd>The restic repository (an R2 bucket), its password and the bucket's keys. Keep the password off the box.</dd></div>314<div><dt>FORGE_BACKUP_AT, _KEEP_DAILY, _KEEP_WEEKLY</dt><dd>When the daily backup runs (UTC) and what retention keeps: 03:30, 7 and 5.</dd></div>315<div><dt>FORGE_MIRROR_S3, _KEY_ID, _SECRET, _DELAY, _MIN_INTERVAL</dt><dd>The pages mirror's bucket and keys, how long it waits after the last build, 600 seconds, and how long at least between syncs, 3600 seconds.</dd></div>316</dl></section>317<section aria-labelledby="k-env"><h3 id="k-env">the environment, for tests</h3><dl>318<div><dt>FORGE_NOW</dt><dd>Pins the clock reviews are stamped with; the demo moves it with its commits.</dd></div>319<div><dt>FORGE_NO_BUILD, FORGE_SYNC_BUILD</dt><dd>Skip the build after a push, or run it in the foreground.</dd></div>320<div><dt>FORGE_STRICT</dt><dd>1 makes a template that prints an undefined or null value fail the build; CI sets it.</dd></div>321</dl></section>322</div>323</section>324325<section class="commit" id="server" aria-labelledby="server-h">326<header><p class="ref"><b>11</b> on the server</p><h2 id="server-h"><a href="#server">Running it, as root</a></h2></header>327<p class="prose">Administer as root on the server (<code>forge-admin ...</code> in the container, which runs <code>forge</code> as the git user; on a VPS, <code>sudo <wbr>-u <wbr>git <wbr>/<wbr>opt/<wbr>forge/<wbr>bin/<wbr>forge</code>), never by logging in as <code>git</code>: every key is forced through <code>forge shell</code>. The forge's admins can run the people commands over ssh instead.</p>328<div class="cheat">329<section aria-labelledby="sv-root"><h3 id="sv-root">as root</h3><dl>330<div><dt>forge-admin user add fox 'ssh-ed25519 AAAA...'</dt><dd>Add a user and key; <code>user rm fox</code> revokes their keys and web credentials. <code>user invite</code> makes a link instead.</dd></div>331<div><dt>forge-admin repo new cat/greet "description"</dt><dd>Or <code>repo import cat/greet &lt;url&gt;</code> to fetch one server side; <code>repo rm</code> and <code>repo restore</code> for the trash.</dd></div>332<div><dt>forge-admin collaborators cat/greet add fox</dt><dd>The same lists the owner keeps over ssh, for any repo; <code>readers</code> too.</dd></div>333<div><dt>forge-admin build [owner/repo]</dt><dd>Rebuild pages; <code>metrics</code> shows where builds spend their time.</dd></div>334<div><dt>forge-admin gc [--dry-run]</dt><dd>Remove unused site and release files, hostnames and leftover previews, then repack each repo (<a href="#upkeep">upkeep</a>). Daily from cron.</dd></div>335<div><dt>forge-admin backups</dt><dd>How the last backup went, then the snapshots. <code>backup</code> takes one now; the container takes one daily.</dd></div>336<div><dt>forge-admin restore latest</dt><dd>Bring the forge back on an empty box, from the newest snapshot or a named one.</dd></div>337</dl></section>338<section aria-labelledby="sv-repo"><h3 id="sv-repo">git config, on the server</h3><dl>339<div><dt>git config forge.requireci tip</dt><dd>Require a verified pass for the new tip of <code>main</code>; <code>all</code> checks every new commit.</dd></div>340<div><dt>git config forge.requireapproval true</dt><dd>Require an approval of the new tip from someone other than the review's author.</dd></div>341<div><dt>git config forge.releaserequireci true</dt><dd>Require a verified pass for a release's tag.</dd></div>342<div><dt>git config forge.allowforce true</dt><dd>Let the owner and collaborators force-push <code>main</code>.</dd></div>343<div><dt>git config --file /srv/git/.gitconfig --add forge.admins cat</dt><dd>In the git user's own config: the forge's admins, who may <code>invite</code>, list <code>users</code> and set the <code>motd</code> over ssh.</dd></div>344</dl></section>345</div>346</section>347348<section class="commit" id="upkeep" aria-labelledby="upkeep-h">349<header><p class="ref"><b>12</b> upkeep</p><h2 id="upkeep-h"><a href="#upkeep">Keeping it tidy</a></h2></header>350<div class="prose">351<p><strong>Previews</strong> go when their branch does: <code>post-receive</code> deletes the preview ref, the served link and the hostname of a deleted branch. Previews of merged branches stay until the branch is deleted.</p>352<p><strong>The store</strong> keeps every tree that was ever deployed until <code>forge gc</code>, which removes</p>353<ul>354<li>previews of review branches that are gone (deleted outside the hook, or before it cleaned up),</li>355<li>hostnames whose deploy is gone,</li>356<li>store trees that no <code>www/sites</code> or <code>www/previews</code> link points at.</li>357</ul>358<p>A deploy extracts its tree before it links it, and touches a tree it reuses, so gc skips store entries changed in the last 10 minutes (<code>--age</code>) and is safe to run while people push. <code>--dry-run</code> shows what would go. Run it daily, from the host's crontab for the container (<code>docker <wbr>exec <wbr>forge <wbr>forge-admin <wbr>gc</code>) or the git user's on a VPS.</p>359<p><strong>Repos</strong> are maintained by the same <code>forge gc</code>, in <code>repack_repo</code>: <code>git <wbr>repack <wbr>-d <wbr>--geometric=2 <wbr>--write-midx <wbr>--write-bitmap-index</code> merges packs only until each is at least twice the next one's size, so a day's run rewrites the day's pushes and loose objects, not the repo, and one multi-pack index with a bitmap serves clones and fetches across the packs. It deletes only packs it rewrote and prunes nothing, so pushes can land while it runs. Then the commit-graph is rewritten as one layer (<code>--split=replace</code>, reusing the Bloom filters it has, adding any that old layers lack) and refs are packed, since every revision pins a loose <code>refs/forge/keep/&lt;sha&gt;</code>. Between runs, each build adds a commit-graph layer for its push.</p>360<p><strong>Repo settings</strong> come from <code>repo_setup</code>, when a repo is created (<code>repo new</code>, <code>ssh new</code>, a push to a missing repo, an import) and for every repo on <code>forge server init</code>, which the container runs on boot: the hooks, <code>receive.advertisePushOptions</code>, <code>transfer.hideRefs refs/forge</code> so clones, fetches and pushes never see the pinned revisions (the build reads them directly; pre-receive also refuses them), and <code>commitGraph.changedPathsVersion <wbr>2</code>, filters that hash non-ASCII paths the same on every platform.</p>361<p><strong>Backups</strong>: <code>forge backup</code> stages a copy of <code>/srv/git</code> without what a restore rebuilds (pages, extracted sites, caches, generated keys files, hooks, logs), uploads it with restic, prunes old snapshots, runs <code>restic check</code> and restores the new snapshot into a scratch directory to <code>git fsck</code> every repo and compare its refs. The copy is consistent mid-push: per repo it copies review logs and comments, then reads the refs, then hard-links the objects, which git writes before anything names them. It refuses a forge with no repos, or half the last backup's, so a new box can't push good snapshots out. <code>forge-admin backups</code> says how the last one went, and loudly when it failed or is a day and a half old.</p>362<p><strong>Moving</strong>: <code>forge restore latest</code> on an empty forge unpacks a snapshot and rebuilds the rest: what <code>server init</code> and <code>keys sync</code> write, hooks, pinned revisions, site files, pages. The restored tailcat key keeps the tunnel address, so the edge follows the forge to a new box by itself. The README walks through a new slate and a VPS.</p>363<p><strong>The offline copy</strong>: after builds, at most hourly, <code>forge mirror</code> syncs the forge's own pages, without sites, previews or the pages of single files and folders, to an R2 bucket. When slate doesn't answer, the edge's Caddy serves them from there with <code>X-Forge-Offline: mirror</code>, and a page the copy leaves out is a 404 that says so; sites, clones and pushes wait for slate.</p>364<p><strong>Updates</strong>: <code>deploy/slate/deploy.sh</code> rsyncs the checkout to the box and rebuilds the container; the boot build refreshes every page. A change to <code>lib/</code> or <code>site/</code> changes the code hash in every fingerprint, so that first build renders everything.</p>365<p><strong>Logs</strong>: <code>.forge/build.log</code> for builds, <code>.forge/events.log</code> for the events server, and <code>backup.log</code> and <code>mirror.log</code>, kept under a megabyte. <code>events.jsonl</code> is cut back to its last 1,000 lines once it passes a megabyte.</p>366</div>367</section>368369<section class="commit" id="hacking" aria-labelledby="hacking-h">370<header><p class="ref"><b>13</b> hacking</p><h2 id="hacking-h"><a href="#hacking">Working on forge</a></h2></header>371<div class="cheat">372<section aria-labelledby="hk-run"><h3 id="hk-run">run it</h3><dl>373<div><dt>npm install</dt><dd>11ty, markdown-it, sanitize-html, Pele; TypeScript only for checking.</dd></div>374<div><dt>sh demo/make-demo.sh</dt><dd>A throwaway forge in <code>demo/root</code>: two users, stacked reviews with revisions, CI that fails and passes, a wiki, sites and previews, all through the real hooks over file:// pushes.</dd></div>375<div><dt>node demo/serve.ts 4817</dt><dd>Serves it with live updates and cache headers, the way Caddy and the events server do.</dd></div>376<div><dt>npm run dev</dt><dd>Serves the demo from <code>_site/</code> on :8080 and rebuilds it, reloading the browser, when templates, assets or <code>lib/</code> change. It never writes to <code>demo/root</code>.</dd></div>377</dl></section>378<section aria-labelledby="hk-check"><h3 id="hk-check">check it</h3><dl>379<div><dt>sh .forge/ci</dt><dd>Lints the shell, type-checks, builds the demo, checks a few pages and checks this guide's links. It is forge's own CI, so <code>forge attest</code> runs it and signs the result.</dd></div>380<div><dt>npx tsc -p .</dt><dd>Type check only. <code>tsc</code> never emits.</dd></div>381<div><dt>node lib/check-links.ts guide</dt><dd>Every relative link and anchor in the guide resolves.</dd></div>382<div><dt>node lib/render-page.ts check &lt;www&gt;</dt><dd>Every page renders the same with plain Nunjucks as with 11ty, the base for rendering pages on demand.</dd></div>383</dl></section>384</div>385<div class="prose" style="margin-top:1.4rem">386<p>Conventions: shell stays POSIX (<code>sh -n bin/forge</code>); TypeScript uses erasable syntax only and imports name their <code>.ts</code> files, since Node strips types and runs the sources; no frameworks; comments say why. Pages must work without JavaScript, which only adds shortcuts, live updates and conveniences.</p>387<p>This guide is plain HTML and CSS with a few lines of script, and every link in it is relative, so it works on its own hostname and under any path. Edit <code>guide/</code>, run the checks and publish from <code>main</code> with <code>forge site deploy guide</code>.</p>388</div>389</section>390391</main>392</div>393394<footer class="end">395<div class="wrap">396<svg class="mark" viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="12" cy="12" r="9"/><path d="M16 15 12 7 8 15Zm-1 0L11.5 8"/></svg>397<p>Back to <a href="./">the guide</a>. The README in the repo has the same facts in plain text.</p>398</div>399</footer>400</body>401</html>