cat / forge
clone

a personal forge on plain git: static pages, stacked-diff reviews, signed local ci, repo sites and releases

Security: sign-in, invites, owner pages, lake, caps and the edge #30

merged cat/sec-hardening → cat/sec-goget, opened by cat, 1 revision, updated
cat description

The rest of the security review, stacked on #29 (go-get and headers). One commit per finding.

  1. Passkey challenges are stateless (expiry + HMAC with a per-process key, used nonces remembered until expiry), bound to the browser by a five-minute __Host-forge-challenge cookie, and /login/options is limited to 20 a minute per address.
  2. Invites check the code first; without a live invite every post gets the same expired answer, web and shell.
  3. Invite posts are counted per address, only those with a live code.
  4. Owner pages only for owners with a public repo; a partial build removes a page that no longer qualifies.
  5. repo rm deletes the slug's rows from every lake table and its sources keys (lib/lake.ts forget). Going public again publishes private-era facts with the history; documented.
  6. private and repo rm build the index synchronously before mirror_kick soon.
  7. builds.parquet reason is push or other.
  8. Events: 2,000 anonymous streams, 16 per address. headersTimeout 10s and requestTimeout 30s on the private, events and pages servers. Caddy rate limiting deferred: the container's Caddy is stock, with no rate-limit module.
  9. Edge: no slash redirect for paths starting // or /.
  10. identity_pick chmods the restored identity.db 600.
  11. Scenario: a CI note's runner is never account@hostname.
  12. Edge sends FORGE_MIRROR_TOKEN (Fly secret) as X-Forge-Mirror-Token to the mirror bucket.

Per-address limits use X-Forge-Client-IP, which Caddy sets from {client_ip}; in the container Caddy trusts Fly-Client-IP from loopback (the tailcat tunnel) only.

stack2
  #30 Security: sign-in, invites, owner pages, lake, caps and the edgecat/sec-hardening10 commits✓ 10/10merged
#29 Security: go get's reflected markup, and headers on the forge's pagescat/sec-goget2 commits✓ 2/2merged
main base
commits10oldest firstcombined diff 31 files, +724 -76
discussion0 comments

No comments yet: forge comment 30 adds one.

1/10 Sign-in challenges keep nothing on the server, belong to one browser, and are rate limited

cat, , 2133113. CI ✓ passed in 127s, signed by cat

/login/options kept challenges in a map of 1000 that dropped the oldest, so anyone could push out a sign-in in progress by asking for options in a loop. A challenge now carries its expiry and an HMAC (a key the server makes when it starts) over it, what it is for and the browser that asked, which gets a five-minute __Host-forge-challenge cookie with a random id; verify checks all of that and remembers used nonces until they expire. A challenge fetched by one client can't be completed by another. Each address may ask for options 20 times a minute (lib/limits.ts). The address is X-Forge-Client-IP, which Caddy sets from {client_ip}, replacing any a client sent; in the container Caddy trusts Fly-Client-IP from the loopback tunnel only. Scenarios: a flood of 1500 options requests doesn't break a sign-in, the per-address limit, the limit through Caddy despite a spoofed header, and a challenge refused from another browser.
11 files changed, +221 -31
MREADME.md7++-
Mbin/forge3+
Alib/limits.ts50+++++++++++++++++
Mlib/login.ts87++++++++++++++++++++++--------
Mlib/private-server.ts22++++++--
Mserver/container/start.sh6++-
Mtest/authenticator.ts22+++++++-
Atest/scenarios/login-challenge-browser.sh15+++++
Atest/scenarios/login-options-caddy.sh15+++++
Atest/scenarios/login-options-flood.sh11++++
Atest/scenarios/login-options-limit.sh14+++++
M README.md +6 -1
@@ -333,7 +333,12 @@ ssh git@git.example.com tokens [rm <id>]
333333 is offered first, so a passkey can be post-quantum. After `passkeys rm`,334334 the next page you open signed in tells your password manager which335335 passkeys still work (the WebAuthn Signal API), so it stops offering the336- removed one; it also signs out the sessions that passkey made.336+ removed one; it also signs out the sessions that passkey made. The337+ server keeps no challenges until they are used: each one carries its338+ expiry and an HMAC, and is bound to the browser that asked for it by a339+ `__Host-forge-challenge` cookie that lasts five minutes, so a flood of340+ requests can't push out anyone's sign-in. Each address may ask for341+ options 20 times a minute.337342 - **Sessions** are a `__Host-forge` cookie (Secure, HttpOnly, SameSite=Lax)338343 that lasts 30 days from its last use. Sign-out is a POST from the forge's339344 page, with a CSRF token. The events server streams a private repo to a
M bin/forge +3 -0
@@ -1089,6 +1089,9 @@ $addr {
10891089 ?Referrer-Policy strict-origin-when-cross-origin10901090 ?Content-Security-Policy "frame-ancestors 'self'; base-uri 'none'; object-src 'none'"10911091 }1092+ # The servers behind Caddy count requests by address (lib/limits.ts);1093+ # this is the one they go by, whatever a client sent in its place.1094+ request_header X-Forge-Client-IP {client_ip}10921095 # Compress pages and assets, never the event stream (it would buffer),10931096 # code.db, which code search reads in byte ranges of the file as stored,10941097 # or release downloads, which are mostly compressed already.
A lib/limits.ts +50 -0
@@ -0,0 +1,50 @@
1+// Limits for the forge's Node servers (lib/private-server.ts,2+// lib/events-server.ts, lib/pages-server.ts): who a request comes from, how3+// often someone may do something, and how long a request may take to arrive.4+import type http from 'node:http';5+6+// The address a request comes from. The servers listen on 127.0.0.1 only,7+// behind Caddy, which sets X-Forge-Client-IP to the client it saw8+// ({client_ip}: behind the edge, Fly-Client-IP; bin/forge server9+// caddyfile, server/container/start.sh), replacing any the client sent.10+// Without Caddy (demo/serve.ts, scenarios), the socket's.11+export function clientIp(req: http.IncomingMessage): string {12+ const h = req.headers['x-forge-client-ip'];13+ const v = (Array.isArray(h) ? h[0] : h)?.trim();14+ return v && v.length <= 64 ? v : req.socket.remoteAddress ?? '';15+}16+17+// At most `max` hits per key in any `ms`. At most `keys` keys are kept,18+// the least recently counted going first, so a flood of new keys costs19+// memory only up to there.20+export class Window {21+ private hits = new Map<string, number[]>();22+ private max: number;23+ private ms: number;24+ private keys: number;25+ constructor(max: number, ms: number, keys = 10000) {26+ this.max = max;27+ this.ms = ms;28+ this.keys = keys;29+ }30+31+ // Whether key may have one more now; if so, it is counted.32+ take(key: string, t = Date.now()): boolean {33+ const mine = (this.hits.get(key) ?? []).filter((x) => t - x < this.ms);34+ if (mine.length >= this.max) return false;35+ mine.push(t);36+ this.hits.delete(key);37+ this.hits.set(key, mine);38+ while (this.hits.size > this.keys) this.hits.delete(this.hits.keys().next().value!);39+ return true;40+ }41+}42+43+// How long a request's headers, and the whole request, may take to arrive:44+// a client that trickles them in holds a socket no longer than this. (A45+// response, such as the event stream, may go on as long as it likes.)46+export function timeouts(server: http.Server): http.Server {47+ server.headersTimeout = 10_000;48+ server.requestTimeout = 30_000;49+ return server;50+}
M lib/login.ts +65 -22
@@ -11,18 +11,45 @@
1111 // algorithms are the library's: ML-DSA-44 first where the runtime has it1212 // (Node 24.7 and later), then EdDSA, ES256 and RS256.1313 //14-// Challenges live in memory for five minutes and are used once. A signed-in14+// Challenges keep nothing on the server until they are used, so asking for15+// many can't push anyone else's out: each is a random nonce, its expiry16+// (five minutes on) and an HMAC over them with a key the server makes when17+// it starts, which also covers what it is for (signing in, or adding a18+// passkey with a given code) and the browser that asked. That browser gets19+// a short-lived cookie (__Host-forge-challenge) with a random id the HMAC20+// covers, so a challenge fetched by one client can't be completed by21+// another. A used challenge is remembered until it expires, so each works22+// once. A restart forgets the key: a sign-in in flight then fails, and the23+// next try works. A signed-in1524 // browser gets the user's accepted credential ids back, so the page can tell1625 // the password manager which passkeys still work (the WebAuthn Signal API,1726 // signalAllAcceptedCredentials), and a passkey the forge doesn't know gets1827 // signalUnknownCredential.1928 import type http from 'node:http';29+import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';2030 import { generateAuthenticationOptions, generateRegistrationOptions, verifyAuthenticationResponse, verifyRegistrationResponse } from '@simplewebauthn/server';2131 import type { AuthenticationResponseJSON, RegistrationResponseJSON } from '@simplewebauthn/server';2232 import type { Identity, Passkey } from './identity.ts';2333 2434 const CHALLENGE_TTL = 5 * 60 * 1000;2535 36+// The cookie that ties a challenge to the browser that asked for it: host37+// only, HTTPS only, out of scripts' reach, sent to this site's own fetches only.38+export const CHALLENGE_COOKIE = '__Host-forge-challenge';39+export const challengeCookie = (value: string): string => `${CHALLENGE_COOKIE}=${value}; Path=/; Max-Age=${CHALLENGE_TTL / 1000}; Secure; HttpOnly; SameSite=Strict`;40+// The browser's id from its cookie, if it has a well-formed one.41+export function browserId(header: string | undefined): string | null {42+ for (const part of (header ?? '').split(';')) {43+ const i = part.indexOf('=');44+ if (i > 0 && part.slice(0, i).trim() === CHALLENGE_COOKIE) {45+ const v = part.slice(i + 1).trim();46+ return /^[A-Za-z0-9_-]{22}$/.test(v) ? v : null;47+ }48+ }49+ return null;50+}51+export const newBrowserId = (): string => randomBytes(16).toString('base64url');52+2653 export interface RelyingParty {2754 id: string;2855 origin: string;
@@ -66,7 +93,9 @@ export class Login {
6693 rp: RelyingParty;6794 private id: () => Identity;6895 private isUser: (name: string) => boolean;69- private challenges = new Map<string, { kind: 'login' | 'enroll'; code?: string; expires: number }>();96+ // The HMAC key for challenges, and the nonces of used ones until they expire.97+ private key = randomBytes(32);98+ private used = new Map<string, number>();7099 71100 constructor(rp: RelyingParty, id: () => Identity, isUser: (name: string) => boolean) {72101 this.rp = rp;
@@ -74,32 +103,46 @@ export class Login {
74103 this.isUser = isUser;75104 }76105 77- private remember(challenge: string, kind: 'login' | 'enroll', code?: string): void {78- const t = Date.now();79- for (const [c, v] of this.challenges) if (v.expires < t || this.challenges.size > 1000) this.challenges.delete(c);80- this.challenges.set(challenge, { kind, code, expires: t + CHALLENGE_TTL });106+ private mac(kind: 'login' | 'enroll', code: string, browser: string, nonce: Buffer, expires: Buffer): Buffer {107+ return createHmac('sha256', this.key).update(`forge-challenge\0${kind}\0${code}\0${browser}\0`).update(nonce).update(expires).digest();81108 }82109 83- // A challenge this server gave out for this, not used or expired: used up.84- private take(challenge: string, kind: 'login' | 'enroll', code?: string): boolean {85- const v = this.challenges.get(challenge);86- if (!v) return false;87- this.challenges.delete(challenge);88- return v.kind === kind && v.code === code && v.expires >= Date.now();110+ // A challenge: nonce (16 bytes), expiry (8, ms), HMAC (32).111+ private challenge(kind: 'login' | 'enroll', browser: string, code = ''): Uint8Array<ArrayBuffer> {112+ const nonce = randomBytes(16);113+ const expires = Buffer.alloc(8);114+ expires.writeBigUInt64BE(BigInt(Date.now() + CHALLENGE_TTL));115+ return new Uint8Array(Buffer.concat([nonce, expires, this.mac(kind, code, browser, nonce, expires)]));116+ }117+118+ // A challenge this server gave this browser for this, not used or119+ // expired: used up.120+ private take(challenge: string, kind: 'login' | 'enroll', browser: string | null, code = ''): boolean {121+ const b = Buffer.from(challenge, 'base64url');122+ if (!browser || b.length !== 56) return false;123+ const nonce = b.subarray(0, 16);124+ const expires = b.subarray(16, 24);125+ if (!timingSafeEqual(b.subarray(24), this.mac(kind, code, browser, nonce, expires))) return false;126+ const t = Date.now();127+ const until = Number(expires.readBigUInt64BE());128+ const n = nonce.toString('base64url');129+ if (until < t || this.used.has(n)) return false;130+ for (const [k, v] of this.used) if (v < t) this.used.delete(k);131+ this.used.set(n, until);132+ return true;89133 }90134 91135 signal(user: string): Signal {92136 return { rpID: this.rp.id, userID: this.id().handle(user), allAcceptedCredentialIDs: this.id().passkeys(user).map((p) => p.credential) };93137 }94138 95- async loginOptions(): Promise<unknown> {96- const opts = await generateAuthenticationOptions({ rpID: this.rp.id, userVerification: 'required', allowCredentials: [] });97- this.remember(opts.challenge, 'login');98- return opts;139+ // Options for the browser whose id this is (browserId, newBrowserId).140+ async loginOptions(browser: string): Promise<unknown> {141+ return generateAuthenticationOptions({ rpID: this.rp.id, userVerification: 'required', allowCredentials: [], challenge: this.challenge('login', browser) });99142 }100143 101144 // The passkey's user, after checking its signature over our challenge.102- async loginVerify(body: unknown): Promise<{ user: string; passkey: Passkey }> {145+ async loginVerify(body: unknown, browser: string | null): Promise<{ user: string; passkey: Passkey }> {103146 const res = body as AuthenticationResponseJSON;104147 if (!res || typeof res.id !== 'string' || typeof res.response !== 'object') throw new LoginError('not a passkey response');105148 const key = this.id().passkey(res.id);
@@ -113,7 +156,7 @@ export class Login {
113156 let v;114157 try {115158 v = await verifyAuthenticationResponse({116- response: res, expectedChallenge: (c) => this.take(c, 'login'), expectedOrigin: this.rp.origin, expectedRPID: this.rp.id, requireUserVerification: true,159+ response: res, expectedChallenge: (c) => this.take(c, 'login', browser), expectedOrigin: this.rp.origin, expectedRPID: this.rp.id, requireUserVerification: true,117160 credential: { id: key.credential, publicKey: new Uint8Array(key.public_key), counter: key.counter, transports: key.transports ? key.transports.split(',') : undefined },118161 });119162 } catch (e) {
@@ -124,7 +167,7 @@ export class Login {
124167 return { user: key.user, passkey: key };125168 }126169 127- async enrollOptions(code: unknown): Promise<unknown> {170+ async enrollOptions(code: unknown, browser: string): Promise<unknown> {128171 const e = typeof code === 'string' ? this.id().enrollment(code) : null;129172 if (!e || !this.isUser(e.user)) throw new LoginError('This link has expired or has been used. Run ssh passkey again for a new one.', { expired: true });130173 const opts = await generateRegistrationOptions({
@@ -133,19 +176,19 @@ export class Login {
133176 attestationType: 'none',134177 excludeCredentials: this.id().passkeys(e.user).map((p) => ({ id: p.credential, transports: p.transports ? p.transports.split(',') : undefined })),135178 authenticatorSelection: { residentKey: 'required', userVerification: 'required' },179+ challenge: this.challenge('enroll', browser, code as string),136180 });137- this.remember(opts.challenge, 'enroll', code as string);138181 return opts;139182 }140183 141184 // Check the new passkey, then use the link up and keep the passkey.142- async enrollVerify(code: unknown, body: unknown, name: string): Promise<{ user: string; passkey: Passkey }> {185+ async enrollVerify(code: unknown, body: unknown, name: string, browser: string | null): Promise<{ user: string; passkey: Passkey }> {143186 const res = body as RegistrationResponseJSON;144187 if (typeof code !== 'string' || !res || typeof res.id !== 'string' || typeof res.response !== 'object') throw new LoginError('not a passkey response');145188 let v;146189 try {147190 v = await verifyRegistrationResponse({148- response: res, expectedChallenge: (c) => this.take(c, 'enroll', code), expectedOrigin: this.rp.origin, expectedRPID: this.rp.id, requireUserVerification: true,191+ response: res, expectedChallenge: (c) => this.take(c, 'enroll', browser, code), expectedOrigin: this.rp.origin, expectedRPID: this.rp.id, requireUserVerification: true,149192 });150193 } catch (e) {151194 throw new LoginError(`The passkey wasn't added: ${(e as Error).message}`);
M lib/private-server.ts +17 -5
@@ -37,7 +37,8 @@ import { fileURLToPath } from 'node:url';
3737 import { INVITE_EXPIRED, Identity, SESSION_TTL, TOKEN_PREFIX, cookieValue, csrfToken, sameSecret, sessionCookie, type Session, type Token } from './identity.ts';3838 import { canRead, isPublic, readable, repoDir, users } from './access.ts';3939 import { DOWNLOAD, DOWNLOAD_HEADERS, resolve, sendFile } from './static.ts';40-import { Login, LoginError, agentName, readBody, relyingParty } from './login.ts';40+import { Login, LoginError, agentName, browserId, challengeCookie, newBrowserId, readBody, relyingParty } from './login.ts';41+import { Window, clientIp } from './limits.ts';4142 import { pageRenderer, type Render } from './render-page.ts';4243 import { assets } from './assets.ts';4344 import { summarize } from './repo.ts';
@@ -190,6 +191,8 @@ const refusal = (err: string, otherwise: string): string => err.split('\n').filt
190191 // (the codes can't be guessed; this keeps anyone from trying), and one191192 // accept at a time, since each writes .forge/keys.192193 const INVITE_POSTS_PER_MINUTE = 30;194+// Sign-in options: each address may ask this many times a minute.195+const OPTIONS_PER_MINUTE = 20;193196 194197 // The sign-in and list pages: no framing, scripts and styles of our own only.195198 const PAGE_HEADERS = {
@@ -435,6 +438,7 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
435438 json(res, 200, { user, enroll: gate.id.enroll(user, '').code });436439 }437440 441+ const optionsAsked = new Window(OPTIONS_PER_MINUTE, 60_000);438442 async function post(req: http.IncomingMessage, res: http.ServerResponse, url: URL): Promise<void> {439443 // Only the forge's own pages may post here (and sign anyone in or out).440444 if (!login || req.headers.origin !== login.rp.origin) return json(res, 403, { error: 'not from this forge' });
@@ -453,17 +457,25 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
453457 } catch {454458 return json(res, 400, { error: 'not JSON' });455459 }460+ // Options are tied to this browser's challenge cookie (lib/login.ts),461+ // which it gets with them, and each address asks only so often.462+ const browser = browserId(req.headers.cookie);463+ const options = async (make: (b: string) => Promise<unknown>): Promise<void> => {464+ if (!optionsAsked.take(clientIp(req))) return json(res, 429, { error: 'That is a lot at once: wait a minute' }, { 'Retry-After': '60' });465+ const b = browser ?? newBrowserId();466+ json(res, 200, await make(b), { 'Set-Cookie': challengeCookie(b) });467+ };456468 try {457- if (url.pathname === '/login/options') return json(res, 200, await login.loginOptions());469+ if (url.pathname === '/login/options') return await options((b) => login.loginOptions(b));458470 if (url.pathname === '/login/verify') {459- const { user, passkey } = await login.loginVerify(body);471+ const { user, passkey } = await login.loginVerify(body, browser);460472 return json(res, 200, { user, signal: login.signal(user) }, signedIn(req, user, passkey.id));461473 }462474 if (url.pathname === '/login/invite/options') return await invite(res, 'options', body);463475 if (url.pathname === '/login/invite/accept') return await invite(res, 'accept', body);464- if (url.pathname === '/login/enroll/options') return json(res, 200, await login.enrollOptions(body.code));476+ if (url.pathname === '/login/enroll/options') return await options((b) => login.enrollOptions(body.code, b));465477 if (url.pathname === '/login/enroll/verify') {466- const { user, passkey } = await login.enrollVerify(body.code, body.response, agentName(req.headers['user-agent']));478+ const { user, passkey } = await login.enrollVerify(body.code, body.response, agentName(req.headers['user-agent']), browser);467479 return json(res, 200, { user, signal: login.signal(user) }, signedIn(req, user, passkey.id));468480 }469481 } catch (e) {
M server/container/start.sh +5 -1
@@ -105,7 +105,11 @@ as_git $forge keys sync
105105 # TLS is terminated in front of us; Caddy serves plain HTTP on 8080. The106106 # config is written anew on every boot, so Caddy keeps no copy of it.107107 {108- printf '{\n\tauto_https off\n\thttp_port 8080\n\tadmin off\n\tpersist_config off\n}\n\n'108+ # Requests come from the edge through tailcat, which connects from109+ # loopback; Fly's proxy in front of the edge puts the client's address110+ # in Fly-Client-IP, which {client_ip} is then (X-Forge-Client-IP,111+ # lib/limits.ts). Nothing else reaches Caddy here.112+ printf '{\n\tauto_https off\n\thttp_port 8080\n\tadmin off\n\tpersist_config off\n\tservers {\n\t\ttrusted_proxies static 127.0.0.1/8 ::1/128\n\t\tclient_ip_headers Fly-Client-IP\n\t}\n}\n\n'109113 as_git $forge server caddyfile :8080 "$pages" "${pages:+http://$pages}"110114 } >/etc/caddy/Caddyfile111115
M test/authenticator.ts +20 -2
@@ -16,6 +16,10 @@
1616 // --no-uv no user verification1717 // --counter <n> the signature counter to send (default: one more than last time)1818 // --replay send the same signed response a second time (prints the second answer)19+// --other-browser send the signed response from another browser: without20+// the challenge cookie the options came with21+// --flood <n> before signing in, n other clients (each its own22+// address) ask for sign-in options1923 // --alg ml-dsa-44 enroll an ML-DSA-44 key instead of ES2562024 import fs from 'node:fs';2125 import { createHash, createPrivateKey, generateKeyPairSync, randomBytes, sign, type KeyObject } from 'node:crypto';
@@ -47,6 +51,8 @@ const rpOverride = flag('--rp');
4751 const counterArg = flag('--counter');4852 const noUv = has('--no-uv');4953 const replay = has('--replay');54+const otherBrowser = has('--other-browser');55+const flood = Number(flag('--flood') ?? 0);5056 const alg = flag('--alg') ?? 'es256';5157 const [mode, server, ...rest] = args;5258
@@ -58,8 +64,12 @@ const u32 = (n: number): Buffer => {
5864 return b;5965 };6066 67+// The challenge cookie the options came with, sent back as a browser would.68+let jar = '';6169 async function call(path: string, body: unknown): Promise<{ status: number; data: Record<string, unknown>; cookie: string | null }> {62- const r = await fetch(server.replace(/\/$/, '') + path, { method: 'POST', headers: { 'Content-Type': 'application/json', Origin: origin, 'User-Agent': 'test-authenticator' }, body: JSON.stringify(body) });70+ const headers: Record<string, string> = { 'Content-Type': 'application/json', Origin: origin, 'User-Agent': 'test-authenticator' };71+ if (jar && !(otherBrowser && /\/verify$/.test(path))) headers.Cookie = jar;72+ const r = await fetch(server.replace(/\/$/, '') + path, { method: 'POST', headers, body: JSON.stringify(body) });6373 const text = await r.text();6474 let data: Record<string, unknown> = {};6575 try {
@@ -68,6 +78,8 @@ async function call(path: string, body: unknown): Promise<{ status: number; data
6878 data = { error: text };6979 }7080 const set = r.headers.get('set-cookie') ?? '';81+ const c = /__Host-forge-challenge=([^;]*)/.exec(set);82+ if (c) jar = `__Host-forge-challenge=${c[1]}`;7183 const m = /__Host-forge=([^;]*)/.exec(set);7284 return { status: r.status, data, cookie: m ? m[1] : null };7385 }
@@ -119,6 +131,12 @@ if (mode === 'enroll' && rest.length === 2) {
119131 const opts = await call('/login/options', {});120132 if (opts.status !== 200) done(opts);121133 const o = opts.data as { challenge: string; rpId: string };134+ for (let i = 0; i < flood; i++) {135+ const r = await fetch(server.replace(/\/$/, '') + '/login/options', {136+ method: 'POST', headers: { 'Content-Type': 'application/json', Origin: origin, 'X-Forge-Client-IP': `10.${(i >> 16) & 255}.${(i >> 8) & 255}.${i & 255}` }, body: '{}',137+ });138+ if (r.status !== 200) done({ status: r.status, data: { error: `flood request ${i}` }, cookie: null });139+ }122140 const counter = counterArg != null ? Number(counterArg) : key.counter + 1;123141 const authData = Buffer.concat([sha256(rpOverride ?? o.rpId), Buffer.from([flags(false)]), u32(counter)]);124142 const clientDataJSON = Buffer.from(JSON.stringify({ type: 'webauthn.get', challenge: o.challenge, origin: clientOrigin, crossOrigin: false }));
@@ -133,6 +151,6 @@ if (mode === 'enroll' && rest.length === 2) {
133151 if (replay) done(await call('/login/verify', response));134152 done(r);135153 } else {136- console.error('usage: node test/authenticator.ts enroll <server> <code> <key.json> | login <server> <key.json> [--origin o] [--client-origin o] [--rp id] [--no-uv] [--counter n] [--replay]');154+ console.error('usage: node test/authenticator.ts enroll <server> <code> <key.json> | login <server> <key.json> [--origin o] [--client-origin o] [--rp id] [--no-uv] [--counter n] [--replay] [--other-browser] [--flood n]');137155 process.exit(2);138156 }
A test/scenarios/login-challenge-browser.sh +15 -0
@@ -0,0 +1,15 @@
1+# A sign-in challenge belongs to the browser that asked for it: the options2+# come with a short-lived cookie the challenge is bound to, so a response3+# signed over it and sent from another browser (without that cookie) is4+# refused, and the same sign-in from the browser that asked works.5+. "$(dirname "$0")/../lib.sh"6+user dami7+export FORGE_BASE_URL=https://forge.test8+private_server9+http -s -i -X POST -H 'Origin: https://forge.test' -d '{}' "$PRIV/login/options" | tr -d '\r' >"$T/opts"10+has "$T/opts" '^Set-Cookie: __Host-forge-challenge=[A-Za-z0-9_-]{22}; Path=/; Max-Age=300; Secure; HttpOnly; SameSite=Strict$'11+code=$(ssh_as dami passkey | sed -n 's/^ .*#enroll=//p')12+fails_with 'HTTP 400: .*challenge' node "$T_HOME/test/authenticator.ts" enroll "$PRIV" "$code" "$T/dami.key" --other-browser13+ok node "$T_HOME/test/authenticator.ts" enroll "$PRIV" "$code" "$T/dami.key"14+fails_with 'HTTP 400: .*challenge' node "$T_HOME/test/authenticator.ts" login "$PRIV" "$T/dami.key" --other-browser15+ok node "$T_HOME/test/authenticator.ts" login "$PRIV" "$T/dami.key"
A test/scenarios/login-options-caddy.sh +15 -0
@@ -0,0 +1,15 @@
1+# Through Caddy, the address the servers count by is the one Caddy saw2+# (X-Forge-Client-IP, set from {client_ip}): a client that sends its own3+# X-Forge-Client-IP, a new one each time, is still one address, and the4+# sign-in options limit stops it after 20.5+. "$(dirname "$0")/../lib.sh"6+user dami7+export FORGE_BASE_URL=https://forge.test8+caddy_server9+ask() { http -s -o /dev/null -w '%{http_code}' -X POST -H 'Origin: https://forge.test' -H "X-Forge-Client-IP: 198.51.100.$1" -d '{}' "$C/login/options"; }10+n=011+while [ "$n" -lt 20 ]; do12+ eq "$(ask "$n")" 20013+ n=$((n + 1))14+done15+eq "$(ask 99)" 429
A test/scenarios/login-options-flood.sh +11 -0
@@ -0,0 +1,11 @@
1+# Sign-in options keep nothing on the server until they are used, so a2+# flood of requests for them (here 1500, each from an address of its own,3+# more than the 1000 challenges the server once held) can't push out the4+# challenge of someone signing in meanwhile.5+. "$(dirname "$0")/../lib.sh"6+user dami7+export FORGE_BASE_URL=https://forge.test8+private_server9+code=$(ssh_as dami passkey | sed -n 's/^ .*#enroll=//p')10+ok node "$T_HOME/test/authenticator.ts" enroll "$PRIV" "$code" "$T/dami.key"11+ok node "$T_HOME/test/authenticator.ts" login "$PRIV" "$T/dami.key" --flood 1500
A test/scenarios/login-options-limit.sh +14 -0
@@ -0,0 +1,14 @@
1+# Each address may ask for sign-in options 20 times a minute; the next is2+# a 429 that says to wait. Another address is counted apart.3+. "$(dirname "$0")/../lib.sh"4+user dami5+export FORGE_BASE_URL=https://forge.test6+private_server7+ask() { http -s -o /dev/null -w '%{http_code}' -X POST -H 'Origin: https://forge.test' -H "X-Forge-Client-IP: $1" -d '{}' "$PRIV/login/options"; }8+n=09+while [ "$n" -lt 20 ]; do10+ eq "$(ask 192.0.2.1)" 20011+ n=$((n + 1))12+done13+eq "$(ask 192.0.2.1)" 42914+eq "$(ask 192.0.2.2)" 200

2/10 Invites check the code first, and count posts per address

cat, , 473a10f. CI ✓ passed in 118s, signed by cat

/login/invite/accept and forge user accept checked the key before the code, so anyone could ask whether a key had an account here ("that key is already on this forge"). The code is checked first now, on the web and in the shell, and without a live invite every post gets the same expired answer, whatever its key. The rate limit was one bucket for the whole forge that every post counted against, so junk posts could use up an invitee's turns: posts are counted per address now, 10 a minute, and only those with a live invite's code. Scenarios: a dead code answers the same for any key, and the per-address, live-only count.
6 files changed, +87 -24
MREADME.md9+++++---
Mbin/forge5++++-
Mlib/identity.ts9++++++--
Mlib/private-server.ts34++++++++++++++----------------
Atest/scenarios/people-invite-key-hidden.sh30++++++++++++++++++++++++++
Atest/scenarios/people-invite-limit.sh24+++++++++++++++++++++
M README.md +6 -3
@@ -171,9 +171,12 @@ in keys, the home directory, `keys sync`. With `--repo`, the new user is
171171 that repo's reader or collaborator too. Then the same page offers to add a172172 passkey, as the link `ssh passkey` prints does. The code is in the URL's173173 fragment and identity.db keeps only its hash, like enrollment codes. The174-posts come only from the forge's own page (Origin), at most 30 a minute175-across the forge, one taking a key at a time. `user add` checks a key the176-same way, so no key adds a second line to keys.174+posts come only from the forge's own page (Origin), at most 10 a minute from175+one address with a live invite's code, one taking a key at a time. The code176+is checked first: without a live invite every post gets the same answer,177+whatever its key, so the page can't be asked whether a key has an account,178+and posts with dead codes aren't counted. `user add` checks a key the same179+way, so no key adds a second line to keys.177180 178181 #### Admins179182
M bin/forge +4 -1
@@ -1365,7 +1365,9 @@ user_invite() {
13651365 }13661366 13671367 # user accept: an invite's code and an SSH public key, on stdin, a line1368-# each, from /login/ (lib/private-server.ts). The key is checked before the1368+# each, from /login/ (lib/private-server.ts). The code is checked first:1369+# without a live invite, every key gets the same refusal, so nobody learns1370+# from it whether a key is on this forge. The key is checked before the13691371 # invite is used up, so a bad one leaves the link working; then the name13701372 # gets it as forge user add gives one, and the invite's repo, if any. Prints13711373 # the new user's name.
@@ -1373,6 +1375,7 @@ user_accept() {
13731375 load_config13741376 IFS= read -r code || die "usage: forge user accept (code and key on stdin)"13751377 key=$(cat)1378+ node "$FORGE_HOME/lib/identity.ts" invitation "$code" || exit 113761379 key_ok "$key" || die "not an SSH public key (one line: ssh-ed25519 AAAA... [comment])"13771380 o=$(key_owner "$key")13781381 [ -z "$o" ] || die "that key is already on this forge"
M lib/identity.ts +7 -2
@@ -655,6 +655,11 @@ function operatorCommand(state: string, cmd: string, args: string[], out: (s: st
655655 case 'invited':656656 if (args.length !== 1) fail('usage: invited <name>');657657 return id.invited(args[0]) ? 0 : 1;658+ case 'invitation':659+ // Is the code a live invite's? Without using it up.660+ if (args.length !== 1) fail('usage: invitation <code>');661+ if (!id.invitation(args[0])) fail(INVITE_EXPIRED);662+ return 0;658663 case 'accept': {659664 if (args.length !== 1) fail('usage: accept <code>');660665 const i = id.accept(args[0]);
@@ -695,12 +700,12 @@ if (process.argv[1] && fs.realpathSync(process.argv[1]) === fileURLToPath(import
695700 const r = Identity.check(args[0]);696701 if (r !== 'ok') fail(`${args[0]}: ${r}`);697702 }698- else if (['invite', 'invites', 'invited', 'accept', 'users'].includes(cmd)) process.exitCode = operatorCommand(state, cmd, args, (s) => process.stdout.write(s));703+ else if (['invite', 'invites', 'invited', 'invitation', 'accept', 'users'].includes(cmd)) process.exitCode = operatorCommand(state, cmd, args, (s) => process.stdout.write(s));699704 else if (['passkey', 'passkeys', 'sessions', 'token', 'tokens'].includes(cmd)) {700705 const user = process.env.FORGE_USER;701706 if (!user) fail('no FORGE_USER');702707 sshCommand(state, user!, cmd, args, (s) => process.stdout.write(s), (s) => process.stderr.write(s));703- } else fail('usage: node lib/identity.ts passkey|passkeys|sessions|token|tokens ... | invite|invites|invited|accept|users ... | init | purge <user> | snapshot <file> | changed <file> | check <file>');708+ } else fail('usage: node lib/identity.ts passkey|passkeys|sessions|token|tokens ... | invite|invites|invited|invitation|accept|users ... | init | purge <user> | snapshot <file> | changed <file> | check <file>');704709 } catch (e) {705710 if (!(e instanceof Usage)) throw e;706711 process.stderr.write(`forge: ${e.message}\n`);
M lib/private-server.ts +16 -18
@@ -187,10 +187,12 @@ function forgeCmd(cfg: PrivateConfig, args: string[], input: string, extra: Node
187187 // What bin/forge said it refused: its "forge: " lines.188188 const refusal = (err: string, otherwise: string): string => err.split('\n').filter((l) => l.startsWith('forge: ')).map((l) => l.slice(7)).join('\n') || otherwise;189189 190-// Invites: the forge as a whole takes at most this many posts a minute191-// (the codes can't be guessed; this keeps anyone from trying), and one192-// accept at a time, since each writes .forge/keys.193-const INVITE_POSTS_PER_MINUTE = 30;190+// Invites: each address may post this many times a minute with a live191+// invite's code, and one accept runs at a time, since each writes192+// .forge/keys. A dead code (the codes can't be guessed) costs a lookup and193+// gets the same answer whatever was wrong, so it isn't counted: nobody can194+// use up an invitee's turns by posting junk.195+const INVITE_POSTS_PER_MINUTE = 10;194196 // Sign-in options: each address may ask this many times a minute.195197 const OPTIONS_PER_MINUTE = 20;196198
@@ -405,23 +407,19 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
405407 // checks and adds the way forge user add does, using the invite up, and406408 // answers with a code for the passkey enrollment that follows on the407409 // same page (the one ssh passkey would print). Nobody is signed in by it.408- const invitePosts: number[] = [];410+ // The code is checked first: without a live invite every post gets the411+ // same answer, whatever its key, so nothing is learned about keys.412+ const invitePosts = new Window(INVITE_POSTS_PER_MINUTE, 60_000);409413 let accepting = false;410- async function invite(res: http.ServerResponse, what: 'options' | 'accept', posted: unknown): Promise<void> {414+ async function invite(req: http.IncomingMessage, res: http.ServerResponse, what: 'options' | 'accept', posted: unknown): Promise<void> {411415 const body = (posted && typeof posted === 'object' ? posted : {}) as Record<string, unknown>;412- const t = Date.now();413- while (invitePosts.length && t - invitePosts[0] >= 60_000) invitePosts.shift();414- if (invitePosts.length >= INVITE_POSTS_PER_MINUTE) return json(res, 429, { error: 'That is a lot at once: wait a minute' }, { 'Retry-After': '60' });415- invitePosts.push(t);416416 const code = typeof body.code === 'string' && /^[A-Za-z0-9_-]{1,64}$/.test(body.code) ? body.code : '';417- if (what === 'options') {418- const i = code ? gate.id.invitation(code) : null;419- if (!i) return json(res, 400, { error: INVITE_EXPIRED, expired: true });420- return json(res, 200, { user: i.user, repo: i.repo, role: i.role });421- }417+ const i = code ? gate.id.invitation(code) : null;418+ if (!i) return json(res, 400, { error: INVITE_EXPIRED, expired: true });419+ if (!invitePosts.take(clientIp(req))) return json(res, 429, { error: 'That is a lot at once: wait a minute' }, { 'Retry-After': '60' });420+ if (what === 'options') return json(res, 200, { user: i.user, repo: i.repo, role: i.role });422421 const key = typeof body.key === 'string' ? body.key.trim() : '';423422 if (!key || key.length > 8192 || /[\r\n]/.test(key)) return json(res, 400, { error: 'Paste one SSH public key: the one line of your .pub file' });424- if (!code) return json(res, 400, { error: INVITE_EXPIRED, expired: true });425423 if (accepting) return json(res, 503, { error: 'The forge is busy: try again in a moment' }, { 'Retry-After': '5' });426424 accepting = true;427425 let r;
@@ -471,8 +469,8 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
471469 const { user, passkey } = await login.loginVerify(body, browser);472470 return json(res, 200, { user, signal: login.signal(user) }, signedIn(req, user, passkey.id));473471 }474- if (url.pathname === '/login/invite/options') return await invite(res, 'options', body);475- if (url.pathname === '/login/invite/accept') return await invite(res, 'accept', body);472+ if (url.pathname === '/login/invite/options') return await invite(req, res, 'options', body);473+ if (url.pathname === '/login/invite/accept') return await invite(req, res, 'accept', body);476474 if (url.pathname === '/login/enroll/options') return await options((b) => login.enrollOptions(body.code, b));477475 if (url.pathname === '/login/enroll/verify') {478476 const { user, passkey } = await login.enrollVerify(body.code, body.response, agentName(req.headers['user-agent']), browser);
A test/scenarios/people-invite-key-hidden.sh +30 -0
@@ -0,0 +1,30 @@
1+# Without a live invite, posting a key to /login/invite/accept gets the same2+# answer whatever the key: one already on the forge, a new one or not a key3+# at all. So the page can't be asked whether a key has an account. The4+# shell side (forge user accept) checks the code first too.5+. "$(dirname "$0")/../lib.sh"6+user cat7+export FORGE_BASE_URL=https://forge.test8+private_server9+ssh-keygen -q -t ed25519 -N '' -C fox -f "$T/keys/fox"10+accept() {11+ node -e 'console.log(JSON.stringify({ code: process.argv[1], key: process.argv[2] }))' -- "$1" "$2" >"$T/body"12+ http -s -i -H 'Origin: https://forge.test' -H 'Content-Type: application/json' -d @"$T/body" "$PRIV/login/invite/accept" | grep -av '^Date: '13+}14+accept nonsense "$(cat "$T/keys/cat.pub")" >"$T/known"15+has "$T/known" '^HTTP/1.1 400 '16+has "$T/known" 'expired, been used or been taken back'17+lacks "$T/known" 'already on this forge'18+for key in "$(cat "$T/keys/fox.pub")" hello ""; do19+ accept nonsense "$key" >"$T/other"20+ cmp -s "$T/known" "$T/other" || { diff "$T/known" "$T/other" >&2 || true; fail "a dead invite answers differently for key '$key'"; }21+done22+# A code that was live and is used up answers the same.23+ok forge user invite fox24+code=$(sed -n 's/^ .*#invite=//p' "$T_LOG")25+ok forge user invites rm fox26+accept "$code" "$(cat "$T/keys/cat.pub")" >"$T/other"27+cmp -s "$T/known" "$T/other" || fail "a taken-back invite answers differently"28+printf '%s\n%s\n' nonsense "$(cat "$T/keys/cat.pub")" | forge user accept >"$T_LOG" 2>&1 && fail "user accept took a dead code"29+has "$T_LOG" 'expired, been used or been taken back'30+lacks "$T_LOG" 'already on this forge'
A test/scenarios/people-invite-limit.sh +24 -0
@@ -0,0 +1,24 @@
1+# Invite posts are counted per address, and only those with a live invite's2+# code: 10 a minute, then a 429. Junk codes, however many, are never3+# counted, so they can't use up an invitee's turns, and another address4+# has its own count.5+. "$(dirname "$0")/../lib.sh"6+user cat7+export FORGE_BASE_URL=https://forge.test8+private_server9+ok forge user invite fox10+code=$(sed -n 's/^ .*#invite=//p' "$T_LOG")11+ask() { http -s -o /dev/null -w '%{http_code}' -H 'Origin: https://forge.test' -H 'Content-Type: application/json' -H "X-Forge-Client-IP: $1" -d "{\"code\":\"$2\"}" "$PRIV/login/invite/options"; }12+n=013+while [ "$n" -lt 40 ]; do14+ eq "$(ask 192.0.2.9 "junk$n")" 40015+ n=$((n + 1))16+done17+n=018+while [ "$n" -lt 10 ]; do19+ eq "$(ask 192.0.2.9 "$code")" 20020+ n=$((n + 1))21+done22+eq "$(ask 192.0.2.9 "$code")" 42923+eq "$(ask 192.0.2.9 junk)" 40024+eq "$(ask 192.0.2.10 "$code")" 200

3/10 Event streams are capped, and the Node servers time out slow requests

cat, , 56690dd. CI ✓ passed in 117s, signed by cat

The events server took any number of open streams. Now at most FORGE_EVENTS_MAX (2,000) are open for people not signed in to a private repo's stream, and FORGE_EVENTS_PER_IP (16) from one address; past either it answers 503 with Retry-After. The private, events and pages servers give a request's headers 10 seconds and the whole request 30 to arrive (a response, such as the event stream, may run on). Caddy has no rate-limit module in the container's build, so per-address limits in Caddy are left for later. Scenario: the per-address and overall caps, and a closed stream freeing its place.
5 files changed, +72 -9
MREADME.md2+-
Mlib/events-server.ts31+++++++++++++++++++++---
Mlib/pages-server.ts5++--
Mlib/private-server.ts4++--
Atest/scenarios/events-caps.sh39++++++++++++++++++++++++++++++
M README.md +1 -1
@@ -99,7 +99,7 @@ review page, signed in ─── HTTPS ────────────> the
9999 | sshd | git over ssh. Every key is forced through `forge shell <name>`. |100100 | Caddy | Serves `.forge/www` as plain files, read-only HTTPS clones (`git http-backend` through fcgiwrap; nothing else inside a bare repo is served; a private repo takes a token) and `go get` meta tags, with security headers (HSTS, nosniff, a referrer policy and a CSP against framing) on the forge's own pages; proxies `/events/`, every path `www` has no file for (to the private server) and, with a pages domain, sites. |101101 | `forge build` | Runs 11ty after each push, in the background and one repo at a time, then swaps the new pages into `www/`. Unchanged pages are hard links to the last build's. First it adds the push's commits to the repo's commit-graph (see [Cleaning up](#cleaning-up)). |102-| `lib/events-server.ts` | Tails `.forge/events.jsonl`, which hooks and builds append to, and streams it as server-sent events at `/events/<owner>/<repo>/`. Open pages say who pushed and swap in fresh content when the build finishes. Without it pages still work; their status line says offline. A private repo streams only to its readers, signed in. |102+| `lib/events-server.ts` | Tails `.forge/events.jsonl`, which hooks and builds append to, and streams it as server-sent events at `/events/<owner>/<repo>/`. Open pages say who pushed and swap in fresh content when the build finishes. Without it pages still work; their status line says offline. A private repo streams only to its readers, signed in. At most 2,000 streams are open for people not signed in, and 16 from one address. |103103 | `lib/private-server.ts` | Serves private repos' pages to their readers, signed in with a passkey or holding a token, the sign-in pages (`/login/`, `/private/`), and the 404 of every page that isn't there; takes comments and approvals posted from review pages (see [Comments and approvals](#comments-and-approvals)); answers Caddy whether a token may clone a private repo over HTTPS (see [Private pages](#private-pages)). |104104 | `lib/pages-server.ts` | Only with a pages domain. Serves each site on its own hostname the way Cloudflare Pages does (`_headers`, `_redirects`, pretty URLs, the nearest `404.html` or a single-page-app fallback) and redirects old `/sites/...` paths there. Runs as the caddy user, since deploys are untrusted content. |105105 | `forge backup` | Daily, from a loop in `start.sh`. Snapshots the forge with restic into a bucket, then checks the snapshots and test-restores the new one (see [Backups and moving hosts](#backups-and-moving-hosts)). |
M lib/events-server.ts +27 -4
@@ -14,12 +14,20 @@
1414 // git-daemon-export-ok at each event, not only from the event's flag, and1515 // the session and the right to read are checked again too, so making a1616 // repo private, removing a reader or signing out takes effect at once.17+//18+// Streams stay open, so there are only so many: at most ANON_MAX for19+// people not signed in to a private repo's stream, and PER_IP_MAX from any20+// one address (lib/limits.ts clientIp). Past either, a 503 with Retry-After.1721 import http from 'node:http';1822 import fs from 'node:fs';1923 import path from 'node:path';2024 import { fileURLToPath } from 'node:url';2125 import { canRead, isPublic as flagged, repoDir, users } from './access.ts';2226 import { Identity, cookieValue } from './identity.ts';27+import { clientIp, timeouts } from './limits.ts';28+29+export const ANON_MAX = Number(process.env.FORGE_EVENTS_MAX) || 2000;30+export const PER_IP_MAX = Number(process.env.FORGE_EVENTS_PER_IP) || 16;2331 2432 export interface ForgeEvent {2533 t: number;
@@ -38,6 +46,7 @@ interface Client {
3846 repo: string | null;3947 // Watching a private repo: the session that may.4048 session: string | null;49+ ip: string;4150 res: http.ServerResponse;4251 ping: NodeJS.Timeout;4352 }
@@ -83,9 +92,16 @@ export function serveEvents(state: string, port: number, root = path.dirname(sta
8392 const dir = repoDir(root, repo);8493 return !!s && !!dir && !flagged(dir) && users(state).has(s.user) && canRead(dir, s.user);8594 };95+ // Open streams by address, and those without a session.96+ const perIp = new Map<string, number>();97+ let anon = 0;8698 const drop = (c: Client): void => {99+ if (!clients.delete(c)) return;87100 clearInterval(c.ping);88- clients.delete(c);101+ const n = (perIp.get(c.ip) ?? 1) - 1;102+ if (n > 0) perIp.set(c.ip, n);103+ else perIp.delete(c.ip);104+ if (!c.session) anon--;89105 };90106 const broadcast = (ev: ForgeEvent): void => {91107 if (ev.private || !isPublic(root, ev.repo)) {
@@ -132,7 +148,7 @@ export function serveEvents(state: string, port: number, root = path.dirname(sta
132148 };133149 fs.watchFile(file, { interval: 300 }, poll);134150 135- return http.createServer((req, res) => {151+ return timeouts(http.createServer((req, res) => {136152 const m = /^\/events\/(?:([A-Za-z0-9_.-]+)\/([A-Za-z0-9_.-]+)\/?)?$/.exec(new URL(req.url ?? '/', 'http://x').pathname);137153 const repo = m?.[1] ? `${m[1]}/${m[2]}` : null;138154 const secret = cookieValue(req.headers.cookie);
@@ -142,14 +158,21 @@ export function serveEvents(state: string, port: number, root = path.dirname(sta
142158 res.writeHead(404).end();143159 return;144160 }161+ const ip = clientIp(req);162+ if ((perIp.get(ip) ?? 0) >= PER_IP_MAX || (!session && anon >= ANON_MAX)) {163+ res.writeHead(503, { 'Retry-After': '60', 'Cache-Control': 'no-store' }).end();164+ return;165+ }145166 res.writeHead(200, { 'Content-Type': 'text/event-stream; charset=utf-8', 'Cache-Control': 'no-cache', 'X-Accel-Buffering': 'no' });146167 res.write('retry: 5000\n');147168 // Connected and idle: nothing to show until someone pushes.148169 res.write('data: \n\n');149- const client: Client = { repo, session, res, ping: setInterval(() => res.write(': ping\n\n'), 20000) };170+ const client: Client = { repo, session, ip, res, ping: setInterval(() => res.write(': ping\n\n'), 20000) };150171 clients.add(client);172+ perIp.set(ip, (perIp.get(ip) ?? 0) + 1);173+ if (!session) anon++;151174 req.on('close', () => drop(client));152- }).listen(port, '127.0.0.1', () => console.log(`events on 127.0.0.1:${port}, following ${file}`));175+ })).listen(port, '127.0.0.1', () => console.log(`events on 127.0.0.1:${port}, following ${file}`));153176 }154177 155178 // Run directly: node lib/events-server.ts
M lib/pages-server.ts +3 -2
@@ -15,6 +15,7 @@ import path from 'node:path';
1515 import { fileURLToPath } from 'node:url';1616 import { Hosts, hostsConfig, parseTarget } from './hosts.ts';1717 import type { HostsConfig } from './hosts.ts';18+import { timeouts } from './limits.ts';1819 1920 const TYPES: Record<string, string> = {2021 html: 'text/html; charset=utf-8', htm: 'text/html; charset=utf-8', css: 'text/css; charset=utf-8',
@@ -290,7 +291,7 @@ export function servePages(cfg: HostsConfig, port: number): http.Server {
290291 }291292 292293 const suffix = cfg.pagesDomain ? `.${cfg.pagesDomain}` : null;293- return http.createServer((req, res) => {294+ return timeouts(http.createServer((req, res) => {294295 try {295296 if (req.method !== 'GET' && req.method !== 'HEAD') {296297 res.writeHead(405, { Allow: 'GET, HEAD' }).end();
@@ -349,7 +350,7 @@ export function servePages(cfg: HostsConfig, port: number): http.Server {
349350 if (!res.headersSent) res.writeHead(500).end();350351 else res.destroy();351352 }352- }).listen(port, '127.0.0.1', () => console.log(`pages on 127.0.0.1:${port} for *${suffix ?? ' (paths only)'}, from ${cfg.www}`));353+ })).listen(port, '127.0.0.1', () => console.log(`pages on 127.0.0.1:${port} for *${suffix ?? ' (paths only)'}, from ${cfg.www}`));353354 }354355 355356 // Run directly: node lib/pages-server.ts
M lib/private-server.ts +2 -2
@@ -38,7 +38,7 @@ import { INVITE_EXPIRED, Identity, SESSION_TTL, TOKEN_PREFIX, cookieValue, csrfT
3838 import { canRead, isPublic, readable, repoDir, users } from './access.ts';3939 import { DOWNLOAD, DOWNLOAD_HEADERS, resolve, sendFile } from './static.ts';4040 import { Login, LoginError, agentName, browserId, challengeCookie, newBrowserId, readBody, relyingParty } from './login.ts';41-import { Window, clientIp } from './limits.ts';41+import { Window, clientIp, timeouts } from './limits.ts';4242 import { pageRenderer, type Render } from './render-page.ts';4343 import { assets } from './assets.ts';4444 import { summarize } from './repo.ts';
@@ -509,7 +509,7 @@ export function privateHandler(cfg: PrivateConfig): http.RequestListener {
509509 }510510 511511 export function servePrivate(cfg: PrivateConfig, port: number): http.Server {512- return http.createServer(privateHandler(cfg)).listen(port, '127.0.0.1', () => console.log(`private pages on 127.0.0.1:${port}, from ${cfg.priv}`));512+ return timeouts(http.createServer(privateHandler(cfg))).listen(port, '127.0.0.1', () => console.log(`private pages on 127.0.0.1:${port}, from ${cfg.priv}`));513513 }514514 515515 // Run directly: node lib/private-server.ts
A test/scenarios/events-caps.sh +39 -0
@@ -0,0 +1,39 @@
1+# The events server holds only so many streams open: at most2+# FORGE_EVENTS_PER_IP from one address and FORGE_EVENTS_MAX in all for3+# anyone not signed in (here 2 and 3); past either, a 503 with4+# Retry-After. A stream that closes frees its place.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo dami/open8+: >"$FORGE_ROOT/.forge/events.jsonl"9+# Another scenario may take the free port first: then try another.10+tries=011+while :; do12+ port=$(free_port)13+ FORGE_STATE=$FORGE_ROOT/.forge FORGE_EVENTS_PORT=$port FORGE_EVENTS_PER_IP=2 FORGE_EVENTS_MAX=3 node "$T_HOME/lib/events-server.ts" >"$T/events.log" 2>&1 &14+ pid=$!15+ n=016+ until grep -q "^events on 127.0.0.1:$port" "$T/events.log" || ! kill -0 "$pid" 2>/dev/null || [ "$n" -ge 100 ]; do sleep 0.1; n=$((n + 1)); done17+ if kill -0 "$pid" 2>/dev/null && grep -q '^events on' "$T/events.log"; then break; fi18+ kill "$pid" 2>/dev/null || true19+ tries=$((tries + 1))20+ [ "$tries" -lt 5 ] || fail "the events server did not start"21+done22+T_PIDS="$T_PIDS $pid"23+es=http://127.0.0.1:$port/events24+# Streams from an address, held open for a few seconds.25+hold() { http -sN --max-time 4 -H "X-Forge-Client-IP: $1" "$es/" >/dev/null & T_PIDS="$T_PIDS $!"; }26+code() { http -s -o /dev/null -w '%{http_code}' --max-time 1 -H "X-Forge-Client-IP: $1" "$es/" || true; }27+hold 192.0.2.128+hold 192.0.2.129+sleep 130+eq "$(code 192.0.2.1)" 50331+hold 192.0.2.232+sleep 133+# Three open: no more for anyone not signed in.34+eq "$(code 192.0.2.3)" 50335+http -s -i --max-time 1 -H 'X-Forge-Client-IP: 192.0.2.3' "$es/" | tr -d '\r' >"$T/res" || true36+has "$T/res" '^Retry-After: 60$'37+# Once they close, there is room again.38+sleep 439+eq "$(code 192.0.2.1)" 200

4/10 Only owners with a public repo have a page

cat, , 51330a1. CI ✓ passed in 162s, signed by cat

Every directory under the forge's root got an owner page, and forge user add makes one for each new user, so /<name>/ said who had an account. Owner pages are built only for owners with at least one public repo now, and a build of the index (or of one repo, or @owner) takes away the page of an owner who no longer has one, whose last repo went private or was removed. Scenario: no page for a user with no repos or only private ones, and one that goes when the last public repo does.
4 files changed, +48 -2
MREADME.md4+++-
Mbin/forge16++++++++++++++++
Mlib/forge.ts6+++++-
Atest/scenarios/people-owner-page.sh24++++++++++++++++++++++++
M README.md +3 -1
@@ -132,7 +132,9 @@ it:
132132 `git`).133133 134134 Repos live at `/srv/git/<owner>/<repo>.git`, and `<owner>` owns them. Anyone135-can read a public repo; a private one, only the people below. The owner and135+can read a public repo; a private one, only the people below. An owner's136+page, `/<owner>/`, is there only while they have a public repo, so a user's137+name alone isn't published. The owner and136138 anyone listed in `<repo>.git/forge/collaborators` can push anything. Other137139 users who can read it can push branches and previews under their own name138140 (`fox/...`) and add CI notes, which must fast-forward. Nobody can force-push
M bin/forge +16 -0
@@ -2897,6 +2897,9 @@ cmd_build() {
28972897 (trap - EXIT && export GIT_DIR="$FORGE_ROOT/$only.git" && OWNER=${only%%/*} NAME=${only#*/} && releases_into "$out") ||28982898 say "forge: linking the release files of $only failed"28992899 fi2900+ # Owner pages were built if the index was (the swap moves it).2901+ owners_built=02902+ [ ! -f "$out/index.html" ] || owners_built=129002903 # Swap generated pages in. sites/, previews/ and hosts/ belong to the hooks. A29012904 # single-repo build refreshes owner index pages but only replaces that repo.29022905 for e in "$out"/*; do
@@ -2915,6 +2918,19 @@ cmd_build() {
29152918 swap "$e" "$FORGE_WWW/$b"29162919 fi29172920 done2921+ # An owner without a public repo has no page (lib/forge.ts): one whose2922+ # last went private or away loses theirs.2923+ if [ -n "$only" ] && [ "$owners_built" = 1 ]; then2924+ for p in "$FORGE_WWW"/*/index.html; do2925+ [ -f "$p" ] || continue2926+ o=$(basename "$(dirname "$p")")2927+ case $o in assets | sites | previews | hosts | insights | design | topics) continue ;; esac2928+ case $only in @*) [ "$o" = "${only#@}" ] || continue ;; esac2929+ [ ! -f "$out/$o/index.html" ] || continue2930+ rm -f "$p" "$FORGE_WWW/$o/activity.json"2931+ rmdir "$FORGE_WWW/$o" 2>/dev/null || true2932+ done2933+ fi29182934 # Made private while this was building: its pages go after all. Public:29192935 # its private pages go, if it had any.29202936 if [ "$only" != - ] && [ -f "$FORGE_ROOT/$only.git/HEAD" ]; then
M lib/forge.ts +5 -1
@@ -470,7 +470,11 @@ export function loadForge(env: NodeJS.ProcessEnv = process.env): Forge {
470470 471471 // The index's panels; a private build has no index.472472 const lately = priv ? { items: [], active: {} } : recent(cfg, summaries);473- const owners = ownerNames.filter((o) => !cfg.only?.startsWith('@') || cfg.only === `@${o}`).map((owner) => ownerPage(cfg, owner, summaries.filter((s) => s.owner === owner)));473+ // Only owners with a public repo have a page: a user's name alone (forge474+ // user add makes their directory) is nobody's business.475+ const owners = ownerNames476+ .filter((o) => (!cfg.only?.startsWith('@') || cfg.only === `@${o}`) && summaries.some((s) => s.owner === o))477+ .map((owner) => ownerPage(cfg, owner, summaries.filter((s) => s.owner === owner)));474478 for (const o of owners) {475479 if (!priv && o.activity) raw.push({ path: `/${o.owner}/activity.json`, data: Buffer.from(JSON.stringify(o.activity.data) + '\n') });476480 }
A test/scenarios/people-owner-page.sh +24 -0
@@ -0,0 +1,24 @@
1+# Only owners with a public repo have a page. A user with no repos, or2+# only private ones, has none, so /<name>/ doesn't say who has an account;3+# an owner whose last public repo goes private loses theirs at the next4+# build, as one whose repo is removed does.5+. "$(dirname "$0")/../lib.sh"6+user dami7+user ana8+user bob9+repo dami/open10+repo --private bob/hidden11+build12+exists "$FORGE_WWW/dami/index.html"13+missing "$FORGE_WWW/ana"14+missing "$FORGE_WWW/bob"15+lacks "$FORGE_WWW/index.html" 'ana|bob'16+repo ana/notes17+build ana/notes18+exists "$FORGE_WWW/ana/index.html"19+sync_builds20+ok ssh_as ana private ana/notes21+missing "$FORGE_WWW/ana"22+exists "$FORGE_WWW/dami/index.html"23+ok ssh_as dami rm dami/open --yes24+missing "$FORGE_WWW/dami"

5/10 A repo gone private or removed leaves the index before the mirror syncs, and a removed one leaves the lake

cat, , 0573918. CI ✓ passed in 140s, signed by cat

ssh private and repo rm started the index and owner build in the background and then the mirror's sync within 30 seconds, while that build could still be waiting on build.lock, so the listing could linger in the offline copy. They build the index before returning now (rebuild_index), unless builds are off. repo rm deleted only the repo's exports, so its rows stayed in the lake and came back in the exports of a new repo under the same name: lib/lake.ts forget takes them out of every table, with the lake's note of where it read the repo's sources, under the lake's lock. Scenarios: the listing is gone when the command returns, and a reused name starts with no history.
5 files changed, +132 -7
MREADME.md4++-
Mbin/forge35+++++++++++++++++++++++-----
Mlib/lake.ts37+++++++++++++++++++++++++++++
Atest/scenarios/lake-repo-rm.sh38++++++++++++++++++++++++++++++
Atest/scenarios/mirror-private-index.sh25++++++++++++++++++++
M README.md +3 -1
@@ -239,7 +239,9 @@ as before.
239239 forge (without `--yes` it says what would go); collaborators can't, and240240 `forge repo rm <owner>/<repo>` does the same as root. Its clones, pages in241241 either root, sites, previews, hostnames, release downloads and insights go242-at once, and a mirror sync within 30 seconds deletes its pages. The bare repo waits in242+at once, as do its rows in the lake, so a new repo of the same name starts243+with no history. The index and owner pages are rebuilt before the command244+returns, and a mirror sync within 30 seconds deletes its pages. The bare repo waits in243245 `.forge/trash/<owner>/<repo>.git.<time>` for `FORGE_TRASH_DAYS` (14) days,244246 backed up like the rest of the forge. `forge repo restore <owner>/<repo>`245247 puts back the newest removal if the name is free, and `forge gc` deletes it
M bin/forge +29 -6
@@ -1473,9 +1473,9 @@ cmd_repo() {
14731473 # trash like any other state. Everything that served it goes now, the way14741474 # going private takes a repo's public side away: its pages in either root14751475 # (with its release downloads), its sites, previews and their hostnames,1476-# its insights exports and its caches. The index, owner pages and1477-# repos.json are rebuilt without it, and the mirror syncs within 30 seconds,1478-# which deletes its pages from the bucket.1476+# its insights exports, its rows in the lake and its caches. The index,1477+# owner pages and repos.json are rebuilt without it before this returns, and1478+# the mirror syncs within 30 seconds, which deletes its pages from the bucket.14791479 repo_rm() {14801480 owner=${1%%/*} name=${1#*/}14811481 valid_name "$owner" && valid_name "$name" && [ "$owner/$name" = "$1" ] && [ -f "$FORGE_ROOT/$1.git/HEAD" ] || die "no repo $1"
@@ -1495,7 +1495,8 @@ repo_rm() {
14951495 rm -rf "${FORGE_WWW:?}/$1" "${FORGE_PRIVATE_WWW:?}/$1" "${FORGE_WWW:?}/sites/$1" "${FORGE_WWW:?}/previews/$1" \14961496 "${FORGE_STATE:?}/lake/export/$1" "${FORGE_STATE:?}/cache/repos/$1.json" "${FORGE_STATE:?}/cache/code/$1"14971497 rm -f "$FORGE_WWW"/insights/*.json "$FORGE_WWW"/insights/*.parquet1498- rebuild - removed1498+ lake_forget "$1" || say "forge: the lake still has $1's rows ($FORGE_STATE/lake.log)"1499+ rebuild_index removed14991500 mirror_kick soon15001501 echo "removed $1; forge repo restore $1 brings it back within ${FORGE_TRASH_DAYS:-14} days (the trash: ${trash#"$FORGE_ROOT"/}/$name.git.$t)"15011502 }
@@ -1807,6 +1808,16 @@ rebuild() {
18071808 fi18081809 }18091810 1811+# rebuild_index <reason>: the index and owner pages, built before this1812+# returns (unless builds are off). For a repo gone private or removed, so1813+# its listing is gone from www before mirror_kick soon syncs the mirror: a1814+# build in the background could still be waiting on build.lock by then.1815+rebuild_index() {1816+ [ "${FORGE_NO_BUILD:-0}" != 1 ] || return 01817+ FORGE_BUILD_REASON=$1 "$FORGE_BIN" build - >>"$FORGE_STATE/build.log" 2>&1 ||1818+ say "forge: the index build failed ($FORGE_STATE/build.log); forge build - again"1819+}1820+18101821 # forge jev [owner/repo]: ask Jev about what the repo's pages show and store18111822 # the answers (lib/jev-fill.ts). Without TYPESAFE_API_KEY it does nothing,18121823 # and a failure never stops the build that follows. Private repos have no
@@ -1999,7 +2010,7 @@ redeploy_sites() {
19992010 20002011 # private <owner>/<repo>: only its owner, collaborators and readers can see it20012012 # from now on. Its pages go, the index, owner pages and repos.json are2002-# rebuilt without it, its sites, previews and their hostnames go unless2013+# rebuilt without it before this returns, its sites, previews and their hostnames go unless20032014 # forge.publicsites, and the mirror syncs within 30 seconds, which deletes20042015 # them from the bucket. Its pages are built again into the private root,20052016 # for its readers, and its release downloads move there. The events server
@@ -2027,7 +2038,7 @@ sc_private() {
20272038 # The forge's insights count public repos only; they are written again20282039 # without this one after the build.20292040 rm -f "$FORGE_WWW"/insights/*.json "$FORGE_WWW"/insights/*.parquet2030- rebuild - private2041+ rebuild_index private20312042 rebuild "$1" private20322043 mirror_kick soon20332044 echo "$1 is private: only ${1%%/*}, its collaborators and its readers (readers $1 add <user>) can clone it, and see its pages once they sign in."
@@ -2985,6 +2996,18 @@ lake_env() {
29852996 esac29862997 }29872998 2999+# lake_forget <owner>/<repo>: a removed repo's rows out of the lake (lib/lake.ts3000+# forget), under the lake's lock, if there is a lake.3001+lake_forget() {3002+ [ -f "$FORGE_STATE/lake/catalog.sqlite" ] || return 03003+ (3004+ lake_env3005+ take_lock "$FORGE_STATE/lake.lock" "${FORGE_LAKE_WAIT:-300}" || exit 13006+ trap 'rm -rf "$FORGE_STATE/lake.lock"' EXIT3007+ node "$FORGE_HOME/lib/lake.ts" forget "$1" >>"$FORGE_STATE/lake.log" 2>&13008+ )3009+}3010+29883011 cmd_lake() {29893012 load_config29903013 lake_env
M lib/lake.ts +37 -0
@@ -17,6 +17,7 @@
1717 // node lib/lake.ts query <sql> [--json|--csv] read-only, no file access1818 // node lib/lake.ts install install the DuckDB extensions (the Dockerfile runs it)1919 // node lib/lake.ts stage <dir> a consistent copy of the lake for a backup20+// node lib/lake.ts forget <owner>/<repo> take a removed repo's rows out (forge repo rm)2021 import fs from 'node:fs';2122 import os from 'node:os';2223 import path from 'node:path';
@@ -228,6 +229,33 @@ interface SourceState extends Offset {
228229 229230 const offsetOf = (s: SourceState | undefined): Offset | null => (s ? { ino: s.ino, pos: s.pos, head: s.head, tail: s.tail } : null);230231 232+// A removed repo (forge repo rm): its rows in every table, and where the233+// lake read its sources up to, so a new repo under the same name starts234+// with none of them, and its exports with none of the removed one's235+// history. events.jsonl and metrics.jsonl were read past its lines236+// already, so they don't come back. Returns how many rows went.237+export async function forget(lake: Lake, slug: string): Promise<number> {238+ const s = lit(slug);239+ const where: Record<string, string> = {240+ events: `repo = ${s}`, builds: `scope = ${s}`, build_repos: `repo = ${s}`, reviews: `repo = ${s}`,241+ review_ends: `repo = ${s}`, commits: `repo = ${s}`, ci: `repo = ${s}`,242+ sources: `source IN (${['main', 'commits', 'notes'].map((k) => lit(`${k} ${slug}`)).join(', ')}) OR starts_with(source, ${lit(`review ${slug} `)})`,243+ };244+ let n = 0;245+ await lake.run('BEGIN TRANSACTION');246+ try {247+ for (const [t, w] of Object.entries(where)) {248+ if (t !== 'sources') n += Number((await lake.rows(`SELECT count(*) FROM lake.${t} WHERE ${w}`))[0]?.[0] ?? 0);249+ await lake.run(`DELETE FROM lake.${t} WHERE ${w}`);250+ }251+ await lake.run('COMMIT');252+ } catch (e) {253+ await lake.run('ROLLBACK').catch(() => {});254+ throw e;255+ }256+ return n;257+}258+231259 // Add what is new in every source. With dry, the same reads and inserts run232260 // and are rolled back, so it says what a run would add.233261 export async function ingest(lake: Lake, { dry = false, now = Date.now() }: { dry?: boolean; now?: number } = {}): Promise<Summary> {
@@ -558,6 +586,15 @@ async function main(argv: string[]): Promise<void> {
558586 const sql = rest.filter((a) => a !== '--json' && a !== '--csv').join(' ');559587 if (!sql.trim()) throw new LakeError('usage: forge lake query "<sql>" [--json|--csv]');560588 process.stdout.write(await query(cfg, sql, fmt));589+ } else if (cmd === 'forget') {590+ if (rest.length !== 1 || !/^[^/]+\/[^/]+$/.test(rest[0])) throw new LakeError('usage: node lib/lake.ts forget <owner>/<repo>');591+ if (!fs.existsSync(cfg.catalog)) return;592+ const lake = await Lake.open(cfg, { write: true });593+ try {594+ console.log(`lake: forgot ${plural(await forget(lake, rest[0]), 'row')} of ${rest[0]}`);595+ } finally {596+ lake.close();597+ }561598 } else if (cmd === 'install') {562599 await install(cfg);563600 } else if (cmd === 'stage') {
A test/scenarios/lake-repo-rm.sh +38 -0
@@ -0,0 +1,38 @@
1+# forge repo rm takes the repo's rows out of every lake table, and the2+# lake's note of where it read the repo's sources up to. So a private repo3+# removed and a new public one made under its name don't share a history:4+# the new one's insights have only its own facts.5+. "$(dirname "$0")/../lib.sh"6+user dami7+user ana8+repo --private dami/greet9+mkdir -p "$FORGE_ROOT/dami/greet.git/forge" && echo ana >"$FORGE_ROOT/dami/greet.git/forge/collaborators"10+clone dami dami/greet "$T/w/d"11+at "$T/w/d" commit "secret plans"12+push_as dami "$T/w/d" origin main13+clone ana dami/greet "$T/w/a"14+at "$T/w/a" commit "idea"15+push_as ana "$T/w/a" origin HEAD:ana/idea16+build dami/greet17+ok forge lake18+for t in events build_repos reviews commits; do19+ [ "$(lq "SELECT count(*) FROM $t WHERE repo = 'dami/greet'")" -gt 0 ] || fail "no $t rows to forget"20+done21+[ "$(lq "SELECT count(*) FROM builds WHERE scope = 'dami/greet'")" -gt 0 ] || fail "no builds to forget"22+ok forge repo rm dami/greet23+for t in events build_repos reviews review_ends commits ci; do24+ eq "$(lq "SELECT count(*) FROM $t WHERE repo = 'dami/greet'")" 025+done26+eq "$(lq "SELECT count(*) FROM builds WHERE scope = 'dami/greet'")" 027+eq "$(lq "SELECT count(*) FROM sources WHERE source LIKE '% dami/greet' OR source LIKE '% dami/greet %'")" 028+29+repo dami/greet30+clone dami dami/greet "$T/w/n"31+at "$T/w/n" commit "fresh start"32+push_as dami "$T/w/n" origin main33+build dami/greet34+ok forge lake35+eq "$(lq "SELECT string_agg(subject, ',') FROM commits WHERE repo = 'dami/greet'")" "fresh start"36+eq "$(lq "SELECT count(*) FROM reviews WHERE repo = 'dami/greet'")" 037+has "$FORGE_WWW/dami/greet/insights/insights.json" '"pushes":1,'38+lacks "$FORGE_WWW/dami/greet/insights/insights.json" 'ana/idea|secret plans'
A test/scenarios/mirror-private-index.sh +25 -0
@@ -0,0 +1,25 @@
1+# Going private or being removed takes a repo off the index, its owner's2+# page and repos.json before the command returns, not in a build in the3+# background: the mirror's sync within 30 seconds (mirror_kick soon) must4+# not find the listing still there.5+. "$(dirname "$0")/../lib.sh"6+user dami7+repo dami/greet8+repo dami/other9+for r in greet other; do10+ clone dami "dami/$r" "$T/w/$r"11+ at "$T/w/$r" commit "first"12+ push_as dami "$T/w/$r" origin main13+done14+build15+has "$FORGE_WWW/repos.json" '"dami/greet"'16+# Builds in the background from here, as on the server.17+export FORGE_NO_BUILD=018+ok ssh_as dami private dami/greet19+lacks "$FORGE_WWW/repos.json" '"dami/greet"'20+lacks "$FORGE_WWW/index.html" 'dami/greet'21+lacks "$FORGE_WWW/dami/index.html" 'href="/dami/greet/"'22+has "$FORGE_WWW/repos.json" '"dami/other"'23+ok ssh_as dami rm dami/other --yes24+lacks "$FORGE_WWW/repos.json" '"dami/other"'25+missing "$FORGE_WWW/dami"

6/10 Insights exports give a build's reason as push or other

cat, , 43f0ccc. CI ✓ passed in 158s, signed by cat

builds.parquet carried each build's reason, and the reasons private and removed said when a repo went private or away. The exports now say push, or other for everything else; the lake keeps the real reason. lib/insights.ts and the README also say what happens to a repo's facts when it goes public again: they are published with it, from while it was private too, as its whole history is then. Scenario: the lake has private and removed, the exports only push and other.
3 files changed, +35 -3
MREADME.md5++++-
Mlib/insights.ts10++++++++--
Atest/scenarios/insights-build-reason.sh23+++++++++++++++++++++++
M README.md +4 -1
@@ -895,7 +895,10 @@ page and exports are in its pages in the private root, which `forge lake`
895895 keeps fresh the same way, for its readers only (see [Private896896 pages](#private-pages)); the mirror never sees them. Making a repo private897897 removes the forge's exports from `www` at once; the next run writes them898-without it.898+without it. Making it public again publishes its facts from while it was899+private as well, as its whole history becomes public then. The builds900+exports give a build's reason as `push` or `other`, since the others would901+say when a repo went private or away.899902 900903 The page has no query console. [DuckDB-Wasm](https://duckdb.org/docs/api/wasm/overview)901904 could run SQL over those files in the browser, but its build is 34.2 MiB
M lib/insights.ts +8 -2
@@ -9,7 +9,13 @@
99 // whose readers alone see them. publishInsights also puts them straight into1010 // the live pages after each run, since the pages fetch them and need no1111 // rebuild. The forge's cover public repos only: a private repo's facts are12-// in no public export.12+// in no public export. A repo made public again publishes its facts from13+// while it was private too, as it publishes all of its history then (the14+// secret check of ssh public reads every commit first); a removed one's15+// leave the lake (forge repo rm, lib/lake.ts forget), so a new repo of the16+// same name starts with none. A build's reason is "push" or "other" in the17+// exports: the others (private, removed, profile...) would say when a repo18+// went private or away.1319 import fs from 'node:fs';1420 import path from 'node:path';1521 import { listRepos } from './lake-sources.ts';
@@ -94,7 +100,7 @@ async function exportScope(lake: Lake, s: Scope, publicRepos: string[]): Promise
94100 await lake.run(`COPY (${sql}) TO ${lit(path.join(tmp, file))} (FORMAT parquet, COMPRESSION zstd)`);95101 };96102 await copy('events.parquet', `SELECT t, repo, type, "user", review, refs, ms, rendered FROM lake.events WHERE ${s.where} ORDER BY t`);97- await copy('builds.parquet', `SELECT id, "at", code, reason, scope, ms_total, ms_data, ms_render, rss_mb, written, linked FROM lake.builds WHERE ${s.builds} ORDER BY "at"`);103+ await copy('builds.parquet', `SELECT id, "at", code, CASE reason WHEN 'push' THEN 'push' ELSE 'other' END AS reason, scope, ms_total, ms_data, ms_render, rss_mb, written, linked FROM lake.builds WHERE ${s.builds} ORDER BY "at"`);98104 await copy('reviews.parquet', `99105 SELECT repo, review, kind, ts, "user", rev, sha, base, tree, branch, title, NULL::BOOLEAN AS observed FROM lake.reviews WHERE ${s.where}100106 UNION ALL SELECT repo, review, kind, ts, "user", NULL, sha, NULL, NULL, NULL, NULL, observed FROM lake.review_ends WHERE ${s.where}
A test/scenarios/insights-build-reason.sh +23 -0
@@ -0,0 +1,23 @@
1+# The builds.parquet exports say why a build ran only as push or other:2+# the lake keeps the real reason (private, removed, motd...), which would3+# say when a repo went private or away.4+. "$(dirname "$0")/../lib.sh"5+user dami6+repo dami/open7+clone dami dami/open "$T/w/open"8+at "$T/w/open" commit "first"9+ok push_as dami "$T/w/open" origin main10+FORGE_BUILD_REASON=push ok forge build dami/open11+FORGE_BUILD_REASON=private ok forge build -12+FORGE_BUILD_REASON=removed ok forge build -13+ok forge lake14+eq "$(lq "SELECT string_agg(DISTINCT reason, ' ' ORDER BY reason) FROM builds")" "private push removed"15+pq() { (cd "$T_HOME" && node --input-type=module -e "16+import { DuckDBInstance } from '@duckdb/node-api';17+const db = await DuckDBInstance.create(':memory:');18+const c = await db.connect();19+const r = await c.runAndReadAll(\"SELECT string_agg(DISTINCT reason, ' ' ORDER BY reason) FROM read_parquet('\" + process.argv[1] + \"')\");20+console.log(r.getRowsJson()[0][0] ?? '');21+" "$1"); }22+eq "$(pq "$FORGE_WWW/insights/builds.parquet")" "other push"23+eq "$(pq "$FORGE_WWW/dami/open/insights/builds.parquet")" "push"

7/10 Offline, the edge never redirects to another site's address

cat, , 835172a. CI ✓ passed in 123s, signed by cat

The edge's offline copy added a folder's slash with redir {path}/, so //evil.example/x answered 308 to //evil.example/x/, another site. A path that starts // or /\ (decoded) isn't redirected now; it is asked of the copy as it is and gets its 404. Caddy's path matchers see the path with its slashes merged, so the check reads {path} itself. Scenario: a folder still gets its slash, and //, /%5C and /%2F paths get no Location.
2 files changed, +34 -0
Mdeploy/edge/Caddyfile5+++++
Atest/scenarios/edge-offline-redirect.sh29+++++++++++++++++++++++++++++
M deploy/edge/Caddyfile +5 -0
@@ -44,9 +44,14 @@
4444 # Like the forge's file server: /a/b -> /a/b/, /a/b/ -> /a/b/index.html.4545 # Release downloads are files whatever their names (greet-linux,4646 # SHA256SUMS), and download rather than open, as from the forge.47+ # Never a path that starts // or /\ (decoded): redirected, that48+ # would be another site's address (//evil.example/x/). Path49+ # matchers see the path cleaned (one slash), so this looks at50+ # the path itself.4751 @bare {4852 path_regexp ^(.*/)?[^./]+$4953 not path_regexp ^/[^/]+/[^/]+/releases/download/.54+ not vars_regexp {path} ^/[/\\]5055 }5156 redir @bare {path}/ 3085257 @download path_regexp ^/[^/]+/[^/]+/releases/download/.
A test/scenarios/edge-offline-redirect.sh +29 -0
@@ -0,0 +1,29 @@
1+# Offline, the edge adds the slash a folder's address lacks (/a/b -> /a/b/),2+# but never for a path that starts // or /\, where the slash would make3+# Location another site's address (//evil.test/x/): those are asked of the4+# copy as they are, and get its 404.5+. "$(dirname "$0")/../lib.sh"6+command -v caddy >/dev/null 2>&1 || { note "caddy is not installed: skipped running it"; exit 0; }7+mkdir -p "$T/bucket/dami/greet"8+echo '<p>greet</p>' >"$T/bucket/dami/greet/index.html"9+mport=$(free_port)10+printf '{\n\tadmin off\n\tauto_https off\n\tpersist_config off\n\tdefault_bind 127.0.0.1\n}\n\n:%s {\n\troot * %s\n\tfile_server\n}\n' "$mport" "$T/bucket" >"$T/Caddyfile.bucket"11+eport=$(free_port)12+dead=$(free_port)13+# On 127.0.0.1 only, so no other scenario's server can take the port there.14+sed "s/^:8080 {/:$eport {/; s/127\.0\.0\.1:8081/127.0.0.1:$dead/" "$T_HOME/deploy/edge/Caddyfile" |15+ awk '{ print } $0 == "\tadmin off" { print "\tdefault_bind 127.0.0.1" }' >"$T/Caddyfile.edge"16+for c in bucket edge; do17+ FORGE_DOMAIN=src.test FORGE_MIRROR_URL=http://127.0.0.1:$mport XDG_DATA_HOME=$T/xdg XDG_CONFIG_HOME=$T/xdg \18+ caddy run --config "$T/Caddyfile.$c" --adapter caddyfile >"$T/caddy.$c.log" 2>&1 &19+ T_PIDS="$T_PIDS $!"20+done21+wait_http "http://127.0.0.1:$mport/"22+wait_http "http://127.0.0.1:$eport/"23+e=http://127.0.0.1:$eport24+eq "$(http -s -o /dev/null -w '%{http_code} %{redirect_url}' -H 'Host: src.test' "$e/dami/greet")" "308 $e/dami/greet/"25+for p in //evil.test/x //evil.test /%5Cevil.test/x /%2Fevil.test/x; do26+ http -s -i -H 'Host: src.test' "$e$p" | tr -d '\r' >"$T/res"27+ lacks "$T/res" '^Location:'28+ has "$T/res" '^HTTP/1.1 404 '29+done

8/10 The edge can send the mirror bucket a token

cat, , 3e6ffab. CI ✓ passed in 120s, signed by cat

With the Fly secret FORGE_MIRROR_TOKEN set, the edge sends it as X-Forge-Mirror-Token on each request for the offline copy, so a Cloudflare WAF rule in front of the bucket can later refuse requests without it. Without the secret the header is empty. Scenario: the bucket sees the token, and the reply doesn't carry it.
5 files changed, +45 -1
MREADME.md4+++-
Mdeploy/edge/Caddyfile4++++
Mdeploy/edge/fly.toml3+++
Mdeploy/edge/start.sh2++
Atest/scenarios/edge-mirror-token.sh33++++++++++++++++++++++++++++++
M README.md +3 -1
@@ -954,7 +954,9 @@ updates and pushes need slate; they get a 503, or for ssh no connection,
954954 until it's back. Caddy checks slate every 10 seconds, so only the first955955 request after it goes away waits. `deploy/edge/fly.toml` sets `FORGE_DOMAIN`956956 (only that host falls back) and `FORGE_MIRROR_URL`, the mirror's public URL;957-without it there is no fallback.957+without it there is no fallback. With the Fly secret `FORGE_MIRROR_TOKEN`958+set, the edge sends it to the bucket as `X-Forge-Mirror-Token`, so a rule in959+front of the bucket can refuse requests without it.958960 959961 On slate, everything lives in `/srv/forge` (`src/`, `data/`, `forge.env`):960962
M deploy/edge/Caddyfile +4 -0
@@ -8,6 +8,9 @@
88 # FORGE_DOMAIN the forge's hostname, e.g. src.rest; only it falls back99 # FORGE_MIRROR_URL the mirror bucket's public URL, e.g. https://mirror.src.rest;1010 # without it there is no fallback11+# FORGE_MIRROR_TOKEN (a Fly secret, optional) sent to the bucket as12+# X-Forge-Mirror-Token, so a rule in front of it (a13+# Cloudflare WAF rule) can turn away requests without it1114 {1215 admin off1316 auto_https off
@@ -64,6 +67,7 @@
6467 rewrite @file {path}?6568 reverse_proxy {$FORGE_MIRROR_URL:https://mirror.invalid} {6669 header_up Host {upstream_hostport}70+ header_up X-Forge-Mirror-Token "{$FORGE_MIRROR_TOKEN:}"6771 header_down -Set-Cookie6872 # Not in the copy: a page that isn't there, or one the copy6973 # leaves out (a file's or a folder's; lib/mirror.ts). The
M deploy/edge/fly.toml +3 -0
@@ -5,6 +5,9 @@
55 # fly certs add src.rest && fly certs add pages.src.rest66 # While slate is down, pages come from the R2 mirror (Caddyfile): set77 # FORGE_MIRROR_URL below to the mirror bucket's public URL. ssh just fails.8+# With fly secrets set FORGE_MIRROR_TOKEN=..., the edge sends that as9+# X-Forge-Mirror-Token to the bucket, so a WAF rule there can refuse10+# requests without it.811 app = "dami-forge"912 primary_region = "dfw"1013
M deploy/edge/start.sh +2 -0
@@ -2,6 +2,8 @@
22 # TAILCAT_CLIENT_KEY (secret): this edge's client key file contents.33 # FORGE_TAILCAT_ADDR (secret): the forge container's tailcat address.44 # FORGE_DOMAIN, FORGE_MIRROR_URL (fly.toml env): the offline copy (Caddyfile).5+# FORGE_MIRROR_TOKEN (secret, optional): sent to the mirror bucket with each6+# request for the offline copy, for a rule in front of the bucket to require.57 set -eu68 : "${TAILCAT_CLIENT_KEY:?set the edge client key secret}"79 : "${FORGE_TAILCAT_ADDR:?set the forge tailcat address secret}"
A test/scenarios/edge-mirror-token.sh +33 -0
@@ -0,0 +1,33 @@
1+# With FORGE_MIRROR_TOKEN set (a Fly secret), the edge sends it to the2+# mirror bucket as X-Forge-Mirror-Token on each request for the offline3+# copy, so a rule in front of the bucket can refuse requests without it.4+. "$(dirname "$0")/../lib.sh"5+command -v caddy >/dev/null 2>&1 || { note "caddy is not installed: skipped running it"; exit 0; }6+# The bucket: answers every request, and writes down the token it came7+# with. It takes a port of its own and says which.8+node -e '9+const fs = require("fs");10+const s = require("http").createServer((req, res) => {11+ fs.appendFileSync(process.argv[1], `${req.url} ${req.headers["x-forge-mirror-token"] ?? "-"}\n`);12+ res.writeHead(200, { "Content-Type": "text/html" }).end("<p>copy</p>");13+}).listen(0, "127.0.0.1", () => fs.writeFileSync(process.argv[2], String(s.address().port)));14+' "$T/seen" "$T/mport" &15+T_PIDS="$T_PIDS $!"16+n=017+until [ -s "$T/mport" ] || [ "$n" -ge 50 ]; do sleep 0.1; n=$((n + 1)); done18+mport=$(cat "$T/mport")19+eport=$(free_port)20+dead=$(free_port)21+# On 127.0.0.1 only, so no other scenario's server can take the port there.22+sed "s/^:8080 {/:$eport {/; s/127\.0\.0\.1:8081/127.0.0.1:$dead/" "$T_HOME/deploy/edge/Caddyfile" |23+ awk '{ print } $0 == "\tadmin off" { print "\tdefault_bind 127.0.0.1" }' >"$T/Caddyfile.edge"24+FORGE_DOMAIN=src.test FORGE_MIRROR_URL=http://127.0.0.1:$mport FORGE_MIRROR_TOKEN=s3cret-token XDG_DATA_HOME=$T/xdg XDG_CONFIG_HOME=$T/xdg \25+ caddy run --config "$T/Caddyfile.edge" --adapter caddyfile >"$T/caddy.edge.log" 2>&1 &26+T_PIDS="$T_PIDS $!"27+wait_http "http://127.0.0.1:$mport/"28+wait_http "http://127.0.0.1:$eport/"29+http -s -i -H 'Host: src.test' "http://127.0.0.1:$eport/dami/greet/" | tr -d '\r' >"$T/res"30+has "$T/res" '^X-Forge-Offline: mirror'31+has "$T/res" '<p>copy</p>'32+has "$T/seen" '^/dami/greet/index\.html s3cret-token$'33+lacks "$T/res" 's3cret-token'

9/10 A restored identity.db from litestream is the git user's alone

cat, , 1283f9e. CI ✓ passed in 115s, signed by cat

forge restore moved litestream's copy of identity.db into place with whatever mode litestream gave it; forge identity restore made it 600 first, and now identity_pick does too. Scenario: a stand-in litestream writes the file readable by anyone, and the restored one is 600.
2 files changed, +28 -0
Mbin/forge2++
Atest/scenarios/identity-restore-mode.sh26++++++++++++++++++++++++++
M bin/forge +2 -0
@@ -3738,6 +3738,8 @@ identity_pick() {
37383738 if [ -f "$db" ]; then bk=$(node "$FORGE_HOME/lib/identity.ts" changed "$db"); fi37393739 if [ "$lt" -ge "$bk" ]; then37403740 rm -f "$db-wal" "$db-shm"3741+ # Only git reads it, as forge identity restore leaves it.3742+ chmod 600 "$tmp"37413743 mv "$tmp" "$db"37423744 say "restore: identity.db is litestream's, which changed last ($(utc "$lt"))"37433745 else
A test/scenarios/identity-restore-mode.sh +26 -0
@@ -0,0 +1,26 @@
1+# When forge restore takes identity.db from litestream's replica (it2+# changed after the backup's copy), the file is the git user's alone (600),3+# however litestream wrote it. A stand-in litestream restores a copy of4+# identity.db that is readable by anyone.5+. "$(dirname "$0")/../lib.sh"6+user dami7+ok ssh_as dami token a8+"$FORGE" backup --dry-run --force >/dev/null 2>&1 || fail "backup --dry-run"9+mv "$FORGE_ROOT/.forge/backup/stage" "$T/stage"10+# The replica, a token later.11+export FORGE_NOW=$((T_NOW + 60))12+ok ssh_as dami token b13+FORGE_STATE=$FORGE_ROOT/.forge ok node "$T_HOME/lib/identity.ts" snapshot "$T/replica.db"14+mkdir -p "$T/bin"15+cat >"$T/bin/litestream" <<SH16+#!/bin/sh17+o=18+while [ \$# -gt 0 ]; do case \$1 in -o) o=\$2; shift ;; esac; shift; done19+cp "$T/replica.db" "\$o" && chmod 644 "\$o"20+SH21+chmod +x "$T/bin/litestream"22+export PATH=$T/bin:$PATH FORGE_BACKUP_REPOSITORY=$T/restic FORGE_BACKUP_PASSWORD=test23+r=$T/restored24+ok restore_into "$r" --from "$T/stage"25+has "$T_LOG" "identity.db is litestream's, which changed last"26+eq "$(mode "$r/.forge/identity.db")" 600

10/10 A scenario: a CI note's runner is never account@hostname

cat, , abf4f2a. CI ✓ passed in 115s, signed by cat

whoami_at already gives the forge user and a made-up machine name (the hostname only seeds its hash), and no other path writes the hostname. This holds it: with the forge user named as the local account, the runner isn't that account at this hostname, and the hostname is nowhere in the note.
1 file changed, +23 -0
Atest/scenarios/attest-runner-host.sh23+++++++++++++++++++++++
A test/scenarios/attest-runner-host.sh +23 -0
@@ -0,0 +1,23 @@
1+# A CI note's runner is never this machine's account@hostname, even when2+# the forge user has the same name as the local account: the machine part3+# is a made-up name, and the hostname appears nowhere in the note.4+. "$(dirname "$0")/../lib.sh"5+me=$(id -un)6+host=$(hostname -s 2>/dev/null || hostname)7+user "$me"8+repo "$me/greet"9+clone "$me" "$me/greet" "$T/w/d"10+mkdir -p "$T/w/d/.forge"11+printf '#!/bin/sh\ntrue\n' >"$T/w/d/.forge/ci"12+chmod +x "$T/w/d/.forge/ci"13+at "$T/w/d" commit "first"14+at "$T/w/d" git config forge.user "$me"15+export FORGE_USER=$me16+ok at "$T/w/d" "$FORGE" attest17+git -C "$T/w/d" notes --ref=ci show HEAD >"$T/note"18+r=$(sed -n 's/^runner //p' "$T/note" | tail -n 1)19+[ -n "$r" ] || fail "no runner in the note"20+[ "$r" != "$me@$host" ] || fail "the runner is $me@$host"21+case $r in "$me"@[a-z]*-[a-z]*-[0-9a-f][0-9a-f]) ;; *) fail "runner is '$r'" ;; esac22+grep -v '^|' "$T/note" | grep -qiwF "$host" && fail "the note names the host $host"23+true