The lake exports every repo, private ones included, and a private repo's build already put its insights page and exports into its pages in the private root. But forge lake published only public repos' exports into the live pages, so a private repo's numbers went stale until its next build. publishInsights now puts a private repo's exports into its page directory in the private root, as it does a public repo's in www, and takes them out again if the repo flipped meanwhile. Private builds now start the lake as public ones do, so a push to a private repo alone brings its insights up to date. The forge's exports still leave private repos out, and nothing of them goes to www or the mirror. insights-private-pages checks the page and exports in the private root, a refresh without a build, a reader seeing them with a token, the missing page's 404 for anyone else, and the mirror seeing none of it.
- commit
- 47539151070d5340c701a6460c1994a816295aeb
- parent
- c86d66765824bc8ea053e9460518308ff2769f5a
- author
- cat <admin@dev.site>
- kind
- feat risk score 1.6 of 3, Jev's estimate
| M | README.md | 14 | +++++---- |
| M | bin/ | 4 | ++ |
| M | lib/ | 27 | +++++++++------- |
| M | lib/ | 4 | ++- |
| A | test/ | 51 | ++++++++++++++++++++++++++++++ |
| M | test/ | 5 | ++- |
@@ -200,7 +200,7 @@ any left in the wrong place.
200200 201201 A private repo's pages are built like any repo's, into202202 `.forge/private/www` instead of `www`, with the same layout: pages, search203-index, `code.db`. They link the same hashed assets, which the build puts in203+index, `code.db`, release downloads, insights exports. They link the same hashed assets, which the build puts in204204 `www/assets` (they are the forge's scripts and styles, nothing of the205205 repo), and their header marks the repo private. Caddy serves a file of206206 `www` when there is one and hands every other path to@@ -679,11 +679,13 @@ reads the Parquet files over HTTP in byte ranges:
679679 duckdb -c "SELECT result, count(*) FROM 'https://src.rest/insights/ci.parquet' GROUP BY result"680680 ```681681 682-Private repos are in no public insights: they have no pages, and the683-forge's exports leave their facts out. Making a repo private removes the684-forge's exports from the pages at once; the next run writes them without685-it. Its own exports stay in `.forge/lake/export`, for pages behind a login686-later.682+Private repos are in no public insights: the forge's exports leave their683+facts out, and `www` has no page of theirs. A private repo's own insights684+page and exports are in its pages in the private root, which `forge lake`685+keeps fresh the same way, for its readers only (see [Private686+pages](#private-pages)); the mirror never sees them. Making a repo private687+removes the forge's exports from `www` at once; the next run writes them688+without it.687689 688690 The page has no query console. [DuckDB-Wasm](https://duckdb.org/docs/api/wasm/overview)689691 could run SQL over those files in the browser, but its build is 34.2 MiB@@ -2399,6 +2399,10 @@ build_private() {
23992399 # Made public while this was building: no private pages after all.24002400 if [ -d "$out/$only" ] && [ ! -f "$FORGE_ROOT/$only.git/git-daemon-export-ok" ]; then swap "$out/$only" "$FORGE_PRIVATE_WWW/$only"24012401 else rm -rf "${FORGE_PRIVATE_WWW:?}/$only"; fi2402+ # The lake takes in this build and the push before it, and puts the2403+ # repo's insights exports into these pages for its readers. Nothing2404+ # here is the mirror's.2405+ lake_kick24022406 say "built $only (private) in $(($(date +%s) - t0))s -> $FORGE_PRIVATE_WWW"24032407 }24042408 @@ -5,9 +5,11 @@
55 // Every repo's go to $FORGE_STATE/lake/export/<owner>/<repo>/insights/, the66 // forge's to .../export/insights/. The build copies a repo's into that repo's77 // pages (lib/forge.ts), so they follow its page directory wherever its pages8-// are served; publishInsights also puts them straight into the live pages9-// after each run, since the pages fetch them and need no rebuild. The forge's10-// cover public repos only: a private repo's facts are in no public export.8+// are served: www for a public repo, the private root for a private one,9+// whose readers alone see them. publishInsights also puts them straight into10+// the live pages after each run, since the pages fetch them and need no11+// rebuild. The forge's cover public repos only: a private repo's facts are12+// in no public export.1113 import fs from 'node:fs';1214 import path from 'node:path';1315 import { listRepos } from './lake-sources.ts';@@ -146,9 +148,10 @@ export async function exportInsights(lake: Lake, touched: Set<string>): Promise<
146148 }147149 148150 // Put each export into the live pages: the forge's into www/insights/, a149-// public repo's into its page directory, if it has one (pages of a repo150-// that just went private are gone, and stay gone). A file at a time, each151-// replaced by a rename, so a page never fetches half of one.151+// public repo's into its page directory in www and a private repo's into152+// its page directory in the private root, if it has one there (pages of a153+// repo that just flipped are gone, and stay gone until its build). A file154+// at a time, each replaced by a rename, so a page never fetches half of one.152155 export function publishInsights(cfg: LakeConfig): number {153156 let count = 0;154157 const put = (from: string, to: string): void => {@@ -167,16 +170,16 @@ export function publishInsights(cfg: LakeConfig): number {
167170 }168171 count++;169172 };170- if (!fs.existsSync(cfg.www)) return 0;171173 const forge = path.join(cfg.exportDir, 'insights');172- if (fs.existsSync(forge)) put(forge, path.join(cfg.www, 'insights'));174+ if (fs.existsSync(cfg.www) && fs.existsSync(forge)) put(forge, path.join(cfg.www, 'insights'));173175 for (const r of listRepos(cfg.root)) {174176 const from = path.join(cfg.exportDir, r.slug, 'insights');175- const pages = path.join(cfg.www, r.slug);176- if (!r.public || !fs.existsSync(from) || !fs.existsSync(pages)) continue;177+ const pages = path.join(r.public ? cfg.www : cfg.priv, r.slug);178+ if (!fs.existsSync(from) || !fs.existsSync(pages)) continue;177179 put(from, path.join(pages, 'insights'));178- // Made private meanwhile: take them out again.179- if (!fs.existsSync(path.join(r.dir, 'git-daemon-export-ok'))) {180+ // Flipped meanwhile: take them out again; its next build puts them181+ // where its pages are now.182+ if (fs.existsSync(path.join(r.dir, 'git-daemon-export-ok')) !== r.public) {180183 for (const f of FILES) fs.rmSync(path.join(pages, 'insights', f), { force: true });181184 }182185 }@@ -43,6 +43,8 @@ export interface LakeConfig {
4343 root: string;4444 state: string;4545 www: string;46+ // The private root, where private repos' pages are (bin/forge build_private).47+ priv: string;4648 dir: string;4749 catalog: string;4850 // A directory, or an s3:// URL (FORGE_LAKE_DATA).@@ -64,7 +66,7 @@ export function lakeConfig(env: NodeJS.ProcessEnv = process.env): LakeConfig {
6466 const extensions = env.FORGE_DUCKDB_EXTENSIONS ||6567 (fs.existsSync(bundled) ? bundled : path.join(env.XDG_CACHE_HOME || path.join(os.homedir(), '.cache'), 'duckdb', 'extensions'));6668 return {67- root, state, www: env.FORGE_WWW || path.join(state, 'www'), dir, catalog: path.join(dir, 'catalog.sqlite'),69+ root, state, www: env.FORGE_WWW || path.join(state, 'www'), priv: env.FORGE_PRIVATE_WWW || path.join(state, 'private', 'www'), dir, catalog: path.join(dir, 'catalog.sqlite'),6870 data: local ? path.resolve(data) + '/' : data.replace(/\/?$/, '/'), local, exportDir: path.join(dir, 'export'), extensions,6971 };7072 }@@ -0,0 +1,51 @@
1+# A private repo has an insights page in the private root, with its own2+# exports next to it, which forge lake keeps fresh there without a build;3+# the private server shows both to its readers only. Nothing of it is in4+# www, the forge's insights or the mirror.5+. "$(dirname "$0")/../lib.sh"6+user dami7+user cat8+repo --private dami/secret "hidden thing"9+repo dami/open10+for r in secret open; do11+ clone dami "dami/$r" "$T/w/$r"12+ at "$T/w/$r" commit "first"13+ ok push_as dami "$T/w/$r" origin main14+done15+ok forge lake16+build17+p=$FORGE_ROOT/.forge/private/www/dami/secret/insights18+exists "$p/index.html"19+has "$p/insights.json" '"scope":"dami/secret"'20+has "$p/insights.json" '"pushes":1,'21+for f in events builds reviews ci; do eq "$(head -c 4 "$p/$f.parquet")" PAR1; done22+23+# Another push: the next run puts the new numbers into the private pages.24+at "$T/w/secret" commit "second"25+ok push_as dami "$T/w/secret" origin main26+ok forge lake27+has "$p/insights.json" '"pushes":2,'28+missing "$FORGE_WWW/dami/secret"29+lacks "$FORGE_WWW/insights/insights.json" 'dami/secret'30+if grep -rlq 'dami/secret' "$FORGE_WWW"; then fail "www names the private repo"; fi31+32+private_server33+tok=$(ssh_as dami token --repo dami/secret 2>/dev/null)34+cat_tok=$(ssh_as cat token 2>/dev/null)35+eq "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $tok" "$PRIV/dami/secret/insights/")" 20036+curl -s -H "Authorization: Bearer $tok" "$PRIV/dami/secret/insights/insights.json" >"$T/got"37+ok cmp "$p/insights.json" "$T/got"38+eq "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $tok" "$PRIV/dami/secret/insights/ci.parquet")" 20039+response "$PRIV/dami/nope/insights/insights.json" >"$T/missing"40+for a in "" "$cat_tok"; do41+ response "$PRIV/dami/secret/insights/insights.json" ${a:+-H "Authorization: Bearer $a"} >"$T/got"42+ cmp -s "$T/missing" "$T/got" || fail "not the missing page's 404: ${a:-anonymous}"43+done44+45+# The mirror (a stand-in rclone that lists what it was given) sees none of it.46+mkdir "$T/bin"47+printf '#!/bin/sh\nfind "$2" >>"%s/rclone.saw"\n' "$T" >"$T/bin/rclone"48+chmod +x "$T/bin/rclone"49+PATH="$T/bin:$PATH" FORGE_MIRROR_S3=https://r2.test/pages ok forge mirror50+has "$T/rclone.saw" '/dami/open/insights/insights\.json$'51+lacks "$T/rclone.saw" 'secret'@@ -1,6 +1,7 @@
11 # Private repos are in no public insights: no insights page or export in2-# the pages, and nothing of theirs in the forge's (JSON or Parquet). Their3-# exports stay in the lake's state, beside where their pages would be. A2+# www, and nothing of theirs in the forge's (JSON or Parquet). Their3+# exports are in the lake's state, and in their pages in the private root4+# for their readers (insights-private-pages). A45 # repo that goes private leaves the forge's insights at once; one that goes56 # public joins them at the next run.67 . "$(dirname "$0")/../lib.sh"