A section on forge lake and the insights pages: what it gathers and why, how runs stay incremental across events.jsonl being cut, the one writer, read-only queries, data on R2, how private repos stay out, why there is no DuckDB-Wasm console, and the measured costs in dependencies, image size, disk, time and export sizes. What runs, the backup contents, the URL table, the keys and the template blocks mention it too.
- commit
- d85609a831f1c1778750c398d2ccac2e76a3f2f7
- parent
- 16632cfd6b4eb48a4aa5b1fa088b0188c378bd67
- author
- cat <admin@dev.site>
- kind
- docs risk score 0.1 of 3, Jev's estimate
| M | README.md | 116 | +++++++++++++++++++++++++++++- |
@@ -39,6 +39,9 @@ It adds what stagit doesn't have, borrowing from [args.io](https://args.io):
3939 - **Search and syntax highlighting.** `/` searches a repo's commits, reviews,4040 wiki and file paths from a static index, and file contents from a SQLite4141 index the browser reads in byte ranges; code is highlighted at build time.42+- **Insights.** The forge's facts (pushes, builds, reviews, CI results) are43+ kept in a [DuckLake](https://ducklake.select/), and static pages chart44+ them for the forge and for each repo.4245 4346 The [guide](guide/index.html) explains all of this with diagrams. It is a4447 static site in `guide/`, declared in `.forge/sites`, so the forge serves it@@ -96,6 +99,7 @@ forge comment, approve ─── ssh ──────────────>
9699 | `lib/pages-server.ts` | Only with a pages domain. Serves each site on its own hostname the way Cloudflare Pages does (`_headers`, `_redirects`, pretty URLs, the nearest `404.html` or a single-page-app fallback) and redirects old `/sites/...` paths there. Runs as the caddy user, since deploys are untrusted content. |97100 | `forge backup` | Daily, from a loop in `start.sh`. Snapshots the forge with restic into a bucket, then checks the snapshots and test-restores the new one (see [Backups and moving hosts](#backups-and-moving-hosts)). |98101 | `forge mirror` | After builds, in the background: syncs the forge's own pages to a bucket with rclone, for the edge to serve while the forge is down. |102+| `forge lake` | After builds, in the background: adds what is new in the event log, build metrics, review logs and CI notes to the lake, and writes the insights exports (see [The lake and insights](#the-lake-and-insights)). |99103 100104 Without a pages domain, Caddy serves sites straight from `/sites/...` and101105 `/previews/...` on the forge's own origin, and `_headers` and `_redirects` don't@@ -513,6 +517,106 @@ files are cached in `.forge/cache/highlight/` by blob id, so pushing a new
513517 branch doesn't redo files that other branches already have. Token colors are514518 CSS variables (`--tok-*`) in `style.css`.515519 520+## The lake and insights521+522+The forge's facts are append-only, but they don't last: `events.jsonl` is523+cut to its last 1000 lines once it passes 1 MB, `metrics.jsonl` isn't backed524+up, and review logs and CI notes are spread over the repos. `forge lake`525+gathers them into a [DuckLake](https://ducklake.select/) (DuckDB 1.5.6526+through `@duckdb/node-api`): its catalog is a SQLite file,527+`.forge/lake/catalog.sqlite`, and its data Parquet files in528+`.forge/lake/data`, or in a bucket (below).529+530+```sh531+forge lake # add what is new; builds run it by themselves532+forge lake --dry-run # say what it would add533+forge lake query "SELECT repo, count(*) FROM events WHERE type = 'push' GROUP BY repo"534+forge metrics # build history, from the lake as well as metrics.jsonl535+```536+537+Tables: `events` (pushes, comments, approvals, builds), `builds` and538+`build_repos` (each line of `metrics.jsonl`), `reviews` (each line of a539+review log: created, revisions, approvals, deleted), `review_ends` (landed540+or closed, and when) and `ci` (each attestation: commit, tree, result,541+duration, runner, signer). A review counts as landed when its branch is542+deleted with its tip in the default branch, as `forge land` leaves it; one543+whose tip reaches the default branch while the branch stays counts from544+when the lake saw it.545+546+- **Incremental.** Each source remembers how far it was read, in the same547+ transaction as its rows, so a run that dies adds nothing and a second run548+ adds nothing new. For `events.jsonl` that is a byte offset plus the549+ file's inode, a hash of its first line and hashes of the last lines read:550+ when the build cuts the file into a new one, the next run finds the last551+ line it read and goes on from there. If the cut went past it (more than552+ 1000 lines came in between runs), it says so, and only those lines are553+ missing. Review logs are read again whole when they change, CI notes when554+ their ref moves.555+- **One writer.** A DuckDB file can't be written by two processes, and556+ DuckLake over SQLite turned away half of 120 concurrent inserts with557+ "database is locked". So `forge lake` holds `.forge/lake.lock`; a second558+ one waits for it (`FORGE_LAKE_WAIT` seconds, 300) or gives up and says559+ so. Builds start it the way they start the mirror, `FORGE_LAKE_DELAY`560+ seconds (10) after the last one; `FORGE_LAKE=0` stops that.561+- **Small writes.** DuckLake keeps inserts under 10 rows in the catalog562+ rather than writing a Parquet file for each; about once a day563+ (`FORGE_LAKE_CHECKPOINT` hours) a run checkpoints, which writes them out564+ and merges small files.565+- **Read-only queries.** `forge lake query` attaches the lake read-only,566+ runs only `SELECT`, `EXPLAIN` and `PRAGMA` statements, and can't touch567+ files other than the lake's data. `--json` and `--csv` change the output.568+- **Offline.** DuckDB downloads extensions at first use. The image installs569+ them at build time (`node lib/lake.ts install`, which checks they are the570+ versions this was tried with), so the container never does. Elsewhere571+ they go to `~/.cache/duckdb/extensions` the first time.572+573+Data on R2 instead of the volume: put `FORGE_LAKE_DATA=s3://<bucket>/lake`574+and `FORGE_LAKE_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com` in575+`forge.env` (the `FORGE_BACKUP_*` keys are used unless `FORGE_LAKE_KEY_ID`576+and `FORGE_LAKE_SECRET` are set). Set it before the first run; the catalog577+stays on the volume either way. This was tried against `rclone serve s3`,578+not R2 itself.579+580+### Insights pages581+582+`/insights/` covers the forge, and each repo has `/<owner>/<repo>/insights/`583+(the Insights tab, `i`): stat tiles, then build time, pushes, time to land,584+revisions per review, CI results and CI duration. The pages are static and585+built once; their script (`insights.js`, on these pages only) fetches586+`insights.json` and draws the charts as SVG, light and dark, with a tooltip587+on hover and keyboard focus and a table of each chart's numbers. After each588+run, `forge lake` writes that JSON and the rows behind it as Parquet589+(`events`, `builds`, `reviews`, `ci`) into `.forge/lake/export` and straight590+into the pages, so they update without a build, and the mirror picks them591+up. A repo's exports sit in its page directory, which builds keep. DuckDB592+reads the Parquet files over HTTP in byte ranges:593+594+```sh595+duckdb -c "SELECT result, count(*) FROM 'https://src.rest/insights/ci.parquet' GROUP BY result"596+```597+598+Private repos are in no public insights: they have no pages, and the599+forge's exports leave their facts out. Making a repo private removes the600+forge's exports from the pages at once; the next run writes them without601+it. Its own exports stay in `.forge/lake/export`, for pages behind a login602+later.603+604+The page has no query console. [DuckDB-Wasm](https://duckdb.org/docs/api/wasm/overview)605+could run SQL over those files in the browser, but its build is 34.2 MiB606+(7.7 MiB gzipped, 5.1 MiB with brotli) to query exports of a few kilobytes,607+and its npm package is 149 MB.608+609+What it costs, measured on an M-series Mac and in the image (linux/arm64):610+611+| | |612+| --- | --- |613+| `@duckdb/node-api` | the native DuckDB library: 61M in the image (71M for x64; npm also fetches the musl build, which the image drops) |614+| Extensions | 80M in the image: ducklake 30.7M, sqlite 29.6M, httpfs 18.9M |615+| Image | about 143M more, all told |616+| Disk | the demo's lake: 225K. 20,000 events and 5,000 builds (3.3M of JSON lines): 406K, a 180K catalog and Parquet |617+| Time | 20,000 events and 5,000 builds in 0.18s; a run after a push 0.1 to 0.5s, plus about half a second to start Node and load DuckDB; the demo's first run 0.7s |618+| Exports | the demo's: insights.json 1.5K, each Parquet file 2 to 4K; with 300 builds, insights.json 8.5K and builds.parquet 9K |619+516620 ## Hosting517621 518622 The forge is one container (`Dockerfile`): sshd for git, Caddy for pages,@@ -688,6 +792,12 @@ Everything in `/srv/git` that can't be rebuilt:
688792 site has;689793 - `.hostkeys/`, so clients see the same ssh host key, and `.tailcat/`, so a690794 restored box keeps its tunnel address and the edge finds it;795+- the lake (`.forge/lake`): the catalog, copied with SQLite's `VACUUM INTO`796+ while no `forge lake` runs, never the live file, and checked; and its data797+ files, as hard links. It can't be rebuilt, since the files it read are cut798+ or left out. Data on R2 stays in its bucket: the lake keeps snapshots for799+ 40 days and deletes a file a week after the last one that used it, so a800+ catalog from any kept backup still finds its files;691801 - `.ssh/`, and anything else under `/srv/git` or `.forge/` that isn't left out692802 below, so state added later is backed up by default.693803 @@ -911,7 +1021,7 @@ Keyboard shortcuts (`?` lists them):
9111021 | `x` | fold or unfold the file in view |9121022 | `v` | mark the file in view as viewed |9131023 | `h` `g` | repo home, all repositories |914-| `f` `l` `b` `r` `w` `s` | code, commits, branches, reviews, wiki, sites |1024+| `f` `l` `b` `r` `w` `s` `i` | code, commits, branches, reviews, wiki, sites, insights |9151025 | `u` | up a level |9161026 9171027 ## Templates@@ -922,7 +1032,7 @@ with plain `git` commands and hands 11ty flat page lists (`forge.pages.logs`,
9221032 rendered by filters when a page is written, so the build never holds all the9231033 markup at once.9241034 925-Every page extends `site/_includes/base.njk` and fills its blocks: `content`,1035+Every page extends `site/_includes/base.njk` and fills its blocks: `title`, `content`,9261036 `head` (the page's description and its OpenGraph tags, so links pasted into9271037 chat apps preview well; `og:url` needs `FORGE_BASE_URL`), `scripts` (the diff9281038 pages add `diffs.js`) and `heading`. Front matter shared by many templates,@@ -986,6 +1096,8 @@ may be missing needs an explicit check or `or ''`.
9861096 | `/<owner>/<repo>/sites/` | deploys and previews |9871097 | `/<owner>/<repo>/releases/` | release notes per tag, and each release's files |9881098 | `/<owner>/<repo>/releases/download/<tag>/<file>` | a release file; `latest/` for the newest release's |1099+| `/<owner>/<repo>/insights/` | charts from the lake; `insights.json` and Parquet files next to it |1100+| `/insights/` | the same for the whole forge, and a table by repo |9891101 | `/sites/<owner>/<repo>/<name>/` | a deployed site |9901102 9911103 ## Not yet