The search section says how code search works and what it costs, with the measured sizes and requests, and drops "file contents are not searched"; the intro, the URL table and the assets paragraph mention code.db and the sqlite-wasm assets.
- commit
- 6dbc768e86675d93ba00550d524f685fa6b1bea0
- parent
- eaeac0c3884f4775fea37212f5e05feaf5608bd1
- author
- cat <admin@dev.site>
- kind
- docs risk score 0.2 of 3, Jev's estimate
| M | README.md | 84 | ++++++++++++++++++++++++++---- |
@@ -34,7 +34,8 @@ It adds what stagit doesn't have, borrowing from [args.io](https://args.io):
3434 branch gets its own preview. The wiki is markdown in `wiki/`.3535 - **Keyboard navigation and clean URLs**, e.g. `/dami/greet/tree/main/greet.sh/`.3636 - **Search and syntax highlighting.** `/` searches a repo's commits, reviews,37- wiki and file paths from a static index; code is highlighted at build time.37+ wiki and file paths from a static index, and file contents from a SQLite38+ index the browser reads in byte ranges; code is highlighted at build time.3839 3940 The [guide](guide/index.html) explains all of this with diagrams. It is a4041 static site in `guide/`, declared in `.forge/sites`, so the forge serves it@@ -325,14 +326,76 @@ until `forge gc` (below).
325326 326327 `/`, or the search button in the header, opens a search over the repo's327328 commits (subject, body, sha, author), reviews (number, title, branch,328-author), wiki pages, and file paths on the default branch. On the index and329-owner pages it searches repo names and descriptions. There is no server: each330-build writes `/<owner>/<repo>/search.json` (and `/repos.json`), which the page331-fetches the first time the search opens, and matching runs in the browser.332-Commit bodies are cut to 240 characters and wiki pages to 2000, to keep the333-index small. File contents are not searched; a full-text code index334-is too big to ship as a static file. `t` still finds files by name on the335-current branch.329+author), wiki pages, file paths, and the contents of the files on the330+default branch. On the index and owner pages it searches repo names and331+descriptions. There is no server: each build writes332+`/<owner>/<repo>/search.json` (and `/repos.json`), which the page fetches the333+first time the search opens, and matching runs in the browser. Commit bodies334+are cut to 240 characters and wiki pages to 2000, to keep the index small.335+`t` still finds files by name on the current branch.336+337+#### Code search338+339+File contents are in a second index per repo, `/<owner>/<repo>/code.db`,340+which `search.json` points at (`"code"`). It is a SQLite database with an341+[FTS5](https://sqlite.org/fts5.html) trigram index, and the browser never342+downloads it whole: the official [sqlite-wasm](https://sqlite.org/wasm) build343+opens it through a small read-only VFS (`site/assets/codesearch.ts`) whose344+reads are HTTP range requests, so a query fetches only the pages it touches.345+Any static server that honors `Range` works: Caddy, `demo/serve.ts`, and the346+pages mirror on R2 through the edge, whose fallback passes the header on.347+348+From 3 characters (a trigram can't match less) the query, as one literal349+string ignoring case, goes to a worker, and a Code section lists each350+matching line as `path:line`, linked to that line of the file's page (the351+source page for markdown). It loads nothing until then: the first code query352+starts the worker, which loads `sqlite3.wasm` and sqlite-wasm's script, both353+content-hashed assets. Results come in path order; the first look reads 12354+chunks, and "more" reads up to 100.355+356+What goes in: every file on the default branch except binary files (a NUL in357+the first 8000 bytes, as git decides), files over the 256K highlight cap,358+lockfiles (`package-lock.json`, `yarn.lock`, `go.sum`, `Cargo.lock`, ...),359+minified files and source maps, images, symlinks and submodules. Files are360+stored as chunks of 8 lines (`file`, `chunk`, and the FTS5 table `code` over361+the chunks' text, with `detail=full`, which substring queries need); the362+browser finds the line inside a chunk itself. The build writes 16K pages,363+merges the index into one segment (`optimize`), and `VACUUM`s, so every table364+is its root page over its leaves, laid out in order.365+366+Each round trip counts, since SQLite reads one page at a time and waits for367+it. So the worker names most of a query's pages before SQLite asks: it opens368+the file by fetching both ends at once (the schema, every table's root and369+the file list are at the front; FTS5's term index is at the back), looks up370+where each term's doclist lives in that term index and fetches those pages371+together, then fetches the pages of the matching chunks together. A query372+looks up every third trigram of the string (and the last), within `NEAR`373+distance, rather than all of them, and checks the lines for the exact string.374+If the index is rebuilt while a page is open, the next query notices the new375+ETag and reopens it.376+377+Measured with `node test/code-search-bench.ts bench <state>`, which serves a378+built forge with `demo/serve.ts` and runs the same worker code in Node:379+380+| Repo | Indexed text | `code.db` | Cold query | Next queries |381+| --- | --- | --- | --- | --- |382+| this repo, 103 files | 640K | 2.7M | 3 round trips, 4 to 11 requests, 240K to 420K | 0 to 2 round trips, up to 96K |383+| an 11M tree (eleventy, markdown-it, highlight.js, typescript's `tsc.js` and more), 662 files | 4.4M | 16.1M | 3 round trips, 6 to 12 requests, 290K to 400K | 1 or 2 round trips, 80K to 160K |384+385+Of a cold query, 128K is the file's head and 64K its tail, read once per386+page. An index under 256K is read whole on the first query. The first code387+search also costs the assets, once and then cached for good: `sqlite3.wasm`388+849K (394K gzipped) and sqlite-wasm's script 628K (171K gzipped). Building the389+index took about 0.3s for this repo and 1.5 to 2s for the 11M tree.390+391+The index is cached in `.forge/cache/code/<owner>/<repo>/` by the default392+branch's tree id and hard-linked into the pages, so a push that leaves that393+tree alone (a review branch, an empty commit) keeps the very same file. A394+new tree starts from the previous index and only deletes and inserts the395+files whose blob changed: about 0.35s for one changed file in the 11M tree.396+Don't compress `code.db` in transit: a range of a compressed response is not397+a range of the file. `forge server caddyfile` compresses `.wasm` and scripts398+but not the index.336399 337400 Files, diffs and fenced code in markdown are highlighted at build time with338401 [highlight.js](https://highlightjs.org/), so pages need no script for it. The@@ -760,7 +823,7 @@ rendered, `/assets/<hash>.css`, shared by pages of a kind. Import
760823 `macros.njk` with context, or blocks in macros don't reach the bundle.761824 762825 Assets ship under content-hashed names (`/assets/style.<hash>.css`,763-`ui.<hash>.js`, mermaid), so browsers can cache them for good. The same826+`ui.<hash>.js`, mermaid, sqlite-wasm), so browsers can cache them for good. The same764827 sources give the same names, and pages' fingerprints include them, so765828 unchanged pages are still hard-linked. A build keeps the files of the last766829 five code versions (listed in `assets/versions.json`), because pages built by@@ -791,6 +854,7 @@ may be missing needs an explicit check or `or ''`.
791854 | `/<owner>/<repo>/reviews/<id>/` | stacked diff view; `diff/` combined, `r<n>/` what changed since the revision before |792855 | `/<owner>/<repo>/wiki/<page>/` | wiki |793856 | `/<owner>/<repo>/reviews.json` | the review list as data, for `forge reviews` |857+| `/<owner>/<repo>/code.db` | the default branch's code index, read by code search in byte ranges |794858 | `/<owner>/<repo>/sites/` | deploys and previews |795859 | `/sites/<owner>/<repo>/<name>/` | a deployed site |796860