architecture.md shows the new layout, the one-way dependencies and what sheet re-exports, the cell store, the lookup seam and the workbook's JEV source, and where the boundaries leave room for the storage, range and file-format work in limits.md, which now points at those seams.
- commit
- a064c75db0a2ff35ccdba178f47608c8d21aca3d
- parent
- 5869f197c1119b93da28dfa5d59e0af8d51954d0
- author
- Dami <dami.etoile@gmail.com>
| M | docs/ | 88 | +++++++++++++++++++++++++----- |
| M | docs/ | 42 | ++++++++++---- |
@@ -5,7 +5,9 @@ Lip Gloss v2.
55 66 ```77 cmd/012 entry point: flags, JEV setup, opening or importing a file8-internal/sheet the engine: cells, parsing, evaluation, recalculation, undo, files8+internal/sheet the engine: cells, evaluation, functions, recalculation, undo, files9+internal/formula the formula language: references, lexer, parser, printer, rewriting10+internal/numfmt number formats, rounding, General, date serials911 internal/fileio import and export: CSV, TSV, XLSX, SQLite, Parquet, Lotus .wk11012 internal/chart chart layout, text rendering and kitty image encoding1113 internal/jev JEV configuration, answer cache and TypeSafe client@@ -16,16 +18,50 @@ oracle/ differential tests against excelize (separate module)
1618 1719 ## The engine1820 19-`internal/sheet` knows nothing about terminals. A `Workbook` holds ordered20-`Sheet`s, the named ranges, the undo history, the arithmetic setting and21-recalculation, as a Sheets spreadsheet does. Each sheet keeps its cells in22-a sparse map from address to cell, with its column widths, frozen panes,23-filter and charts; each cell keeps what was typed, its parsed expression,24-its computed value, and its format and style.21+`internal/sheet` knows nothing about terminals. It builds on two packages22+that know nothing about cells, and dependencies point one way:2523 26-- **Parsing.** A hand-written Pratt parser turns formulas into an AST,27- keeping absolute markers so references can be rewritten when cells move.28- A printer turns ASTs back into text in Sheets' spelling.24+```25+internal/ui, internal/fileio, internal/chart, oracle26+ |27+internal/sheet ---> internal/formula28+ |29+ +--------> internal/numfmt30+```31+32+- `internal/formula` is the language: `Addr` and `Rect`, A1 references33+ with their `$` markers, sheet names in references and their quoting,34+ the lexer and Pratt parser, the syntax tree, the printer and the35+ reference rewriting for copies, moves and inserted or deleted rows and36+ columns. The parser learns which functions exist through a small37+ `Func` interface that the engine's `FuncDef` implements.38+- `internal/numfmt` renders numbers: number format patterns (as in TEXT39+ and custom formats), Sheets' General form, rounding on 15 significant40+ digits, and the calendar of serial day numbers.41+42+`sheet` re-exports what its callers used before (`sheet.Addr`,43+`sheet.Rect`, `sheet.Parse`, `sheet.FormatPattern`, the sheet-name44+helpers) as aliases and thin wrappers, so the UI, importers and charts see45+one engine package. Typed-entry recognition (`ParseValue`, which turns46+"$1,200" or "9/26" into a number and a format) stays in `sheet`: it yields47+the engine's `Format` and reads its clock.48+49+A `Workbook` holds ordered `Sheet`s, the named ranges, the undo history,50+the arithmetic setting, the JEV source and recalculation, as a Sheets51+spreadsheet does. Each sheet keeps its cells, sparse, by address, with52+its column widths, frozen panes, filter and charts; each cell keeps what53+was typed, its parsed expression, its computed value, and its format and54+style.55+56+- **Cells.** Each sheet keeps its cells in a `cellStore` (`store.go`):57+ get, set, delete, count, iterate all or a range. Nothing else touches58+ the map beneath, so the representation can change without the rest of59+ the engine noticing.60+- **Parsing.** `internal/formula`'s hand-written Pratt parser turns61+ formulas into an AST, keeping absolute markers so references can be62+ rewritten when cells move. Its printer turns ASTs back into text in63+ Sheets' spelling. Decimal arithmetic marks the operators it computes64+ with node types of the engine's own, so trees stay as parsed.2965 - **Recalculation.** Formulas record the cells and ranges they read on3066 their own sheet; references that name a sheet (`Sheet2!A1`) are kept by3167 name and resolved when evaluated, so renaming rewrites them and a deleted@@ -34,7 +70,9 @@ its computed value, and its format and style.
3470 sheet, then evaluates the marked cells lazily in dependency order; a cell3571 reached again while it is being evaluated is part of a cycle. Volatile3672 functions (TODAY, RAND, the JEV functions) are recomputed on every37- recalculation.73+ recalculation (`recalc.go`). Formulas read other cells through a74+ `lookup`: `cell` for one, `cells` for a whole range, which the75+ aggregates (SUM, AVERAGE, COUNT and the rest) use.3876 - **Functions.** One table (`FuncDef`) holds every function's name,3977 signature, description and arity; it drives parsing, evaluation,4078 autocomplete, in-app help and [functions.md](functions.md).@@ -43,9 +81,31 @@ its computed value, and its format and style.
4381 on any sheet, so a step can be reversed exactly; multi-cell operations4482 are one step. A deleted sheet keeps its cells, so undo brings it back.4583 - **JEV.** The engine never touches the network. JEV functions describe a46- question and look up the answer in a pluggable source; `internal/jev`47- answers from a cache and queues new questions, and the UI sends them as48- background commands.84+ question and look up the answer in the workbook's `RemoteSource`, set85+ with `SetRemote`; `internal/jev` answers from a cache and queues new86+ questions, and the UI sends them as background commands.87+88+### Where the engine can grow89+90+The package boundaries leave the bounds in [limits.md](limits.md) room to91+move without touching callers:92+93+- **Storage.** Compact cell storage (column blocks of values, formulas and94+ formats in side tables) replaces `cellStore`'s map; its methods are the95+ whole contract.96+- **Range reads.** Shared range results, prefix sums for running totals97+ and column-block scans go behind `lookup.cells`, the one place98+ aggregates read ranges.99+- **Files.** A streaming or columnar `.012` format replaces `file.go`'s100+ whole-document JSON; the format is already separate from the store.101+- **Depth limits.** A cap on nesting goes in `internal/formula`'s parser,102+ so a pathological formula fails to parse before anything evaluates it.103+- **Functions.** The function library (values, evaluation, `FuncDef` and104+ the `functions_*.go` tables, about a third of `sheet`) is the next105+ boundary: it needs cells only through `lookup`. It stays in `sheet`106+ for now because `lookup` is a concrete type (an interface across a107+ package boundary would put every range callback on the heap) and108+ `Value` and `Format` would have to move below the engine with it.49109 50110 ## The UI51111 @@ -89,7 +89,14 @@ terminal (`BenchmarkFrame`).
8989 | 20 charts (text) | 0.82 ms | 2.9 ms | 7.3 ms |9090 9191 About half of a frame at 400 x 120 is Bubble Tea's parse and diff of the92-view string. Key presses add little: an arrow key through to its frame92+view string.93+94+A chart drawn as text costs 4 to 50 us at 24 x 10 to 120 x 40 cells,95+whatever its data: only the categories that fit are drawn (a pie of 819296+slices, 0.44 ms, is the worst). An image for kitty graphics is redrawn97+when its data, size or theme changes: 25 to 260 us at 24 x 10 cells and98+1 to 1.5 ms at 120 x 40, a pie 8.6 ms (`internal/chart`,99+`BenchmarkDraw`, `BenchmarkImage`). Key presses add little: an arrow key through to its frame93100 is 0.21 ms at 80 x 24 and 1.06 ms at 200 x 60; Page Down 1.26 ms.94101 95102 With the whole of an 8192 x 256 sheet selected, the status line's Sum,@@ -113,8 +120,12 @@ Real, openly licensed datasets fetched by `scripts/stress-data.sh`
113120 | Parquet alltypes_tiny_pages | 7300 x 13 | 40 ms | 2.4 M cells/s | 276 B |114121 115122 Past the limits, imports keep the first 8192 rows and 256 columns and116-say how much they left out; reading the rest of the file still costs117-(the airport codes file is read to its end).123+say how much they left out. Parquet files and SQLite tables stop reading124+at the last row and take the count of the rest from the file; CSV, TSV125+and SQLite queries are read to their end to count it, without keeping126+it (the airport codes file is read to its end). Every importer but XLSX127+streams, so memory follows the sheet, not the file; excelize holds an128+XLSX worksheet in memory while its rows are read.118129 119130 ## Undo120131 @@ -196,8 +207,9 @@ would take; without them a 10 M-cell sheet would need about 3 GB of
196207 heap, take 10 s to open and 140 ms per key to extend a selection over197208 it.198209 199-1. **Compact cell storage** (L, 2 to 3 weeks). Cells live in200- `map[Addr]*Cell`, 300 B each: a 216 B struct carrying formula-only210+1. **Compact cell storage** (L, 2 to 3 weeks). Cells live in a211+ `map[Addr]*Cell` behind `cellStore` (`internal/sheet/store.go`), 300 B212+ each: a 216 B struct carrying formula-only201213 fields (expression, reference lists, inferred format) on every cell,202214 plus the map entry, the input string and a boxed literal. Storing203215 columns in row blocks (say 1024 rows) of compact values, with@@ -206,7 +218,8 @@ it.
206218 of the value, would bring numbers to about 20 to 40 B per cell and207219 turn every cell read in recalc from two hash lookups (20 ns) into an208220 index (2 to 3 ns). Cells are already per sheet in a workbook, so the209- storage can change sheet by sheet.221+ storage can change sheet by sheet, and nothing outside `cellStore`222+ touches the map, so the change stays inside it.210223 2. **Range dependency index and shared range results** (M, 1 week).211224 Range users are now indexed by column; an interval index per column212225 (or per row block) would make finding them independent of how many@@ -214,12 +227,14 @@ it.
214227 recalc would make 1000 identical `SUM(A1:A8192)` cost one, and215228 running totals could be computed from a prefix sum per column block:216229 the fan-in and running-total rows of the table above would drop to217- milliseconds.230+ milliseconds. Aggregates already read ranges through `lookup.cells`231+ (`internal/sheet/recalc.go`), where such results would be served.218232 3. **Streaming, compact file format** (M, 1 week). The `.012` file is219233 JSON decoded whole (1.3 GB allocated to open 47 MB). A streaming220234 decoder over the same format would roughly halve open time and cut221235 allocation tenfold; a columnar or gzip-compressed variant would cut222- the size about fourfold.236+ the size about fourfold. The format is `internal/sheet/file.go`, apart237+ from the cell store.223238 4. **Incremental selection statistics** (S, 2 days). Keep per-column224239 sums and counts, updated in `place` and after recalc, and compute a225240 selection's Sum and Count from column totals minus the rows outside@@ -235,9 +250,10 @@ it.
235250 7. **Frame rate** (S, hours). Bubble Tea draws at most 60 frames a236251 second; asking for 120 would halve the key-to-screen floor from about237252 16 ms to 8 ms at the cost of more redraws.238-8. **Recursion limits** (S, hours). The parser and evaluator recurse239- without a limit; a depth cap (Excel allows 64 nested functions) would240- turn a pathological file into an error rather than a deep stack.253+8. **Recursion limits** (S, hours). The parser (`internal/formula`) and254+ the evaluator recurse without a limit; a depth cap (Excel allows 64255+ nested functions) would turn a pathological file into an error rather256+ than a deep stack.241257 242258 ## Measuring243259 @@ -250,8 +266,10 @@ make stress-e2e # key press to screen through libghostty
250266 251267 Benchmarks are built only with `-tags stress`, so `go test ./...` stays252268 fast. They live in `internal/sheet/stress_test.go` (engine),269+`internal/sheet/hotpath_stress_test.go` (parsing and number formats),253270 `internal/ui/stress_test.go` (View, frames, keystrokes, JEV, telemetry254-overhead), `internal/fileio/stress_test.go` (imports and exports) and271+overhead), `internal/fileio/stress_test.go` (imports and exports),272+`internal/chart/stress_test.go` (charts as text and images) and255273 `e2e/stress_test.go`; the synthetic sheets are built by256274 `internal/stress`. Results accumulate in `.deps/stress/results` and, with257275 the observability stack up, in ClickHouse.