architecture.md describes internal/functions and internal/value, the Book the engine implements and why it passes only values; extending.md updates the registry, range-read and side-effect patterns; limits.md records the before and after numbers.
- commit
- c247b57c1e3eaddceb7b70305a53222155be30f9
- parent
- 719e2beff43d02a809c666898a3723cfcc3502b1
- author
- Dami <dami.etoile@gmail.com>
| M | docs/ | 145 | +++++++++++++++++++++++------- |
| M | docs/ | 44 | +++++++-- |
| M | docs/ | 31 | ++++++- |
@@ -6,7 +6,9 @@ Lip Gloss v2.
66 ```77 cmd/012 entry point: flags and config, 012 config, 012 serve, JEV setup, opening or importing a file88 internal/config the config file and the registry of options9-internal/sheet the engine: cells, evaluation, functions, recalculation, undo, files9+internal/sheet the engine: cells, recalculation, undo, files, names, pivots10+internal/functions the function library: the FuncDef table, evaluation, decimal arithmetic, JEV questions11+internal/value cell values, number formats, typed-entry parsing, the clock1012 internal/formula the formula language: references, lexer, parser, printer, rewriting1113 internal/numfmt number formats, rounding, General, date serials1214 internal/fileio import and export: CSV, TSV, XLSX, SQLite, Parquet, Lotus .wk1@@ -28,33 +30,45 @@ oracle/ differential tests against excelize's calculation (separate mod
2830 2931 ## The engine3032 31-`internal/sheet` knows nothing about terminals. It builds on two packages32-that know nothing about cells, and dependencies point one way:33+`internal/sheet` knows nothing about terminals. It builds on the function34+library, which knows nothing about sheets or storage, and on packages35+that know nothing about cells; dependencies point one way:3336 3437 ```3538 internal/ui, internal/fileio, internal/chart, oracle3639 |37-internal/sheet ---> internal/formula38- |39- +--------> internal/numfmt40+internal/sheet ------------------------------+41+ | |42+internal/functions ---> internal/formula |43+ | |44+internal/value ---> internal/numfmt <------+4045 ```4146 47+(`sheet` also uses `formula`, `value` and `numfmt` directly.)48+4249 - `internal/formula` is the language: `Addr` and `Rect`, A1 references4350 with their `$` markers, sheet names in references and their quoting,4451 the lexer and Pratt parser, the syntax tree, the printer and the4552 reference rewriting for copies, moves and inserted or deleted rows and4653 columns. The parser learns which functions exist through a small47- `Func` interface that the engine's `FuncDef` implements.54+ `Func` interface that the library's `FuncDef` implements.55+- `internal/value` is what a cell computes to and how it is shown:56+ `Value` with its kinds and error codes, the coercions and ordering57+ every function shares (`ToNum`, `Compare`), `Format` and its kinds, and58+ typed-entry recognition (`ParseValue`, which turns "$1,200" or "9/26"59+ into a number and a format), with the clock it reads (`value.Now`,60+ which TODAY and NOW read too; tests replace it).61+- `internal/functions` is the function library; see62+ [Functions](#functions) below.4863 - `internal/numfmt` renders numbers: number format patterns (as in TEXT4964 and custom formats), Sheets' General form, rounding on 15 significant5065 digits, and the calendar of serial day numbers.5166 5267 `sheet` re-exports what its callers used before (`sheet.Addr`,53-`sheet.Rect`, `sheet.Parse`, `sheet.FormatPattern`, the sheet-name54-helpers) as aliases and thin wrappers, so the UI, importers and charts see55-one engine package. Typed-entry recognition (`ParseValue`, which turns56-"$1,200" or "9/26" into a number and a format) stays in `sheet`: it yields57-the engine's `Format` and reads its clock.68+`sheet.Rect`, `sheet.Parse`, `sheet.Value`, `sheet.Format`,69+`sheet.ParseValue`, `sheet.FuncDef`, `sheet.Funcs`, the remote types,70+`sheet.FormatPattern`, the sheet-name helpers) as aliases and thin71+wrappers, so the UI, importers and charts see one engine package.5872 5973 A `Workbook` holds ordered `Sheet`s, the named ranges, the undo history,6074 the arithmetic setting, the JEV source and recalculation, as a Sheets@@ -75,7 +89,8 @@ style.
7589 formulas into an AST, keeping absolute markers so references can be7690 rewritten when cells move. Its printer turns ASTs back into text in7791 Sheets' spelling. Decimal arithmetic marks the operators it computes78- with node types of the engine's own, so trees stay as parsed.92+ with node types of the function library's own, so trees stay as93+ parsed.7994 - **Recalculation.** Formulas record the cells and ranges they read on8095 their own sheet; references that name a sheet (`Sheet2!A1`) are kept by8196 name and resolved when evaluated, so renaming rewrites them and a deleted@@ -84,15 +99,17 @@ style.
8499 sheet, then evaluates the marked cells lazily in dependency order; a cell85100 reached again while it is being evaluated is part of a cycle. Volatile86101 functions (TODAY, RAND, the JEV functions) are recomputed on every87- recalculation (`recalc.go`). Formulas read other cells through a88- `lookup`: `cell` for one, `cells` for the cells a range holds, which89- the aggregates (SUM, AVERAGE, COUNT and the rest) use, sharing running90- aggregates within a recalculation (`rangememo.go`). The formulas whose91- ranges contain a changed cell are found through interval trees per92- column (`rangeindex.go`).93-- **Functions.** One table (`FuncDef`) holds every function's name,94- signature, description and arity; it drives parsing, evaluation,95- autocomplete, in-app help and [functions.md](functions.md).102+ recalculation (`recalc.go`). Formulas read other cells through each103+ sheet's `reader`, the engine's side of the function library's `Book`104+ (below): one cell, the cells a range holds in chunks, a range's bounds,105+ and the running aggregates SUM-like functions share within a106+ recalculation (`rangememo.go`). The formulas whose ranges contain a107+ changed cell are found through interval trees per column108+ (`rangeindex.go`).109+- **Functions.** One table (`FuncDef`, in `internal/functions`) holds110+ every function's name, signature, description and arity; it drives111+ parsing, evaluation, autocomplete, in-app help and112+ [functions.md](functions.md). See [Functions](#functions).96113 - **Undo.** Every mutation goes through a small set of paths that snapshot97114 the cells, widths, names, charts, view state and sheet list they change,98115 on any sheet, so a step can be reversed exactly; multi-cell operations@@ -113,9 +130,73 @@ style.
113130 and never saved: the file keeps the definition (`pivotfile.go`). A114131 frequency table is a pivot with a preset definition.115132 - **JEV.** The engine never touches the network. JEV functions describe a116- question and look up the answer in the workbook's `RemoteSource`, set117- with `SetRemote`; `internal/jev` answers from a cache and queues new118- questions, and the UI sends them as background commands.133+ question and ask the `Book` for the answer, which the engine looks up134+ in the workbook's `RemoteSource`, set with `SetRemote`; `internal/jev`135+ answers from a cache and queues new questions, and the UI sends them as136+ background commands.137+138+### Functions139+140+`internal/functions` holds the function library: the `FuncDef` table141+(one file per category: `everyday.go`, `math.go`, `stats.go`,142+`logic.go`, `text.go`, `lookup.go`, `date.go`, `finance.go`, `link.go`,143+`jev.go`, which [functions.md](functions.md) is grouped by), the144+evaluation of formulas (`eval.go`: operators and calls), the helpers145+functions share (arguments and blocks of cells in `args.go`, criteria146+over aligned ranges in `masked.go`, searched lines in `seq.go`), format147+inference for Automatic cells (`format.go`), the decimal twins and148+`Decimalize` (`decimal.go`) and the questions JEV functions ask149+(`remote.go`, `remotekey.go`). It imports `formula`, `value` and `numfmt`,150+never the engine.151+152+A formula reads cells through a `Reader` over a `Book`, the interface153+the engine implements once per sheet (`reader` in `recalc.go`):154+155+| `Book` method | What it is |156+|---|---|157+| `Cell(sheet, a)` | one cell's value, evaluated first if it's dirty |158+| `Scan(sheet, r, from, addrs, vals)` | the next chunk of the cells a range holds, row by row: their addresses and, when asked, their values |159+| `Bounds(sheet, r)` | the smallest range holding a range's cells, for the data area of blocks and lookups |160+| `RangeAgg(sheet, r)` | the running aggregate SUM-like functions share within a recalculation |161+| `Fold(sheet, r, agg)` | a range added to a SUM-like function's aggregate, as the engine reads it |162+| `Ask(call)` | a JEV question's answer, `Pending` or `ErrNoRemote` |163+164+Sheets are named as references write them, so resolving names, missing165+sheets (`#REF!`), cycles and the depth limit stay the engine's. The166+engine finds functions through the table (`LookupFunc`, which the parser167+uses) and evaluates with `functions.Eval`.168+169+The boundary costs no allocation on the hot paths, which an interface170+usually would: a callback passed through an interface escapes to the171+heap, so a `Book` that took one (as the engine's own range read did)172+would allocate for every range a formula reads. Every `Book` method takes173+and returns plain values instead:174+175+- SUM-like functions hand the engine their aggregate (`Agg`, a value) and176+ it adds a range's cells as it reads them (`Fold`), or answers from the177+ running aggregates (`RangeAgg`), calling `Agg.Add` directly.178+- Other range walks (`Reader.cells`: AND, MEDIAN, TEXTJOIN, JEV lists)179+ ask `Scan` for a chunk of cells into buffers the `Reader` keeps (they180+ start at 8 cells and double up to 1024 as a range proves long; nested181+ reads each have their own) and walk it with the function's callback,182+ which stays on the stack. A chunk evaluates no cell past the first183+ error, where every walk stops, so no cell is evaluated that reading184+ one at a time wouldn't have.185+- Lookups and criteria walk the cells' positions only (`Reader.walk`, a186+ cursor, and `stored`), in chunks that start small at every search as a187+ search stops at its match, and read the value of each cell they reach188+ (`Cell`).189+190+The engine fills a chunk from the occupancy index without a call per cell191+(`colFill`, `rangeFill`), and keeps each sheet's reader between192+recalculations, so the buffers are made once. See193+[limits.md](limits.md#the-function-library) for before and after.194+195+What stays in the engine is what needs cells or the workbook: the196+running aggregates' storage and extension (`rangememo.go`), the links a197+HYPERLINK cell opens (`link.go`), the questions a cell's JEV formulas ask198+(`RemoteCalls`), explaining errors (`explain.go`) and the decimal199+setting (`SetDecimal`, which marks formulas with `functions.Decimalize`).119200 120201 ### Where the engine can grow121202 @@ -126,8 +207,8 @@ move without touching callers:
126207 formats in side tables) replaces `cellStore`'s map; its methods are the127208 whole contract.128209 - **Range reads.** Shared range results, prefix sums for running totals129- and column-block scans go behind `lookup.cells`, the one place130- aggregates read ranges.210+ and column-block scans go behind the engine's `Scan` and `RangeAgg`,211+ the places functions read ranges.131212 - **Files.** A streaming or columnar `.012` format replaces `file.go`'s132213 whole-document JSON; the format is already separate from the store.133214 - **Depth limits.** `internal/formula`'s parser caps nesting at@@ -135,12 +216,10 @@ move without touching callers:
135216 parse before anything evaluates it; evaluation puts off cells past136217 65,536 levels of a chain rather than recursing further137218 (`evaluate.go`).138-- **Functions.** The function library (values, evaluation, `FuncDef` and139- the `functions_*.go` tables, about a third of `sheet`) is the next140- boundary: it needs cells only through `lookup`. It stays in `sheet`141- for now because `lookup` is a concrete type (an interface across a142- package boundary would put every range callback on the heap) and143- `Value` and `Format` would have to move below the engine with it.219+- **Functions.** A new function is an entry in `internal/functions`220+ that reads cells through its `Reader`; a new way of reading them (a221+ column block, a cached result) is a `Book` method that takes and222+ returns values, so the boundary stays free of allocations.144223 145224 ## The UI146225 @@ -14,7 +14,7 @@ shows it is derived from the table rather than listing it again.
1414 1515 | Registry | Entry | Derived from it |1616 |---|---|---|17-| `sheet.FuncDef` | a function: name, signature, description, arity, eval | parsing, autocomplete, argument hints, help, [functions.md](functions.md) |17+| `functions.FuncDef` (one file per category in `internal/functions`) | a function: name, signature, description, arity, eval, result format, decimal twin | parsing, autocomplete, argument hints, help, [functions.md](functions.md) |1818 | `ui.command` | an action: id, title, description, run, enabled, checked, what it edits, and how macros treat it | key bindings (Sheets and vim), menu bar, context menus, palette, shortcuts help, the `:` command line and its completions, macro recording and `run()` in scripts |1919 | `fileio` formats (`formats.go`) | a format: name, extensions, labels, traits, importer, exporter | `Import`, `Export`, detection, import picker, File > Download, command line |2020 | `chart.types`, with `sheet.ChartTypes` | a type: name and order (sheet, saved in files), a layout drawing text and image (chart) | `chart.Draw`, `chart.Image`, chart editor, Insert > Chart |@@ -23,6 +23,12 @@ shows it is derived from the table rather than listing it again.
2323 2424 Adding a function, command, format, chart type, option or role means2525 adding an entry (and its file), not editing switch statements elsewhere.26+A function goes in the category file of `internal/functions` it belongs27+to (a new file is a new section of functions.md: add it to the28+generator's `categories`), reads its arguments with the shared helpers29+(`numArg`, `matrixArg`, `each`, `criteriaArgs`), and `go test30+./internal/functions -run FunctionsDoc -update-docs` regenerates the31+reference.2632 A new process-wide setting is a `config.Options` entry read with2733 `Config.String`, `Bool`, `Int` or `Duration`; one that belongs to a2834 workbook goes in the file instead (pattern 9). A new role goes in@@ -62,10 +68,11 @@ a comment in the script, so a gap shows.
6268 6369 ### 4. Side effects stay at the edges6470 65-`internal/sheet` does no I/O, network or terminal work. Anything outside the66-process is a small interface the engine asks (`RemoteSource`, set per67-workbook with `SetRemote`) and the UI fulfils asynchronously with68-`tea.Cmd`, answering from a cache. Tests substitute fakes (`jev.Client`,71+`internal/sheet` and `internal/functions` do no I/O, network or terminal72+work. Anything outside the process is a small interface the engine asks73+(`RemoteSource`, set per workbook with `SetRemote`; functions reach it74+through `Book.Ask`) and the UI fulfils asynchronously with `tea.Cmd`,75+answering from a cache. Tests substitute fakes (`jev.Client`,6976 the fake TypeSafe server, `demos/fakejev`).7077 New remote function families (other models, web lookups, databases) plug in7178 the same way.@@ -100,13 +107,24 @@ edges and the next filled cell are found without scanning.
100107 101108 ### 7. Read ranges as ranges102109 103-Formulas read single cells through `lookup.cell` and ranges through104-`lookup.cells`, which yields only the cells a range holds, so aggregates105-(`SUM(A:A)`, running totals) cost their data, and within a recalculation106-SUM-like functions share running aggregates per range (`rangememo.go`).110+Functions read cells through their `Reader` (`internal/functions/reader.go`):111+single cells through `cell`, and ranges through `cells`, which yields only112+the cells a range holds, so aggregates (`SUM(A:A)`, running totals) cost113+their data, and within a recalculation SUM-like functions share running114+aggregates per range (`Book.RangeAgg`, the engine's `rangememo.go`) or have115+the engine add a range up as it reads it (`Book.Fold`).107116 Functions that need positions (`MATCH`, `SUMIF`, `INDEX`) take a `matrix`,108117 which knows the range's full size and which cells hold something; they109-walk those and account for the blanks between at once.118+walk those (`stored`) and account for the blanks between at once.119+120+The `Reader` is concrete, and the engine's side of it, the `Book`, takes121+and returns only values: a callback or pointer handed through an122+interface escapes to the heap, so it would allocate on every range read.123+Ranges come through `Book.Scan` in chunks, into buffers the `Reader`124+reuses, and a function's own callbacks stay on the stack; aggregates pass125+their `Agg` by value to `Book.Fold`. A new way to126+read cells follows suit: a `Book` method of values, and the loop over127+what it returns in the `Reader`.110128 111129 ### 8. Work proportional to what changed or what's visible112130 @@ -144,7 +162,11 @@ the bounds. A change to a seam reports before and after numbers.
144162 145163 `make lint`: no function over cognitive complexity 25, no Go file over 500146164 lines. A package that keeps growing past a few thousand lines gets split147-along a boundary where dependencies point one way.165+along a boundary where dependencies point one way, as the function166+library (`internal/functions`) and values (`internal/value`) were split167+from the engine: the lower package defines the small interface it needs168+(`functions.Book`) and the engine implements it, keeping its API through169+aliases so callers don't change.148170 149171 ### 12. Every feature at three levels150172 @@ -419,7 +419,7 @@ Code that cost the grid, and what it does now:
419419 420420 | Where | Was | Now |421421 |---|---|---|422-| Range reads for aggregates (`reader.cells`) | every address of the range | the stored cells, in row order |422+| Range reads for aggregates (then `reader.cells`; now `Book.Fold` and `Book.Scan`) | every address of the range | the stored cells, in row order |423423 | SUMIF(S), COUNTIF(S), AVERAGEIF(S), SUMPRODUCT, COUNTBLANK | a mask over every cell | the cells any range holds; the blanks between counted at once |424424 | MATCH, VLOOKUP, HLOOKUP, XLOOKUP | every entry of the column searched | the stored entries, the blanks between as one; ROWS and INDEX keep the full size |425425 | Range users (`rangeIndex`) | a fixed array of 256 column buckets, scanned per change | interval trees per used column, whole rows in one |@@ -488,6 +488,35 @@ whole. XLSX imports set column widths and styles for the data's columns
488488 (at least 256), without an undo step each: setting 16,384 of them one489489 step at a time took the POI file from 9.7 to 44 ms before the fix.490490 491+## The function library492+493+The function library moved out of the engine into `internal/functions`,494+behind a `Book` interface whose methods pass only values (see495+[architecture.md](architecture.md#functions)): SUM-like functions have496+the engine add a range up (`Fold`), other range walks read chunks into497+buffers the `Reader` reuses (`Scan`), and lookups walk positions with a498+cursor. The criteria and lookup shapes (`criteria-60xSUMIF8192`: SUMIF,499+COUNTIFS and AVERAGEIF over whole columns of 8192 rows;500+`lookup-300xVLOOKUP8192`: VLOOKUP, MATCH and XLOOKUP into 8192 keys) were501+added for it. Before and after, back to back (`-benchtime 1s`, medians of502+three alternating runs; differences under about 3% are noise):503+504+| Benchmark | Before | After |505+|---|---|---|506+| Edit fan-in, 1000 x SUM(A1:A8192) / SUM(A:A) | 742 / 741 us | 749 / 750 us |507+| Edit running totals | 2.79 ms | 2.85 ms |508+| Edit sparse-1M | 2.64 ms, 9611 allocs | 2.45 ms, 3610 allocs |509+| Edit criteria / lookup | 40.7 / 26.3 ms | 37.1 / 25.3 ms |510+| Edit chain / fan-out / volatile / dense | 1.88 / 1.39 / 1.29 ms / 673 ns | 1.85 / 1.40 / 1.29 ms / 662 ns |511+| RecalcAll fan-in / running / sparse-1M | 0.97 / 2.50 / 3.82 ms | 0.97 / 2.51 / 3.62 ms |512+| RecalcAll criteria / lookup | 66.6 / 27.5 ms | 58.8 / 26.5 ms |513+| RecalcAll dense 8192 x 26 | 32.5 ms | 31.8 ms |514+| Arithmetic, binary / decimal (1000 formulas and a SUM) | 20.6 / 160 us | 19.8 / 156 us |515+516+Allocations are the same or fewer everywhere: lookups no longer make a517+closure per search, and criteria collect positions from the occupancy518+index without a call per cell.519+491520 ## Hotspots found and fixed492521 493522 Each with its benchmark before and after, in the commit that fixed it.