architecture.md describes spills in the engine and arrays in the function library, extending.md a function's arrays field and arrayArg, limits.md what arrays store and cost with the 1000-array stress shape, and ROADMAP.md marks dynamic arrays and the text and regex functions done. Co-Authored-By: Claude <noreply@anthropic.com>
- commit
- 5b9b9b39c98bd7a73e41aca3bae9127f71b9aef8
- parent
- d7fd93f36280ce94da44255f78415269dd2fcc38
- author
- Dami <dami.etoile@gmail.com>
| M | ROADMAP.md | 4 | +- |
| M | docs/ | 46 | +++++++++++++++++++++++++++--- |
| M | docs/ | 4 | ++- |
| M | docs/ | 34 | ++++++++++++++++++++++ |
@@ -118,8 +118,8 @@ work left open. Sizes: S (a day or two), M (about a week), L (weeks).
118118 119119 | Item | Notes | Size |120120 |---|---|---|121-| Dynamic arrays: FILTER, SORT, UNIQUE, SEQUENCE, spill ranges, `#SPILL!`; then LET and LAMBDA | The biggest gap in the function library; spills need engine support for ranges a formula owns | L |122-| Text and regex: TEXTJOIN, SPLIT, REGEXMATCH, REGEXEXTRACT, REGEXREPLACE | Sheets staples | S |121+| Dynamic arrays: FILTER, SORT, UNIQUE, SEQUENCE, spill ranges, `#SPILL!`; then LET and LAMBDA. Done: arrays in array contexts and ARRAYFORMULA, spills owned by their formula (`#REF!` with Sheets' explanation when blocked), FILTER, SORT, SORTN, UNIQUE, SEQUENCE, TRANSPOSE, FLATTEN, CHOOSECOLS and CHOOSEROWS, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL and MAKEARRAY, and XLSX's dynamic array formulas; see [formulas.md](docs/formulas.md#arrays-and-spills) | The biggest gap in the function library; spills need engine support for ranges a formula owns | L |122+| Text and regex: TEXTJOIN, SPLIT, REGEXMATCH, REGEXEXTRACT, REGEXREPLACE. Done, on RE2 as Sheets ([formulas.md](docs/formulas.md#regular-expressions)) | Sheets staples | S |123123 | Conditional formatting (color scales, rules on values and formulas), drawn in theme roles | Visual; must read under all 349 schemes | M |124124 | Data validation: dropdown lists (a picker in the cell), number and date rules, checkboxes | Pairs with the filter picker | M |125125 | Wrap text, row heights, borders, merged cells | Layout changes in the grid renderer | M to L |@@ -142,6 +142,20 @@ style.
142142 constants to formulas, values to copies and exports, refused by `Set`,143143 and never saved: the file keeps the definition (`pivotfile.go`). A144144 frequency table is a pivot with a preset definition.145+- **Spills.** A formula computing an array (its anchor) shows the first146+ value and spills the rest into the cells to its right and below147+ (`spill.go`): spilled cells are derived cells as a pivot's results148+ are, flagged `spilled`. Each evaluation pass notes the arrays anchors149+ computed; after it, each is written (only the cells that changed, and150+ nothing when the array and its cells are as they were), outside the151+ undo history, then what reads the changed cells is recalculated, which152+ may spill again (at most 64 passes). A sheet indexes its anchors by153+ the cells they cover or would (an interval tree, as range users are),154+ so a cell placed in a spill or in the way of a blocked one has its155+ anchor recalculate, and a spilled cell placed (by undo, a move or a156+ format) keeps only its formatting and note. A blocked anchor shows157+ `#REF!` with the reason. `Set` refuses spilled cells; files keep only158+ the anchor.145159 - **JEV.** The engine never touches the network. JEV functions describe a146160 question and ask the `Book` for the answer, which the engine looks up147161 in the workbook's `RemoteSource`, set with `SetRemote`; `internal/jev`@@ -152,9 +166,11 @@ style.
152166 153167 `internal/functions` holds the function library: the `FuncDef` table154168 (one file per category: `everyday.go`, `math.go`, `stats.go`,155-`logic.go`, `text.go`, `lookup.go`, `date.go`, `finance.go`, `link.go`,156-`jev.go`, which [functions.md](functions.md) is grouped by), the157-evaluation of formulas (`eval.go`: operators and calls), the helpers169+`logic.go`, `text.go`, `regex.go`, `lookup.go`, `dynamic.go`,170+`lambda.go`, `date.go`, `finance.go`, `link.go`, `jev.go`, which171+[functions.md](functions.md) is grouped by), the evaluation of formulas172+(`eval.go`: operators and calls; arrays in `array.go`, functions mapped173+over them in `lift.go`, the names LET and LAMBDA bind in `scope.go`), the helpers158174 functions share (arguments and blocks of cells in `args.go`, criteria159175 over aligned ranges in `masked.go`, searched lines in `seq.go`), format160176 inference for Automatic cells (`format.go`), the decimal twins and@@ -177,10 +193,30 @@ the engine implements once per sheet (`reader` in `recalc.go`):
177193 Sheets are named as references write them, so resolving names, missing178194 sheets (`#REF!`), cycles and the depth limit stay the engine's. The179195 engine finds functions through the table (`LookupFunc`, which the parser180-uses) and evaluates a cell's formula with `functions.EvalAt`, giving the196+uses) and evaluates a cell's formula with `functions.EvalCell`, giving the181197 cell's address, so a range used where one value is wanted reads the cell182198 in the formula's row or column (implicit intersection; the `Reader`183-keeps the address, and the one before it while formulas nest).199+keeps the address, and the one before it while formulas nest), and200+getting back the array it computed, if any, to spill.201+202+Arrays (`functions.Array`: rows and columns of values, of which only the203+top-left block holding data is stored, the rest one fill value, so a204+whole column read as an array costs what it holds) travel through the205+evaluator as a `Value` of kind `value.Array` whose number indexes the206+`Reader`'s arena, so a cell's `Value` stays as small as it is; the arena207+is emptied when the outermost formula is done. Where an array goes208+depends on what asked: an operator works value by value, a function209+that takes ranges reads it whole (`arrayArg`, `matrixArg`, `each`), and210+anywhere else one value is wanted it reads as its first. In an array211+context (ARRAYFORMULA, or an argument taking ranges) a range reads whole212+and a function of one value is called once with its arguments standing213+in (`liftArg`): those it reads as one value are the ones mapped over, so214+VLOOKUP maps over its keys and not its table without a list of which215+argument is which. Each `FuncDef` says whether it takes one value, may216+pass an array through (IF, IFERROR, INDEX) or takes arrays itself217+(FILTER, LET). Ranges read whole are shared by the formulas of a218+recalculation (`Reader.Forget` empties them as one starts and ends), as219+running aggregates are.184220 185221 The boundary costs no allocation on the hot paths, which an interface186222 usually would: a callback passed through an interface escapes to the@@ -26,7 +26,9 @@ adding an entry (and its file), not editing switch statements elsewhere.
2626 A function goes in the category file of `internal/functions` it belongs2727 to (a new file is a new section of functions.md: add it to the2828 generator's `categories`), reads its arguments with the shared helpers29-(`numArg`, `matrixArg`, `each`, `criteriaArgs`), and `go test29+(`numArg`, `matrixArg`, `each`, `criteriaArgs`, `arrayArg` for arrays),30+says in `arrays` whether it takes arrays itself or may return one31+(otherwise array contexts map it over arrays for free), and `go test3032 ./internal/functions -run FunctionsDoc -update-docs` regenerates the3133 reference.3234 A new process-wide setting is a `config.Options` entry read with@@ -33,6 +33,7 @@ it lags, and past a second it stalls.
3333 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |3434 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 8.7 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 120 fps pacing; per session, the terminal's cell buffers |3535 | Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 1 to 4 ms | | Results past the grid (the pivot shows #REF!) | Reading each source cell of its fields: a map lookup each |36+| Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 55 ms | Full recalculation of those 1000: 204 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |3637 3738 ## Sheet size3839 @@ -400,6 +401,39 @@ times the edit, the recalculation and the pivot together:
400401 | Rows: 1000 items sorted by their sum; SUM and COUNTUNIQUE | 1001 | 3.5 to 4.2 ms | 14 k |401402 | Frequency table of the 1000 items | 1001 | 2.5 to 2.8 ms | 24 k |402403 404+## Arrays and spills405+406+An array stores only its block holding data: a whole column read as one407+(`FILTER(A:A, ...)`, `SORT(B:B)`) holds the rows with data, and the408+blank rows past them are one fill value, spilled as nothing. A range409+read whole is read once per recalculation and shared by every formula410+reading it (`Reader.Forget`), so a thousand FILTERs of one column read411+it once. An array stores at most `maxArray` values (2,097,152; about 64412+MB), past which the formula is `#VALUE!`; a spill writes at most413+`max-cells` cells. Spilled cells are written after each evaluation pass414+(`spill.go`), only those that changed, and not at all when an anchor415+computes the array it already spilled with nothing in its cells416+changed; formulas reading spilled cells that changed are recalculated in417+another pass, at most 64 of them.418+419+`BenchmarkEdit/arrays-1000xFILTER8192` (`internal/stress`: a column of420+8192 numbers beside eight categories, 500 FILTERs of the numbers by a421+category, 1024 rows each, and 500 UNIQUEs of the categories, all from422+row 1, 516 k spilled cells) edits one number, recomputing the 500423+FILTERs, 63 of whose arrays change:424+425+| Measure | Time | Allocated |426+|---|---|---|427+| Edit a number | 55 ms | 161 MB, 8.3 k allocations |428+| Undo and redo it | 110 ms | |429+| Full recalculation | 204 ms | |430+| Build (load and recalculate) | 310 ms | |431+432+Most of an edit is the FILTERs' own work: comparing 8192 categories each433+(`B1:B8192="alpha"`, about 20 ns a value) and picking the rows that434+pass. Array arithmetic allocates the array it computes, so an edit435+allocates about 32 bytes per value its conditions compare.436+403437 ## Macros404438 405439 A macro's script runs on its own goroutine; every call it makes to the