limits.md gets the new hard limits and max-cells, the before and after numbers, and the list of code that cost the grid; formulas.md whole columns and rows; files.md sizes and column and row formats in the file; extending.md and architecture.md the occupancy index, line formats and the guardrail test; ROADMAP marks Step A done.
- commit
- 0fdf6cdc2d0f7415dd678aeca1fcb1c06fd9ae13
- parent
- ef8c0a76996d61c62cc02cba9cc74975c9161495
- author
- Dami <dami.etoile@gmail.com>
| M | ROADMAP.md | 7 | +- |
| M | docs/ | 13 | ++- |
| M | docs/ | 32 | +++++-- |
| M | docs/ | 36 | ++++++++- |
| M | docs/ | 12 | ++- |
| M | docs/ | 135 | ++++++++++++++++++++++-------- |
@@ -82,12 +82,13 @@ Avoid in the main binary: DuckDB and automerge (both need cgo).
8282 8383 ## Next: vast data8484 85-The 8192 x 256 grid is 1-2-3's, not the engine's: cells are stored sparsely.86-Raise it in measured steps (see [docs/limits.md](docs/limits.md)):85+The grid is Excel's, 1,048,576 x 16,384, since step A; memory is bounded86+by the `max-cells` budget rather than the grid. Raise that budget in87+measured steps (see [docs/limits.md](docs/limits.md)):8788 8889 | Step | What | Result |8990 |---|---|---|90-| A | Make every operation cost what the data costs, not the grid: clip whole-column and whole-row ranges to the used area, sparse range reads for aggregates, column and row formats instead of per-cell formatting, a range index sized to the used columns, three-letter columns (to XFD), wider row headers. Then raise the grid to 1,048,576 x 16,384 with a `max-cells` budget in the config file, and a stress test that fails when any command on an empty full-grid selection costs more than its data | Excel-sized grids, memory still bounded |91+| A | Make every operation cost what the data costs, not the grid: clip whole-column and whole-row ranges to the used area, sparse range reads for aggregates, column and row formats instead of per-cell formatting, a range index sized to the used columns, three-letter columns (to XFD), wider row headers. Then raise the grid to 1,048,576 x 16,384 with a `max-cells` budget in the config file, and a stress test that fails when any command on an empty full-grid selection costs more than its data. **Done**: occupancy indexes beside the cell map, sparse and clipped range reads, running aggregates shared per recalculation and interval trees of range users (1000 full-column SUMs: 174 ms to 0.5 ms per edit; 8192 running totals: 0.73 s to 2.6 ms), line formats for columns, rows and the sheet, `A:C` and `2:5` references, `max-cells` for imports and pastes, and `TestCommandsCostTheDataNotTheGrid` | Excel-sized grids, memory still bounded |9192 | B | Compact column storage behind `cellStore`: typed value blocks, formulas and formats in side tables (about 20 to 40 B per cell instead of 300) | A budget of 10 M+ cells in a few hundred MB |9293 | C | A streaming or binary file format next to the readable JSON one | Open and save scale with the data |9394 | Later | Linked, paged read-only ranges over Parquet and SQLite that feed pivots and formulas by streaming | Sources too big for any grid |@@ -66,7 +66,11 @@ style.
6666 - **Cells.** Each sheet keeps its cells in a `cellStore` (`store.go`):6767 get, set, delete, count, iterate all or a range. Nothing else touches6868 the map beneath, so the representation can change without the rest of69- the engine noticing.69+ the engine noticing. Beside it, occupancy indexes (`occupancy.go`,70+ bitmaps of 1024 rows per column) record which cells are stored and71+ which have contents, so a range of the million-row grid is read at the72+ cost of what it holds. Formats of whole columns and rows, and of the73+ whole sheet, live on the lines (`lines.go`); a cell falls back on them.7074 - **Parsing.** `internal/formula`'s hand-written Pratt parser turns7175 formulas into an AST, keeping absolute markers so references can be7276 rewritten when cells move. Its printer turns ASTs back into text in@@ -81,8 +85,11 @@ style.
8185 reached again while it is being evaluated is part of a cycle. Volatile8286 functions (TODAY, RAND, the JEV functions) are recomputed on every8387 recalculation (`recalc.go`). Formulas read other cells through a84- `lookup`: `cell` for one, `cells` for a whole range, which the85- aggregates (SUM, AVERAGE, COUNT and the rest) use.88+ `lookup`: `cell` for one, `cells` for the cells a range holds, which89+ the aggregates (SUM, AVERAGE, COUNT and the rest) use, sharing running90+ aggregates within a recalculation (`rangememo.go`). The formulas whose91+ ranges contain a changed cell are found through interval trees per92+ column (`rangeindex.go`).8693 - **Functions.** One table (`FuncDef`) holds every function's name,8794 signature, description and arity; it drives parsing, evaluation,8895 autocomplete, in-app help and [functions.md](functions.md).@@ -93,21 +93,37 @@ Code reads and writes cells through a narrow API (`cellStore` in
9393 a range, a column being a range), not the map underneath. That lets the9494 store change shape (compact column blocks, side tables for formulas and9595 formats; see [limits.md](limits.md)) without touching the rest of the96-engine.96+engine. Beside the map, occupancy indexes (`occupancy.go`) say which rows97+of each column hold a cell, and which hold contents, so a range yields its98+cells in row order at the cost of what it holds, and the used range, data99+edges and the next filled cell are found without scanning.97100 98101 ### 7. Read ranges as ranges99102 100103 Formulas read single cells through `lookup.cell` and ranges through101-`lookup.cells`, so aggregates (`SUM(A:A)`, running totals) can be served102-from column blocks, prefix sums or cached aggregates instead of one lookup103-per cell.104+`lookup.cells`, which yields only the cells a range holds, so aggregates105+(`SUM(A:A)`, running totals) cost their data, and within a recalculation106+SUM-like functions share running aggregates per range (`rangememo.go`).107+Functions that need positions (`MATCH`, `SUMIF`, `INDEX`) take a `matrix`,108+which knows the range's full size and which cells hold something; they109+walk those and account for the blanks between at once.104110 105111 ### 8. Work proportional to what changed or what's visible106112 107-Recalculation touches the cells an edit affects (dependency indexes by108-column and by sheet), rendering touches only the visible cells, and status109-statistics are cached against a version counter. A feature that scans the110-whole sheet per keystroke or per frame needs a cache or an index.113+Recalculation touches the cells an edit affects (interval trees of range114+users by column, dependency indexes by sheet), rendering touches only the115+visible cells, and status statistics are cached against a version116+counter. A feature that scans the whole sheet per keystroke or per frame117+needs a cache or an index.118+119+The grid is a million rows by 16,384 columns, so no loop may walk the120+addresses of a selection, a range or a line: walk the cells the store121+holds (`cellsIn`, `inRange`, the occupancy indexes), and keep formats of122+whole columns and rows on the line (`lines.go`). `TestCommandsCostTheDataNotTheGrid`123+(`internal/ui/vast_test.go`) holds every registered command to this: it124+runs each with the whole sheet, a whole column and a whole row selected125+and fails past 250 ms or 8 MB, so a new command that walks its selection126+fails it.111127 112128 ### 9. Additive, versioned files113129 @@ -10,7 +10,7 @@ bar; Esc cancels. Saving an imported sheet asks whether to save it as a
1010 | Format | Import | Download |1111 |---|---|---|1212 | CSV, TSV | Delimiter (`,` `;` tab `\|`), UTF-8 BOM, UTF-16 and Windows-1252 detected; entries become numbers, dates, currency and percentages as if typed; formulas stay text | Values as shown, as Sheets' Download does |13-| Excel `.xlsx` | Every sheet, opening on the one Excel showed: values, formulas (references between sheets too), workbook named ranges, number formats, bold, italic, underline, strikethrough, alignment, column widths. Formulas 012 can't read (unknown functions) keep their values; Excel pivot tables come in as the values they showed | Every sheet, the same, with formulas in Excel's syntax and their results cached. JEV functions and `#AND#` save as values, and so do pivot tables: Excel gets the results, not a pivot |13+| Excel `.xlsx` | Every sheet, opening on the one Excel showed: values, formulas (references between sheets too), workbook named ranges, number formats, bold, italic, underline, strikethrough, alignment, column widths, column and row styles. Formulas 012 can't read (unknown functions) keep their values; Excel pivot tables come in as the values they showed | Every sheet, the same, with formulas in Excel's syntax and their results cached, and column and row formats as column and row styles. JEV functions and `#AND#` save as values, and so do pivot tables: Excel gets the results, not a pivot |1414 | SQLite | Pick a table or view, or type a query; a header row names the columns | The sheet shown or the selection as a table, first row as column names; a table of that name is replaced |1515 | Parquet | Every column, with dates and timestamps; lists joined with commas | |1616 | Lotus 1-2-3 `.wk1`, `.wks` | Numbers, labels with their alignment, formats, column widths, formulas translated (references, operators, `@SUM`, `@AVG`, `@IF`, `@ROUND`, `@PMT` and 60 more) or kept as values | |@@ -48,7 +48,39 @@ made or trusted on; neither raises the version, and older builds ignore
4848 them. Opening a file never runs its macros. See [macros.md](macros.md#in-the-file).4949 5050 Other formats import as one sheet named after the file (or the SQLite51-table); CSV and TSV downloads write the sheet shown, as Sheets' do.51+table); CSV and TSV downloads write the sheet shown, as Sheets' do, and52+a selection of whole columns or rows downloads their data, not a million53+blank lines.54+55+## Size56+57+A sheet is 1,048,576 rows by 16,384 columns (A to XFD), as in Excel.58+Imports keep at most `max-cells` cells ([config.md](config.md), two59+million by default, about 600 MB): whole rows, as many as fit, and the60+context line says how many rows were left out, e.g. `only the first61+166,666 rows fit in max-cells (2,000,000 cells); 12,000 rows left out`.62+Data past the grid's edges is left out the same way. WK1 files keep their63+own 8,192 by 256. Pastes and fills that would write more than `max-cells`64+cells at once are refused. See [limits.md](limits.md).65+66+## Column and row formats67+68+Formatting whole columns or rows (select them with Ctrl+Space or69+Shift+Space, or click their headers) keeps the format on the column or70+row, as Sheets does, rather than on each of their million cells: every71+cell of it shows the format unless it has its own, and a cell typed into72+later takes it. A cell's format comes from the cell, else its row, else73+its column, else the whole sheet's (Ctrl+A twice, then a format); the74+number format and the text style fall back separately. The file keeps75+them in a `lines` field after the widths, runs of lines with the same76+formatting together, and older builds ignore it:77+78+```json79+ "lines": {"A:XFD": {"italic":true}, "B:D": {"format":"currency","decimals":2}, "1:1": {"bold":true}},80+```81+82+`A:XFD` is the whole sheet's format. A cell whose formatting its lines83+can't express (Automatic in a currency column) has `"own": true`.5284 5385 ## Several sheets5486 @@ -21,7 +21,10 @@ caret on the problem and says what's wrong on the context line, e.g.
2121 2222 ## References2323 24-- Cells: `A1`, ranges: `A1:B3` (1-2-3's `A1..B3` also works).24+- Cells: `A1`, ranges: `A1:B3` (1-2-3's `A1..B3` also works), whole25+ columns `A:C` and whole rows `2:5` (`$A:$A`, `$2:$2` absolute). Whole26+ columns and rows stay whole when copied, filled or when lines are27+ inserted and deleted; `A1:A1048576` reads back as `A:A`.2528 - Absolute parts: `$A$1`, `A$1`, `$A1`. F4 while typing cycles the reference2629 at the caret through them. Copying, filling, sorting and inserting or2730 deleting rows and columns adjust the relative parts, as in Sheets;@@ -38,8 +41,11 @@ caret on the problem and says what's wrong on the context line, e.g.
3841 and deleting rows moves references into that sheet from every sheet, and3942 leaves references to other sheets alone. Named ranges belong to the whole4043 file and may point into any sheet.41-- Each sheet is 256 columns (A to IV) by 8,192 rows, 1-2-3's size; see42- [limits.md](limits.md) for what that means in practice.44+- Each sheet is 16,384 columns (A to XFD) by 1,048,576 rows, Excel's size;45+ a reference past them (`XFE1`, `A1048577`) reads as a name. Formulas cost46+ what their ranges hold, not their size: `SUM(A:A)` over ten numbers reads47+ ten cells, and `ROWS(A:A)` is still 1,048,576. See [limits.md](limits.md)48+ for what that means in practice.4349 4450 ## Building formulas4551 @@ -10,6 +10,9 @@ Machine: Apple M5 Pro (18 cores), 48 GB, macOS, Go 1.27.1, arm64. Times
1010 are single-threaded: the engine and the UI run on one goroutine. Last full1111 run: commit 4dbe37d (after the engine and UI restructure), `BENCHTIME=2s`1212 on an otherwise idle machine; it matched the numbers below within noise.13+The grid step (see [Step A: the grid](#step-a-the-grid)) was measured14+before and after back to back, `-benchtime 1s`, commits bdc5259 and15+bb4ff53.1316 1417 The yardstick is a keystroke through to its frame: under 16 ms feels1518 instant (one frame at 60 Hz), under 100 ms feels responsive, past that@@ -19,24 +22,28 @@ it lags, and past a second it stalls.
1922 2023 | Dimension | Comfortable | Degraded | Unsupported | Dominant cost |2124 |---|---|---|---|---|22-| Sheet size | Everything up to the hard limit, 8192 x 256 (2.1 M cells): navigation and drawing don't depend on size | Loading and saving a full sheet: 1 to 2 s; 600 MB of heap | More than 8192 rows or 256 columns: imports keep the first 8192 x 256 and say what they dropped | Heap per cell (about 300 B), JSON file format |23-| Incremental recalc | A change that makes formulas read under about 500,000 cells in total (fan-out, chains and volatiles of 8192 cells: 1 to 2 ms) | 1000 SUMs over a full column (1000 x 8192 reads): 170 ms per edit | 8192 running totals (`=SUM($A$1:An)`, 33 M reads): 0.8 s per edit | About 20 ns per cell read: two map lookups |24-| Full recalc | Any sheet without heavy range fan-in: under 40 ms for 213 k numbers | 1000 full-column SUMs: 170 ms | 8192 running totals: 0.7 s | Same as above |25+| Sheet size | A grid of 1,048,576 x 16,384 (A..XFD), with up to `max-cells` cells (two million by default): navigation, drawing and every command cost what the cells cost, not the grid | Loading and saving two million cells: 1 to 2 s; 600 MB of heap | More than `max-cells` cells: imports keep whole rows up to it and say what they dropped; larger pastes and fills are refused | Heap per cell (about 300 B), JSON file format |26+| Incremental recalc | A change that makes formulas read under about 500,000 cells in total (fan-out, chains and volatiles of 8192 cells: 1 to 2 ms); 1000 SUMs over a full column: 0.7 ms; 8192 running totals: 2.7 ms | | | About 20 ns per cell read: two map lookups |27+| Full recalc | Any sheet: under 40 ms for 213 k numbers; 1000 full-column SUMs 1.1 ms; 8192 running totals 2.6 ms | | | Same as above |2528 | Rendering | Any sheet at up to 200 x 60: 1 ms a frame; 400 x 120: 4.5 ms | 20 charts at 400 x 120: 7 ms | | View building styled strings, then Bubble Tea parsing and diffing them |2629 | Selection statistics | Any selection, once computed (cached) | Extending a selection over 2 M cells: 30 ms per key | | Summing 2 M map entries per change |27-| Imports | CSV, SQLite, Parquet: 1 to 3 M cells/s (a full 2.1 M-cell sheet in about 1 s) | XLSX with formulas: 200 k cells/s | Data past the limits (dropped, with a note) | Building cells one at a time; XLSX formula translation |30+| Imports | CSV, SQLite, Parquet: 1 to 3 M cells/s (two million cells in about 1 s) | XLSX with formulas: 200 k cells/s | Data past `max-cells` or the grid (dropped, with a note) | Building cells one at a time; XLSX formula translation |2831 | Undo | One step of any size: undo costs what the edit cost | History capped at 100 steps and 256 MB of before-images: 100 whole-column steps hold 250 MB | | Whole-cell before-images, about 300 B per cell per step |2932 | JEV | Up to about 1000 JEV cells: 0.3 ms of CPU per answer | 4000 JEV cells: 1.3 ms per answer, 5 s of CPU to answer them all | | Every answer recalculates every JEV cell (they're volatile) |3033 | Formula depth | 10,000 nested parentheses or IFs: under 5 ms | | No explicit limit; recursion grows the stack | Recursive parser and evaluator |3134 | Macros | Replaying 1000 recorded actions: 2.2 ms, one undo step; a script's call to the sheet: about 1.4 us | | Scripts past 10 M Starlark steps: stopped, with the line | One message per call to the sheet, served in batches on the UI goroutine |3235 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |3336 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 16.6 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 60 fps pacing; per session, the terminal's cell buffers |34-| Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 1 to 4 ms | | Results past 8192 x 256 (the pivot shows #REF!) | Reading each source cell of its fields: a map lookup each |37+| Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 1 to 4 ms | | Results past the grid (the pivot shows #REF!) | Reading each source cell of its fields: a map lookup each |3538 3639 ## Sheet size3740 38-Hard limits: `sheet.MaxRows = 8192`, `sheet.MaxCols = 256` (A..IV), as in39-Lotus 1-2-3 Release 2.41+Hard limits: `sheet.MaxRows = 1,048,576`, `sheet.MaxCols = 16,384`42+(A..XFD), as in Excel. Memory is bounded by `max-cells` (the config file,43+two million cells by default): imports keep whole rows up to it, pastes44+and fills write at most that many cells at once. WK1 files keep their own45+8192 x 256. Past about two million cells the costs below grow with the46+cells, as they always did; only the grid is no longer a cost.4047 4148 | Measure | 8192 x 26 (213 k cells) | 8192 x 256 (2.1 M cells) |4249 |---|---|---|@@ -65,8 +72,9 @@ and a boxed number for the parsed literal.
6572 | Fan-out: one cell read by 8192 formulas | 1.3 ms | 0.9 ms | 2.7 ms |6673 | Volatile: 8192 TODAY() and RAND() | 1.25 ms | 0.8 ms | 2.5 ms |6774 | 1000 named ranges, one formula each | 48 us | 2.8 ms | 99 us |68-| Fan-in: 1000 x SUM(A1:A8192) | 174 ms | 172 ms | 350 ms |69-| Running totals: 8192 x SUM($A$1:An) | 0.73 to 0.80 s | 0.75 s | 1.45 s |75+| Fan-in: 1000 x SUM(A1:A8192), or SUM(A:A) | 0.72 ms | 1.1 ms | 1.4 ms |76+| Running totals: 8192 x SUM($A$1:An) | 2.7 ms | 2.6 ms | 5.4 ms |77+| Sparse: 10,000 numbers spread down a million rows, 1000 each of SUM(A:A), running totals and VLOOKUP over A:B | 2.6 ms | 3.9 ms | 5.1 ms |7078 7179 Every volatile formula is recomputed on every change, as in Sheets, so7280 8192 volatile cells add about 1.2 ms to every edit anywhere.@@ -124,11 +132,14 @@ Real, openly licensed datasets fetched by `scripts/stress-data.sh`
124132 | Chinook PlaylistTrack (SQLite) | 8715 x 2 | 5.5 ms | 3.0 M cells/s | 293 B |125133 | Parquet alltypes_tiny_pages | 7300 x 13 | 40 ms | 2.4 M cells/s | 276 B |126134 127-Past the limits, imports keep the first 8192 rows and 256 columns and128-say how much they left out. Parquet files and SQLite tables stop reading129-at the last row and take the count of the rest from the file; CSV, TSV130-and SQLite queries are read to their end to count it, without keeping131-it (the airport codes file is read to its end). Every importer but XLSX135+Past `max-cells` (or the grid), imports keep whole rows up to it and136+say how many rows they left out. Parquet files and SQLite tables stop137+reading at the last row and take the count of the rest from the file;138+CSV, TSV and SQLite queries are read to their end to count it, without139+keeping it. The table above is from before the grid grew, when files140+were cut at row 8192; now the airport codes (855 k cells, 0.80 s) and141+OWID energy (1.04 M cells, 0.95 s) come in whole, at the same rate of142+about a million cells a second. Every importer but XLSX132143 streams, so memory follows the sheet, not the file; excelize holds an133144 XLSX worksheet in memory while its rows are read.134145 @@ -275,11 +286,67 @@ step whatever its size (`BenchmarkMacroReplay`, `BenchmarkMacroScript`,
275286 | A script's loop: 5000 `set` and 5000 `get` calls | 13.7 ms | 1.4 us per call | 220 k, 7.8 MB |276287 277288 Reading a range walks its cells; a range larger than 4096 cells (whole278-columns) is first trimmed to the used range, which costs a pass over the279-sheet, so reading single cells never does. Every run stops after 10289+columns) is first trimmed to its last row and column with contents,290+found through the sheet's index. Every run stops after 10280291 million Starlark steps (`macro.DefaultMaxSteps`), about a second of pure281292 computation.282293 294+## Step A: the grid295+296+The grid grew from 8192 x 256 to 1,048,576 x 16,384 once nothing cost297+what the grid costs. Beside the cell map, occupancy indexes (bitmaps of298+1024 rows per column) say which cells are stored and which have299+contents; ranges are read through them, so a whole column of ten cells is300+ten visits. Formats of whole columns, rows and the sheet live on the301+lines. `TestCommandsCostTheDataNotTheGrid` (`internal/ui`) runs every302+command with the whole sheet, a whole column and a whole row selected on303+the full grid and fails past 250 ms or 8 MB; each takes 0.1 to 4 ms.304+305+Code that cost the grid, and what it does now:306+307+| Where | Was | Now |308+|---|---|---|309+| Range reads for aggregates (`reader.cells`) | every address of the range | the stored cells, in row order |310+| SUMIF(S), COUNTIF(S), AVERAGEIF(S), SUMPRODUCT, COUNTBLANK | a mask over every cell | the cells any range holds; the blanks between counted at once |311+| MATCH, VLOOKUP, HLOOKUP, XLOOKUP | every entry of the column searched | the stored entries, the blanks between as one; ROWS and INDEX keep the full size |312+| Range users (`rangeIndex`) | a fixed array of 256 column buckets, scanned per change | interval trees per used column, whole rows in one |313+| Formatting whole columns or rows | up to 65,536 cells made, then only existing ones | column, row and sheet formats |314+| Used range, Ctrl+arrow, text running into view | every cell, or every row or column between | the index of filled cells |315+| Filter over whole columns | every row tested and hidden one by one | rows with data, one test for the blank rest, stepped over at once |316+| Sort | every row of the range | the rows with cells |317+| Paste, fill of blanks | every address of the destination | the clip's cells per tile; what blanks land on is cleared |318+| Copy to the system clipboard | every blank row between values | refused past `max-cells` cells |319+| Pivot sources, JEV value ranges, chart detection | every row of the range | the rows with data; JEV ranges past 4096 cells trimmed to their data |320+| Exports of a selection, script reads | every row of whole columns | trimmed to the data |321+| Row numbers, name box | 5 digits, IV | 7 digits (the header widens past 9999), XFD |322+| Column width reset, set | an undo step per column of the selection | one step; a reset touches only columns with a width |323+| XLSX column widths and styles | 256 columns | the data's width (at least 256), a style on A and XFD as the sheet's |324+325+Before and after, back to back (`-benchtime 1s`, medians of one run):326+327+| Benchmark | Before | After |328+|---|---|---|329+| Edit fan-in, 1000 x SUM(A1:A8192) | 176 ms | 0.72 ms |330+| Edit running totals, 8192 x SUM($A$1:An) | 0.73 s | 2.7 ms |331+| RecalcAll fan-in / running totals | 178 / 739 ms | 1.1 / 2.6 ms |332+| Undo + redo fan-in / running totals | 352 ms / 1.47 s | 1.4 / 5.4 ms |333+| Typing an entry under 1000 SUMs, through to its frame (200 x 60) | 59 ms | 0.91 ms |334+| Edit sparse-1M (10,000 numbers over a million rows, 3000 whole-column formulas) | | 2.6 ms |335+| Frame at 200 x 60 on sparse-1M; arrow keys at XFD1048576 | | 0.68 ms; 0.65 ms |336+| Edit a cell of 8192 x 26 numbers | 619 ns | 662 ns |337+| Open 8192 x 26 / 8192 x 256 | 127 ms / 1.97 s | 130 ms / 1.99 s |338+| Build 8192 x 26 through `Load` | 88 ms | 90 ms |339+| BigUndo: clear 8192 x 26, undo | 159 ms | 164 ms |340+| Sort 8191 rows | 31 ms | 32 ms |341+| Pivot over 8191 rows, 8 categories | 0.89 ms | 0.77 ms |342+| Extend a selection of 8192 x 256 by a row | 31 ms | 35 ms (measured alone) |343+344+The index costs a few percent on edits and loads (keeping two bitmaps345+current per cell), and running aggregates cost memory for their346+checkpoints (a 1000-SUM fan-in allocates 2 MB per recalculation instead347+of 0.2 MB). A range is shared only from its second read in a348+recalculation, so ranges read once cost nothing extra.349+283350 ## Hotspots found and fixed284351 285352 Each with its benchmark before and after, in the commit that fixed it.@@ -293,14 +360,17 @@ Each with its benchmark before and after, in the commit that fixed it.
293360 | `Names()` upper-cased both names in every sort comparison, every frame | 21 k allocations per View with 1000 names | View/names 80 x 24: 662 us | 197 us |294361 | Status line summed a whole-sheet selection with 2 M map lookups, every frame | CPU profile: `RangeStats` 87% of a frame | 91 ms per frame | 0.3 ms (cached), 29 ms when the selection changes |295362 | JEV cache keys encoded with `encoding/json` | CPU profile of answers: JSON 60% of recalc | JEV/1000 575 ms, JEV/4000 9.3 s | 263 ms, 5.1 s |363+| Aggregates read every cell of every range, 1000 identical SUMs a thousand times, running totals O(n^2) | Fan-in and running-total edits | Edit/fanin 176 ms, running 0.73 s | 0.72 ms, 2.7 ms (running aggregates shared per recalculation) |364+| Range users bucketed by a fixed array of 256 columns, scanned whole per changed cell | 16,384 columns; whole-row ranges in every bucket | | Interval trees per column, wide ranges in one tree |365+| A range read visited every address, blank or not | A whole column is a million addresses | | Occupancy index: only stored cells are visited |296366 297367 ## What would raise the bounds298368 299-In order of value for effort. The first three are what raising the300-limits toward Google Sheets' 10 M cells (or Excel's 1,048,576 x 16,384)369+In order of value for effort. The grid is Excel's since step A; the370+first two are what raising `max-cells` toward Google Sheets' 10 M cells301371 would take; without them a 10 M-cell sheet would need about 3 GB of302-heap, take 10 s to open and 140 ms per key to extend a selection over303-it.372+heap and take 10 s to open. (Shared range results, which was the second,373+is done: see Step A.)304374 305375 1. **Compact cell storage** (L, 2 to 3 weeks). Cells live in a306376 `map[Addr]*Cell` behind `cellStore` (`internal/sheet/store.go`), 300 B@@ -315,38 +385,29 @@ it.
315385 index (2 to 3 ns). Cells are already per sheet in a workbook, so the316386 storage can change sheet by sheet, and nothing outside `cellStore`317387 touches the map, so the change stays inside it.318-2. **Range dependency index and shared range results** (M, 1 week).319- Range users are now indexed by column; an interval index per column320- (or per row block) would make finding them independent of how many321- share a column. Memoizing aggregate results per range within one322- recalc would make 1000 identical `SUM(A1:A8192)` cost one, and323- running totals could be computed from a prefix sum per column block:324- the fan-in and running-total rows of the table above would drop to325- milliseconds. Aggregates already read ranges through `lookup.cells`326- (`internal/sheet/recalc.go`), where such results would be served.327-3. **Streaming, compact file format** (M, 1 week). The `.012` file is388+2. **Streaming, compact file format** (M, 1 week). The `.012` file is328389 JSON decoded whole (1.3 GB allocated to open 47 MB). A streaming329390 decoder over the same format would roughly halve open time and cut330391 allocation tenfold; a columnar or gzip-compressed variant would cut331392 the size about fourfold. The format is `internal/sheet/file.go`, apart332393 from the cell store.333-4. **Incremental selection statistics** (S, 2 days). Keep per-column394+3. **Incremental selection statistics** (S, 2 days). Keep per-column334395 sums and counts, updated in `place` and after recalc, and compute a335396 selection's Sum and Count from column totals minus the rows outside336397 it; extending a whole-sheet selection would drop from 29 ms to well337398 under 1 ms.338-5. **JEV recalc by question** (S to M, 3 days). Index JEV cells by399+4. **JEV recalc by question** (S to M, 3 days). Index JEV cells by339400 question key and recalculate only the cells whose question was340401 answered (and their dependents), and coalesce answers that arrive in341402 the same frame: answering 4000 questions would cost 4000 small342403 recalcs instead of 4000 full ones.343-6. **Smaller undo steps** (S, 1 day). The history is capped by the404+5. **Smaller undo steps** (S, 1 day). The history is capped by the344405 memory its before-images hold; storing formatting-only changes as345406 diffs rather than whole cells would let it keep more of them.346-7. **Frame rate** (S, hours). Bubble Tea draws at most 60 frames a407+6. **Frame rate** (S, hours). Bubble Tea draws at most 60 frames a347408 second; asking for 120 would halve the key-to-screen floor from about348409 16 ms to 8 ms at the cost of more redraws.349-8. **Recursion limits** (S, hours). The parser (`internal/formula`) and410+7. **Recursion limits** (S, hours). The parser (`internal/formula`) and350411 the evaluator recurse without a limit; a depth cap (Excel allows 64351412 nested functions) would turn a pathological file into an error rather352413 than a deep stack.@@ -360,6 +421,10 @@ make stress-report # latest against previous and baseline, with trends
360421 make stress-e2e # key press to screen through libghostty361422 ```362423 424+`TestCommandsCostTheDataNotTheGrid` runs with `go test ./internal/ui`425+(it skips under `-short`) and guards the grid: a command whose cost grows426+with its selection's area fails it.427+363428 Benchmarks are built only with `-tags stress`, so `go test ./...` stays364429 fast. They live in `internal/sheet/stress_test.go` (engine),365430 `internal/sheet/hotpath_stress_test.go` (parsing and number formats),