Bounds of support takes the measurements at ten million cells and the store's costs per cell, drops compact storage from what would raise the bounds, and names the file format and undo before-images as what stands in the way now. Pattern 6 says what get hands out and which narrower reads to use. The roadmap lists compact storage as shipped.
- commit
- 6ef84602a33593c2ab1a1d5e28e83b19bfb33912
- parent
- d1a4ab3aa488ed9a2518a1f2b979d4f71bf11905
- author
- Dami <dami.etoile@gmail.com>
| M | ROADMAP.md | 8 | +- |
| M | docs/ | 21 | ++- |
| M | docs/ | 22 | ++- |
| M | docs/ | 216 | ++++++++++++++++-------------- |
| M | docs/ | 6 | +- |
@@ -32,14 +32,13 @@ Sizes: S (a day or two), M (about a week), L (weeks).
3232 ### 3. Scale3333 3434 The grid is Excel's, 1,048,576 x 16,384, and memory is bounded by the35-`max-cells` budget rather than the grid. The budget rises in measured36-steps; see [Bounds of support](docs/contributing/limits.md#what-would-raise-the-bounds).35+`max-cells` budget (ten million cells) rather than the grid. The budget36+rises in measured steps; see [Bounds of support](docs/contributing/limits.md#what-would-raise-the-bounds).3737 3838 | Item | Result | Size |3939 |---|---|---|40-| Compact column storage behind `cellStore`: typed value blocks, formulas and formats in side tables (about 20 to 40 B per cell instead of 300) | A budget of 10 M+ cells in a few hundred MB, cell reads 2 to 3 ns | L |4140 | A streaming or binary `.012` format next to the readable JSON one | Open and save scale with the data | M |42-| Smaller undo steps: formatting changes as diffs | More history in the same memory | S |41+| Smaller undo steps: plain cells' before-images as slots, formatting changes as diffs | More history in the same memory; clearing a full sheet costs what the sheet does | S to M |4342 | Linked, paged read-only ranges over Parquet and SQLite that feed pivots and formulas by streaming | Sources too big for any grid | L |4443 4544 ### 4. Macros, keys and the terminal@@ -87,6 +86,7 @@ by a gateway dialing the iroh ticket.
8786 - Dynamic arrays that spill (FILTER, SORT, UNIQUE, SEQUENCE and more, ARRAYFORMULA), LET and LAMBDA, SPLIT and the REGEX functions: [Arrays and spills](docs/formulas/arrays.md)8887 - Opt-in decimal arithmetic for money: [Decimal arithmetic](docs/formulas/decimal.md)8988 - An Excel-sized grid with a `max-cells` budget; operations cost the data, not the grid: [Bounds of support](docs/contributing/limits.md#sheet-size)89+- Compact column storage: 20 B a number, a ten-million-cell `max-cells` budget, 3 ns cell reads: [Bounds of support](docs/contributing/limits.md#sheet-size)9090 9191 **Finding and using features**9292 @@ -92,12 +92,19 @@ was typed, its parsed expression, its computed value, and its format and
9292 style.9393 9494 - **Cells.** Each sheet keeps its cells in a `cellStore` (`store.go`):95- get, set, delete, count, iterate all or a range. Nothing else touches96- the map beneath, so the representation can change without the rest of97- the engine noticing. Beside it, occupancy indexes (`occupancy.go`,95+ get, set, delete, count, iterate all or a range, and narrower reads96+ (a value, presence, formatting, the formula cells). Nothing else97+ touches the representation beneath. Occupancy indexes (`occupancy.go`,9898 bitmaps of 1024 rows per column) record which cells are stored and9999 which have contents, so a range of the million-row grid is read at the100- cost of what it holds. Formats of whole columns and rows, and of the100+ cost of what it holds; the stored index's blocks hold the cells101+ themselves, a 16-byte slot each in row order (`slot.go`). A plain cell102+ (a number, boolean or text as typed, with a format and style) lives in103+ its slot, with its text in a table of strings and its formatting in a104+ table of looks; formulas, notes, and what pivots and spills write are105+ whole `Cell`s in a side table, which recalculation updates in place.106+ `get` hands out a plain cell as a `Cell` made for the caller, a copy107+ whose changes reach nothing; `set` is the one way to change a cell. Formats of whole columns and rows, and of the101108 whole sheet, live on the lines (`lines.go`); a cell falls back on them.102109 Copy, paste and move carry the formatting cells show (`clipfmt.go`):103110 whole lines as line formats, blocks as the cells' own. A line's format@@ -273,9 +280,9 @@ setting (`SetDecimal`, which marks formulas with `functions.Decimalize`).
273280 The package boundaries leave the bounds in [Bounds of support](limits.md) room to274281 move without touching callers:275282 276-- **Storage.** Compact cell storage (column blocks of values, formulas and277- formats in side tables) replaces `cellStore`'s map; its methods are the278- whole contract.283+- **Storage.** `cellStore`'s methods are the whole contract, so what is284+ kept whole or in a slot can change (spilled and pivot cells in slots,285+ plain before-images in the history) without touching callers.279286 - **Range reads.** Shared range results, prefix sums for running totals280287 and column-block scans go behind the engine's `Scan` and `RangeAgg`,281288 the places functions read ranges.@@ -117,13 +117,21 @@ modified.
117117 118118 Code reads and writes cells through a narrow API (`cellStore` in119119 `internal/sheet/store.go`: get, set, delete, count, iterate everything or120-a range, a column being a range), not the map underneath. That lets the121-store change shape (compact column blocks, side tables for formulas and122-formats; see [Bounds of support](limits.md)) without touching the rest of the123-engine. Beside the map, occupancy indexes (`occupancy.go`) say which rows124-of each column hold a cell, and which hold contents, so a range yields its125-cells in row order at the cost of what it holds, and the used range, data126-edges and the next filled cell are found without scanning.120+a range, a column being a range), never the representation underneath:121+column blocks of 16-byte slots holding plain cells' values, with text,122+formats and whole formula cells in side tables (see123+[Bounds of support](limits.md#sheet-size)). `get` returns a `*Cell`:124+a formula's own, which recalculation updates, or for a plain cell one125+made for the caller, which it may keep but whose changes reach nothing;126+every change goes through `set`. Making that `Cell` costs an allocation127+or three, so code that visits many cells asks for only what it needs:128+`value` and `peek` for values, `has` and `filledAt` for presence,129+`look` for formatting, `richAt` and `richCells` for formulas and notes,130+and the key iterators (`keys`, `keysIn`, `anyKeysIn`) for addresses. The131+stored occupancy index (`occupancy.go`) holds the blocks; a second says132+which rows hold contents, so a range yields its cells in row order at133+the cost of what it holds, and the used range, data edges and the next134+filled cell are found without scanning.127135 128136 ### 7. Read ranges as ranges129137 @@ -25,51 +25,63 @@ it lags, and past a second it stalls.
2525 2626 | Dimension | Comfortable | Degraded | Unsupported | Dominant cost |2727 |---|---|---|---|---|28-| Sheet size | A grid of 1,048,576 x 16,384 (A..XFD), with up to `max-cells` cells (two million by default): navigation, drawing and every command cost what the cells cost, not the grid | Loading and saving two million cells: 1 to 2 s; 600 MB of heap | More than `max-cells` cells: imports keep whole rows up to it and say what they dropped; larger pastes and fills are refused | Heap per cell (about 300 B), JSON file format |29-| Incremental recalc | A change that makes formulas read under about 500,000 cells in total (fan-out, chains and volatiles of 8192 cells: 1 to 2 ms); 1000 SUMs over a full column: 0.75 ms; 8192 running totals: 2.8 ms | 60 criteria functions (SUMIF, COUNTIFS, AVERAGEIF) over whole columns of 8192 rows: 37 ms an edit | | About 20 ns per cell read: two map lookups |30-| Full recalc | Any sheet: under 40 ms for 213 k numbers; 1000 full-column SUMs 1 ms; 8192 running totals 2.5 ms | 60 whole-column criteria functions over 8192 rows: 59 ms | | Same as above |28+| Sheet size | A grid of 1,048,576 x 16,384 (A..XFD), with up to `max-cells` cells (ten million by default): navigation, drawing and every command cost what the cells cost, not the grid; ten million numbers take 205 MB and build in 2.2 s | Opening a `.012` file of ten million cells: 8 s and 2.7 GB at its peak; saving it: 3.7 s | More than `max-cells` cells: imports keep whole rows up to it and say what they dropped; larger pastes and fills are refused | JSON file format, decoded whole; heap per cell (about 20 B for numbers, 20 to 60 B for real data, 750 B for formulas) |29+| Incremental recalc | A change that makes formulas read under about 500,000 cells in total (fan-out, chains and volatiles of 8192 cells: 1.3 to 1.9 ms); 1000 SUMs over a full column: 0.53 ms; 8192 running totals: 2.7 ms | 60 criteria functions (SUMIF, COUNTIFS, AVERAGEIF) over whole columns of 8192 rows: 29 ms an edit | | About 3 ns per cell read: an index into the column's block |30+| Full recalc | Any sheet: numbers and text hold their values and cost nothing; 1000 full-column SUMs 0.5 ms; 8192 running totals 1.8 ms | 60 whole-column criteria functions over 8192 rows: 44 ms | | Same as above |3131 | Rendering | Any sheet at up to 200 x 60: 1 ms a frame; 400 x 120: 4.5 ms; a color scale on every cell shown adds 0.4 ms at 200 x 60 | 20 charts at 400 x 120: 7 ms | | View building styled strings, then Bubble Tea parsing and diffing them |32-| Selection statistics | Any selection: extending one over all 2 M cells costs 0.4 ms a key | | | Per column with data, 1024 or 64 rows at a time from an index on the blocks of filled cells, plus the rows at the selection's ends |33-| Imports | CSV, SQLite, Parquet: 1 to 3 M cells/s (two million cells in about 1 s); XLSX numbers or text: 0.7 to 1.1 M cells/s | XLSX with formulas: 0.4 to 0.6 M cells/s | Data past `max-cells` or the grid (dropped, with a note); XLSX files past the reader's limits (refused) | Building cells one at a time; XML decoding; XLSX formula translation |34-| Undo | One step of any size: undo costs what the edit cost | History capped at 100 steps and 256 MB of before-images: 100 whole-column steps hold 250 MB | | Whole-cell before-images, about 300 B per cell per step |32+| Selection statistics | Any selection: extending one over all 2 M cells costs 0.3 ms a key | | | Per column with data, 1024 or 64 rows at a time from an index on the blocks of filled cells, plus the rows at the selection's ends |33+| Imports | CSV, SQLite, Parquet: 2 to 4 M cells/s (a million cells in 0.25 to 0.4 s); XLSX numbers or text: 0.8 to 1.4 M cells/s | XLSX with formulas: 0.4 to 0.6 M cells/s | Data past `max-cells` or the grid (dropped, with a note); XLSX files past the reader's limits (refused) | Building cells one at a time; XML decoding; XLSX formula translation |34+| Undo | One step of any size: undo costs what the edit cost | History capped at 100 steps and 256 MB of before-images: 100 whole-column steps hold 280 MB | | Whole-cell before-images, about 300 B per cell per step |3535 | JEV | 4000 JEV cells: 4 us of CPU per answer, 15 ms to answer them all | | | An answer recalculates the cells that asked it; answers within a frame recalculate together |3636 | Formula depth | 1000 nested parentheses or IFs: under 0.5 ms; chains of formulas through every cell of a sheet | | More than 1024 levels of nesting in one formula: a parse error | Recursive parser; evaluation puts off cells past 65,536 levels |3737 | Macros | Replaying 1000 recorded actions: 2.2 ms, one undo step; a script's call to the sheet: about 1.4 us | | Scripts past 10 M Starlark steps: stopped, with the line | One message per call to the sheet, served in batches on the UI goroutine |3838 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |3939 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 8.7 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 120 fps pacing; per session, the terminal's cell buffers |40-| Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 1 to 4 ms | | Results past the grid (the pivot shows #REF!) | Reading each source cell of its fields: a map lookup each |41-| Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 55 ms | Full recalculation of those 1000: 204 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |40+| Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 0.8 to 2.5 ms | | Results past the grid (the pivot shows #REF!) | Grouping the source rows |41+| Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 52 ms | Full recalculation of those 1000: 217 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |4242 4343 ## Sheet size4444 4545 Hard limits: `sheet.MaxRows = 1,048,576`, `sheet.MaxCols = 16,384`4646 (A..XFD), as in Excel. Memory is bounded by `max-cells` (the config file,47-two million cells by default): imports keep whole rows up to it, pastes47+ten million cells by default): imports keep whole rows up to it, pastes4848 and fills write at most that many cells at once. WK1 files keep their own49-8192 x 256. Past about two million cells the costs below grow with the50-cells; the grid itself costs nothing (see [The grid](#the-grid)).49+8192 x 256. The costs below grow with the cells; the grid itself costs50+nothing (see [The grid](#the-grid)). At ten million cells the file51+format is what degrades: a `.012` file is JSON decoded whole.5152 52-| Measure | 8192 x 26 (213 k cells) | 8192 x 256 (2.1 M cells) |53-|---|---|---|54-| Build through `Load`, then full recalc | 90 ms | 1.4 s |55-| Live heap | 64 MB | 602 MB (about 300 B per number cell) |56-| Save `.012` (JSON, one line per cell) | 74 ms, 4.6 MB | 1.04 s, 47 MB |57-| Open `.012` | 120 to 130 ms (116 MB allocated) | 1.9 to 2 s (1.3 GB allocated) |58-| Export CSV / TSV / XLSX / SQLite | 41 / 55 / 62 / 40 ms | 0.71 / 0.66 / 0.81 / 0.72 s |59-| Frame at 200 x 60, whole | 1.06 ms | 1.05 ms |53+| Measure | 8192 x 26 (213 k cells) | 8192 x 256 (2.1 M cells) | 1,000,000 x 10 (10 M cells) |54+|---|---|---|---|55+| Build through `Load`, then full recalc | 48 ms | 0.47 s | 2.2 s (396 MB peak resident) |56+| Live heap | 4.4 MB | 43 MB (20.5 B per number cell) | 205 MB |57+| Save `.012` (JSON, one line per cell) | 77 ms, 4.6 MB | 0.83 s, 47 MB | 3.7 s, 236 MB |58+| Open `.012` | 108 ms (118 MB allocated) | 1.5 s (1.3 GB allocated) | 8.1 to 8.3 s (5.7 GB allocated, 2.7 GB peak resident) |59+| Export CSV / TSV / XLSX / SQLite | 33 / 31 / 45 / 38 ms | 0.51 / 0.52 / 0.65 / 0.68 s | |60+| Edit a number, undo and redo it | 0.7 us, 1.2 us | | 0.7 us, 1.2 us |61+| Frame at 200 x 60, whole | 0.88 ms | 0.87 ms | |6062 6163 XLSX exports are written by 012 with archive/zip: 1.9 MB allocated for6264 8192 x 26 and 17 MB for 8192 x 256, whose time is mostly compression.6365 6466 Heap per non-blank cell after loading (`BenchmarkMemory`,65-`BenchmarkImport`): numbers 301 B, formulas 707 B (the parsed tree and66-reference lists), 200-character text 790 B, imported CSV 275 to 300 B.67-A `Cell` struct is 216 B; the rest is the map entry, the input string68-and a boxed number for the parsed literal.67+`BenchmarkImport`): numbers 20.5 B, opened from a file too; formulas 75068+B (the parsed tree and reference lists); imported CSV, SQLite and69+Parquet 21 to 63 B; text about 60 B plus its length, once per distinct string70+(a column of 90 texts repeated takes 27 B a cell). The store71+(`internal/sheet/store.go`, `slot.go`) keeps each column in blocks of72+1024 rows, a 16-byte slot per cell with contents or formatting: a plain73+cell (a number, boolean or text as typed, with a format and style) holds74+its value there, its text in a table of strings kept once each, its75+format and style in a table of looks, and its input only when that76+isn't the value's own text ("1.50" is kept as 1.5 printed with two77+decimals). Formulas, notes, and the results pivots and spills write are78+whole `Cell`s (216 B) in a side table. Reading a cell's value is an79+index into its column's block: 2.8 ns (`BenchmarkRead`), 8.6 ns a cell80+read by a SUM in a full recalculation.6981 7082 ## The grid7183 72-Nothing costs what the grid costs. Beside the cell map, occupancy84+Nothing costs what the grid costs. The cells' occupancy7385 indexes (bitmaps of 1024 rows per column) say which cells are stored and7486 which have contents; ranges are read through them, so a whole column of7587 ten cells is ten visits. Formats of whole columns, rows and the sheet@@ -101,7 +113,7 @@ What each part of 012 visits when given a range or selection:
101113 | XLSX column widths and styles | the data's width (at least 256), a style on A and XFD as the sheet's |102114 103115 A sheet of 10,000 numbers spread down a million rows under 3000104-whole-column formulas (sparse-1M) edits in 2.5 ms, draws a 200 x 60116+whole-column formulas (sparse-1M) edits in 1.6 ms, draws a 200 x 60105117 frame in 0.68 ms, and answers arrow keys at XFD1048576 in 0.65 ms.106118 Typing an entry under 1000 SUMs over a column, through to its frame at107119 200 x 60, takes 0.91 ms.@@ -119,19 +131,22 @@ read once cost nothing extra.
119131 120132 | Topology | Edit | Full recalc | Undo + redo |121133 |---|---|---|---|122-| Independent cell in 8192 x 26 numbers | 0.66 us | 32 to 38 ms | 0.8 us |123-| Chain: A(n) = A(n-1)+1 down 8192 rows | 1.85 ms | 1.1 to 1.4 ms | 3.4 ms |124-| Fan-out: one cell read by 8192 formulas | 1.4 ms | 0.9 ms | 2.7 ms |125-| Volatile: 8192 TODAY() and RAND() | 1.3 ms | 0.8 ms | 2.5 ms |126-| 1000 named ranges, one formula each | 48 us | 2.8 ms | 99 us |127-| Fan-in: 1000 x SUM(A1:A8192), or SUM(A:A) | 0.75 ms | 1 ms | 1.4 ms |128-| Running totals: 8192 x SUM($A$1:An) | 2.85 ms | 2.5 ms | 5.4 ms |129-| Sparse: 10,000 numbers spread down a million rows, 1000 each of SUM(A:A), running totals and VLOOKUP over A:B | 2.45 ms | 3.6 ms | 5.1 ms |130-| Criteria: 60 SUMIF, COUNTIFS and AVERAGEIF over whole columns of 8192 rows | 37 ms | 59 ms | |131-| Lookup: 300 VLOOKUP, MATCH and XLOOKUP into 8192 keys | 25 ms | 26.5 ms | |132-133-Every volatile formula is recomputed on every change, as in Sheets, so134-8192 volatile cells add about 1.2 ms to every edit anywhere. Criteria134+| Independent cell in 8192 x 26 numbers | 0.7 us | nothing to compute | 1.2 us |135+| Chain: A(n) = A(n-1)+1 down 8192 rows | 1.9 ms | 1 ms | 3.8 ms |136+| Fan-out: one cell read by 8192 formulas | 1.3 ms | 0.8 ms | 2.6 ms |137+| Volatile: 8192 TODAY() and RAND() | 1.3 ms | 0.8 ms | 2.6 ms |138+| 1000 named ranges, one formula each | 55 us | 1.2 ms | 110 us |139+| Fan-in: 1000 x SUM(A1:A8192), or SUM(A:A) | 0.53 ms | 0.48 ms | 1.1 to 1.2 ms |140+| Running totals: 8192 x SUM($A$1:An) | 2.7 ms | 1.8 ms | 5.8 ms |141+| Sparse: 10,000 numbers spread down a million rows, 1000 each of SUM(A:A), running totals and VLOOKUP over A:B | 1.6 ms | 1.5 ms | 3.2 ms |142+| Criteria: 60 SUMIF, COUNTIFS and AVERAGEIF over whole columns of 8192 rows | 29 ms | 44 ms | 59 ms |143+| Lookup: 300 VLOOKUP, MATCH and XLOOKUP into 8192 keys | 12 ms | 12 ms | 27 ms |144+| 10 M numbers (1,000,000 x 10) | 0.7 us | nothing to compute | 1.2 us |145+146+A full recalculation computes only formulas and what pivots and spills147+write: numbers and text hold their values from the moment they are148+stored. Every volatile formula is recomputed on every change, as in149+Sheets, so 8192 volatile cells add about 1.2 ms to every edit anywhere. Criteria135150 and lookups cost what their ranges hold, not what the grid holds: the136151 criteria collect positions from the occupancy index, and lookups walk137152 the stored entries with a cursor.@@ -169,7 +184,7 @@ terminal (`BenchmarkFrame`).
169184 | Sheet | 80 x 24 | 200 x 60 | 400 x 120 |170185 |---|---|---|---|171186 | Empty | 0.19 ms | 0.89 ms | 3.2 ms |172-| 8192 x 256 numbers | 0.20 ms | 1.05 ms | 4.1 ms |187+| 8192 x 256 numbers | 0.18 ms | 0.87 ms | 3.4 ms |173188 | 8192 rows of 500-character unicode text | 0.34 ms | 1.37 ms | 4.4 ms |174189 | 1000 named ranges | 0.30 ms | 1.06 ms | 3.5 ms |175190 | 20 charts (text) | 0.82 ms | 2.9 ms | 7.3 ms |@@ -189,7 +204,7 @@ every cell on screen a shade), at 200 x 60 (`BenchmarkFrame`,
189204 | | Without rules | With the scale |190205 |---|---|---|191206 | A frame, or an arrow key through to it | 0.89 ms | 1.28 ms |192-| Typing a number and Enter, through to the frame | 0.91 ms | 2.9 ms (the scale's percentile over 213 k numbers) |207+| Typing a number and Enter, through to the frame | 0.89 ms | 2.3 ms (the scale's percentile over 213 k numbers) |193208 194209 Shades and rule colors keep their escape codes, so a plain cell on one195210 costs a string concatenation, not a style render. A custom formula is@@ -202,8 +217,8 @@ slices, 0.44 ms, is the worst). An image for kitty graphics is redrawn
202217 when its data, size or theme changes: 25 to 260 us at 24 x 10 cells and203218 1 to 1.5 ms at 120 x 40, a pie 8.6 ms (`internal/chart`,204219 `BenchmarkDraw`, `BenchmarkImage`). Key presses add little: an arrow key205-through to its frame is 0.21 ms at 80 x 24 and 1.06 ms at 200 x 60;206-Page Down 1.26 ms.220+through to its frame is 0.19 ms at 80 x 24 and 0.9 ms at 200 x 60;221+Page Down 1.1 ms.207222 208223 ## Selection statistics209224 @@ -223,9 +238,9 @@ a selection over the whole sheet, through to its frame
223238 224239 | Key, 8192 x 256 numbers | 80 x 24 | 200 x 60 |225240 |---|---|---|226-| Shift+Up and Shift+Down with the data selected (`extend-data`) | 0.41 to 0.53 ms | 2.1 ms |227-| Ctrl+Shift+Down and Ctrl+Shift+Up over the first row (`extend-edge`) | 0.43 ms | 1.7 ms |228-| Ctrl+A, the selection standing (`select-data`) | 0.28 ms | 1.8 ms |241+| Shift+Up and Shift+Down with the data selected (`extend-data`) | 0.33 ms | 1.85 ms |242+| Ctrl+Shift+Down and Ctrl+Shift+Up over the first row (`extend-edge`) | 0.29 ms | 1.56 ms |243+| Ctrl+A, the selection standing (`select-data`) | 0.29 ms | 1.78 ms |229244 230245 The index costs memory only on sheets that had a large selection, in231246 proportion to their data, and a nil check per stored or recalculated@@ -238,13 +253,13 @@ Real, openly licensed datasets fetched by `scripts/stress-data.sh`
238253 239254 | File | Shape | Import | Throughput | Heap per cell |240255 |---|---|---|---|---|241-| OWID energy (CSV) | 23,377 x 130, 1.04 M cells | 0.84 to 0.95 s | 1.1 to 1.2 M cells/s | 277 B |242-| Airport codes (CSV) | 86,134 x 13, 855 k cells | 0.78 to 0.80 s | 1.1 M cells/s | 274 B |243-| NOAA daily CO2 (CSV) | 18,304 x 2 | 6 ms | 2.7 M cells/s | 294 B |244-| Country codes (CSV, Arabic, CJK) | 249 x 56 | 5.6 ms | 2.2 M cells/s | 272 B |245-| Apache POI formula tests (XLSX) | 788 rows, 1189 formulas, 3828 cells | 9.5 to 9.7 ms | 395 k cells/s | 425 B |246-| Chinook PlaylistTrack (SQLite) | 8715 x 2 | 5.5 ms | 3.0 M cells/s | 293 B |247-| Parquet alltypes_tiny_pages | 7300 x 13 | 40 ms | 2.4 M cells/s | 276 B |256+| OWID energy (CSV) | 23,377 x 130, 1.04 M cells | 0.25 s | 4.2 M cells/s | 23 B |257+| Airport codes (CSV) | 86,134 x 13, 855 k cells | 0.39 s | 2.2 M cells/s | 55 B |258+| NOAA daily CO2 (CSV) | 18,304 x 2 | 12 ms | 3.1 M cells/s | 63 B |259+| Country codes (CSV, Arabic, CJK) | 249 x 56 | 5.4 ms | 2.3 M cells/s | 59 B |260+| Apache POI formula tests (XLSX) | 788 rows, 1189 formulas, 3828 cells | 8.5 to 10 ms | 380 to 450 k cells/s | 239 B |261+| Chinook PlaylistTrack (SQLite) | 8715 x 2 | 6.1 ms | 2.9 M cells/s | 21 B |262+| Parquet alltypes_tiny_pages | 7300 x 13 | 27 ms | 3.5 M cells/s | 44 B |248263 249264 Past `max-cells` (or the grid), imports keep whole rows up to it and250265 say how many rows they left out. Parquet files and SQLite tables stop@@ -263,10 +278,10 @@ importing, per cell kept, and includes garbage not yet collected:
263278 264279 | Workbook | Time | Allocated | Allocations | Heap per cell after | Peak heap per cell |265280 |---|---|---|---|---|---|266-| 8192 x 26 numbers (213 k cells) | 189 to 199 ms | 170 MB | 3.3 M | 290 B | 589 B |267-| 8192 x 26 table, text and numbers | 296 to 312 ms | 234 MB | 7.4 M | 297 B | 722 B |268-| 8192 formulas (`=A1+1` down a column) | 13.8 ms | 17 MB | 321 k | 746 B | 2188 B |269-| Apache POI formula tests | 9.5 to 9.7 ms | 9.7 MB | 179 k | 425 B | 1288 B |281+| 8192 x 26 numbers (213 k cells) | 147 to 154 ms | 171 MB | 3.3 M | 20.5 B | 114 B |282+| 8192 x 26 table, text and numbers | 262 to 275 ms | 246 MB | 7.7 M | 21 B | 119 B |283+| 8192 formulas (`=A1+1` down a column) | 14.5 ms | 18.6 MB | 321 k | 755 B | 2061 B |284+| Apache POI formula tests | 8.5 to 10 ms | 9 MB | 162 k | 239 B | 965 B |270285 271286 The reader refuses files past `xlsxLimits` (in `internal/fileio/xlsxpkg.go`),272287 with a message naming the limit: 10,000 files in the zip; 1 GB@@ -282,9 +297,12 @@ entities, so entity bombs fail as unknown entities.
282297 283298 ## Undo284299 285-Each step keeps a copy of every cell it changed. Undoing costs about300+Each step keeps a copy of every cell it changed, as a whole `Cell`301+(about 300 B, where the sheet keeps a number in 20). Undoing costs about286302 what the change cost, since it recalculates the same cells. Clearing287-213 k cells and undoing it takes 158 to 164 ms.303+213 k cells and undoing it takes 144 ms; clearing all ten million cells304+of a full `max-cells` sheet would keep about 3 GB of before-images, the305+one step the history always keeps however large.288306 289307 The history is bounded by steps (`MaxUndo = 100`) and by an estimate of290308 the memory its before-images hold (`MaxUndoBytes = 256 MB`); past either,@@ -298,9 +316,9 @@ measured heap within about 10% (`BenchmarkHistoryFull`,
298316 299317 | Benchmark | Result |300318 |---|---|301-| 100 edits rewriting a whole column (819 k before-images) | 250 MB held (239 MB estimated), all 100 kept |302-| 12 edits rewriting all of 8192 x 26 (2.6 M before-images) | 230 MB held (248 MB estimated), the last 4 kept |303-| Undo + redo of one cell (dense, names) | 1.0 us, 106 us |319+| 100 edits rewriting a whole column (819 k before-images) | 281 MB held (252 MB estimated), all 100 kept |320+| 12 edits rewriting all of 8192 x 26 (2.6 M before-images) | 199 MB held (196 MB estimated), the last 3 kept |321+| Undo + redo of one cell (dense, names) | 1.2 us, 110 us |304322 305323 ## JEV306324 @@ -372,8 +390,8 @@ read-heavy recalculation.
372390 373391 | Benchmark (`internal/sheet`, `go test -bench SheetEdit`) | Edit |374392 |---|---|375-| 5000 formulas on one sheet reading `A(n)` and `SUM(A1:A10)` | 2.7 ms |376-| The same 5000 formulas reading `Data!A(n)` and `SUM(Data!A1:A10)` | 3.5 ms |393+| 5000 formulas on one sheet reading `A(n)` and `SUM(A1:A10)` | 1.65 ms |394+| The same 5000 formulas reading `Data!A(n)` and `SUM(Data!A1:A10)` | 2 ms |377395 378396 ## Serving over SSH379397 @@ -388,10 +406,10 @@ for 120 frames a second, as the local app does (`-benchtime 200x`):
388406 389407 | Sessions, sheet | Heap per session | Key to frame p50 / p95 / max | CPU per frame |390408 |---|---|---|---|391-| 10, new sheet | 1.30 MiB | 5.1 / 8.3 / 9.6 ms | 0.67 ms |392-| 10, 1000 x 26 numbers | 8.5 MiB | 4.3 / 8.3 / 8.5 ms | 0.71 ms |409+| 10, new sheet | 1.35 MiB | 5.1 / 8.3 / 9.8 ms | 0.83 ms |410+| 10, 1000 x 26 numbers | 1.9 MiB | 3.9 / 8.3 / 10.8 ms | 0.79 ms |393411 | 50, new sheet | 1.30 MiB | 5.0 / 8.7 / 17.8 ms | 0.53 ms |394-| 50, 1000 x 26 numbers | 8.5 MiB | 5.1 / 8.9 / 12.8 ms | 0.66 ms |412+| 50, 1000 x 26 numbers | 1.9 MiB | 5.1 / 8.9 / 12.8 ms | 0.66 ms |395413 396414 Latency doesn't move from 10 to 50 sessions: it's Bubble Tea's frame397415 pacing (at most one frame every 8.3 ms, so a key waits half a frame on@@ -400,7 +418,7 @@ average), not load. A heap profile with 50 sessions open
400418 Bubble Tea's render buffers (about 1 MiB at 120 x 40, growing with the401419 window), 130 KiB in its input key table and under 50 KiB in both ends402420 of the SSH connection, so the figure is close to the server's own. On403-top come about 300 B per cell of the sheets a session has open;421+top come 20 to 60 B per cell of data of the sheets a session has open;404422 sessions opening the same file each hold a copy. CPU per frame covers405423 both ends of the connection (the encryption twice, the client reading406424 the frame); at 0.7 ms, 50 sessions typing continuously at 120 frames a@@ -420,10 +438,10 @@ times the edit, the recalculation and the pivot together:
420438 421439 | Pivot | Result rows | Edit | Allocations |422440 |---|---|---|---|423-| Rows: 8 categories; SUM and COUNTA | 10 | 0.77 to 0.95 ms | 196 |424-| Rows: 8 x 8 categories with subtotals; columns: 8; SUM and AVERAGE | 75 | 2.7 to 2.9 ms | 2.5 k |425-| Rows: 1000 items sorted by their sum; SUM and COUNTUNIQUE | 1001 | 3.5 to 4.2 ms | 14 k |426-| Frequency table of the 1000 items | 1001 | 2.5 to 2.8 ms | 24 k |441+| Rows: 8 categories; SUM and COUNTA | 10 | 0.79 ms | 196 |442+| Rows: 8 x 8 categories with subtotals; columns: 8; SUM and AVERAGE | 75 | 2.2 ms | 2.5 k |443+| Rows: 1000 items sorted by their sum; SUM and COUNTUNIQUE | 1001 | 2.5 ms | 14 k |444+| Frequency table of the 1000 items | 1001 | 1.8 ms | 24 k |427445 428446 ## Arrays and spills429447 @@ -448,10 +466,10 @@ FILTERs, 63 of whose arrays change:
448466 449467 | Measure | Time | Allocated |450468 |---|---|---|451-| Edit a number | 55 ms | 161 MB, 8.3 k allocations |452-| Undo and redo it | 110 ms | |453-| Full recalculation | 204 ms | |454-| Build (load and recalculate) | 310 ms | |469+| Edit a number | 52 ms | 161 MB, 7.9 k allocations |470+| Undo and redo it | 103 ms | |471+| Full recalculation | 217 ms | |472+| Build (load and recalculate) | 298 ms | |455473 456474 Most of an edit is the FILTERs' own work: comparing 8192 categories each457475 (`B1:B8192="alpha"`, about 20 ns a value) and picking the rows that@@ -481,32 +499,25 @@ computation.
481499 482500 ## What would raise the bounds483501 484-In order of value for effort. The first two are what raising485-`max-cells` toward Google Sheets' 10 M cells would take; without them a486-10 M-cell sheet would need about 3 GB of heap and take 10 s to open.487-488-1. **Compact cell storage** (L, 2 to 3 weeks). Cells live in a489- `map[Addr]*Cell` behind `cellStore` (`internal/sheet/store.go`), 300 B490- each: a 216 B struct carrying formula-only491- fields (expression, reference lists, inferred format) on every cell,492- plus the map entry, the input string and a boxed literal. Storing493- columns in row blocks (say 1024 rows) of compact values, with494- formulas, formats and styles in side tables keyed by address, and495- the input string kept only when it differs from the canonical text496- of the value, would bring numbers to about 20 to 40 B per cell and497- turn every cell read in recalc from two hash lookups (20 ns) into an498- index (2 to 3 ns). Cells are already per sheet in a workbook, so the499- storage can change sheet by sheet, and nothing outside `cellStore`500- touches the map, so the change stays inside it.501-2. **Streaming, compact file format** (M, 1 week). The `.012` file is502- JSON decoded whole (1.3 GB allocated to open 47 MB). A streaming503- decoder over the same format would roughly halve open time and cut504- allocation tenfold; a columnar or gzip-compressed variant would cut505- the size about fourfold. The format is `internal/sheet/file.go`, apart506- from the cell store.507-3. **Smaller undo steps** (S, 1 day). The history is capped by the508- memory its before-images hold; storing formatting-only changes as509- diffs rather than whole cells would let it keep more of them.502+In order of value for effort. `max-cells` stands at ten million, as503+Google Sheets' limit does; the sheet itself would hold several times504+that in a few GB, and these are what stand in the way.505+506+1. **Streaming, compact file format** (M, 1 week). The `.012` file is507+ JSON decoded whole: ten million cells take 8 s and 2.7 GB at the508+ peak to open, 5.7 GB allocated, for a sheet that then holds 205 MB.509+ A streaming decoder over the same format would roughly halve open510+ time and cut allocation tenfold; a columnar or gzip-compressed511+ variant would cut the size about fourfold. The format is512+ `internal/sheet/file.go`, apart from the cell store.513+2. **Smaller undo steps** (S to M). A step keeps a whole `Cell` for each514+ cell it changed, about 300 B where the store keeps a number in 20,515+ and formatting-only changes as whole cells too. Keeping plain cells'516+ before-images as slots, and formatting changes as diffs, would let a517+ step over a full sheet cost what the sheet does.518+3. **Compact spilled and pivot cells.** What arrays and pivots write is519+ kept as whole `Cell`s (about 300 B each, 516 k of them in the arrays520+ stress shape); their values and inferred formats would fit in slots.510521 511522 ## Measuring512523 @@ -531,5 +542,6 @@ overhead), `internal/fileio/stress_test.go` (imports and exports),
531542 `internal/chart/stress_test.go` (charts as text and images),532543 `internal/serve/stress_test.go` (concurrent SSH sessions) and533544 `e2e/stress_test.go`; the synthetic sheets are built by534-`internal/stress`. Results accumulate in `.deps/stress/results` and, with545+`internal/stress`, up to `dense-1Mx10`, ten million numbers, the546+default `max-cells`. Results accumulate in `.deps/stress/results` and, with535547 the observability stack up, in ClickHouse.@@ -190,8 +190,8 @@ check protects the local app from other programs writing the file.
190190 - An idle session is closed without saving over its file; unsaved191191 changes go to a recovery file instead.192192 - A dropped connection loses unsaved changes.193-- Each session holds about 1.3 MiB of heap on a new sheet, plus about194- 300 B per cell of the sheets it opens; two sessions opening one file195- hold two copies. With 50 sessions typing at once on loopback, frames193+- Each session holds about 1.3 MiB of heap on a new sheet, plus 20 to194+ 60 B per cell of data of the sheets it opens (a formula takes about195+ 750); two sessions opening one file hold two copies. With 50 sessions typing at once on loopback, frames196196 still arrive within one frame interval (p95 16.6 ms), the same as with197197 10: see [Bounds of support](../contributing/limits.md#serving-over-ssh).