Bounds of support measures both (open, save, clear, history) and drops them from what would raise the bounds; the format page says how the file is read and written and why there is one format; architecture, editing and testing describe the steps, the question before a change too large to undo, and FuzzRead. The roadmap moves both items to Shipped.
- commit
- 7e6ca6ecb3677812cdd0a37663ce125b49d73fc1
- parent
- 652244f4165f28435486fba1ad48e73f16bc88f5
- author
- Dami <dami.etoile@gmail.com>
- kind
- docs risk score 0.1 of 3, Jev's estimate
| M | ROADMAP.md | 4 | +- |
| M | docs/ | 23 | +++++-- |
| M | docs/ | 89 | ++++++++++++++++-------------- |
| M | docs/ | 6 | +- |
| M | docs/ | 23 | ++++++++ |
| M | docs/ | 3 | + |
@@ -37,8 +37,6 @@ rises in measured steps; see [Bounds of support](docs/contributing/limits.md#wha
3737 3838 | Item | Result | Size |3939 |---|---|---|40-| A streaming or binary `.012` format next to the readable JSON one | Open and save scale with the data | M |41-| Smaller undo steps: plain cells' before-images as slots, formatting changes as diffs | More history in the same memory; clearing a full sheet costs what the sheet does | S to M |4240 | Compact spilled and pivot cells: derived cells in the compact slot form instead of whole `Cell`s | Large spills and pivots cost what plain cells cost ([Bounds of support](docs/contributing/limits.md#what-would-raise-the-bounds)) | M |4341 | Linked, paged read-only ranges over Parquet and SQLite that feed pivots and formulas by streaming | Sources too big for any grid | L |4442 @@ -80,6 +78,7 @@ by a gateway dialing the iroh ticket.
8078 - A locale per file, as Sheets' File > Settings > Locale: decimal commas, date order, currency and `;` in formulas, typed and shown while files store en-US's form: [Locale](docs/sheets/locale.md)8179 - An Excel-sized grid with a `max-cells` budget; operations cost the data, not the grid: [Bounds of support](docs/contributing/limits.md#sheet-size)8280 - Compact column storage and a ten-million-cell `max-cells` budget: [Bounds of support](docs/contributing/limits.md#sheet-size)81+- Undo steps in the compact form: clearing a full ten-million-cell sheet holds about what the sheet does, and a step past 1 GB asks first: [Bounds of support](docs/contributing/limits.md#undo)8382 8483 **Finding and using features**8584 @@ -110,6 +109,7 @@ by a gateway dialing the iroh ticket.
110109 **Files**111110 112111 - Import CSV, TSV, XLSX, SQLite, Parquet and Lotus `.wk1`; export CSV, TSV, XLSX and SQLite; import locations; save-as and overwrite checks: [Files](docs/files/README.md)112+- `.012` files read and written as a stream, cells straight into and out of the store: [The .012 format](docs/files/format.md#reading-and-writing)113113 114114 **Upkeep**115115 @@ -158,9 +158,16 @@ style.
158158 - **Undo.** Every mutation goes through a small set of paths that snapshot159159 the cells, widths, names, charts, view state and sheet list they change,160160 on any sheet, so a step can be reversed exactly; multi-cell operations161- are one step. A deleted sheet keeps its cells, so undo brings it back.162- The history keeps at most 100 steps and about 256 MB of before-images,163- counted as they're recorded, dropping the oldest steps first.161+ are one step. A step keeps each sheet's before-images as an image162+ (`historyimage.go`) in the store's own form: plain cells as slots in163+ columns of blocks, their text and formatting in the image's tables,164+ and whole `Cell`s only for rich ones, so clearing a whole sheet holds165+ about what the sheet does. A deleted or replaced sheet keeps its166+ cells, so undo brings it back. The history keeps at most 100 steps and167+ about 256 MB of before-images, counted as they're recorded, dropping168+ the oldest steps first; a change whose step would pass 1 GB169+ (`UndoCost`, `MaxStepBytes`) asks first, and runs `WithoutUndo` if told170+ to go on.164171 - **Pivot tables.** A sheet may hold a pivot (`pivot.go`): a definition165172 naming its source by sheet name, as formulas do, and a region of166173 derived cells from A1 that the engine owns. When a recalculation marks@@ -300,13 +307,15 @@ The package boundaries leave the bounds in [Bounds of support](limits.md) room t
300307 move without touching callers:301308 302309 - **Storage.** `cellStore`'s methods are the whole contract, so what is303- kept whole or in a slot can change (spilled and pivot cells in slots,304- plain before-images in the history) without touching callers.310+ kept whole or in a slot can change (spilled and pivot cells in slots)311+ without touching callers.305312 - **Range reads.** Shared range results, prefix sums for running totals306313 and column-block scans go behind the engine's `Scan` and `RangeAgg`,307314 the places functions read ranges.308-- **Files.** A streaming or columnar `.012` format replaces `file.go`'s309- whole-document JSON; the format is already separate from the store.315+- **Files.** The `.012` reader and writer stream (`fileread.go`,316+ `filescan.go`) and meet the store only through its methods, so another317+ encoding would sit beside them; [The .012 format](../files/format.md#reading-and-writing)318+ says why there is one.310319 - **Depth limits.** `internal/formula`'s parser caps nesting at311320 `formula.MaxDepth` (1024 levels), so a pathological formula fails to312321 parse before anything evaluates it; evaluation puts off cells past@@ -25,13 +25,13 @@ it lags, and past a second it stalls.
2525 2626 | Dimension | Comfortable | Degraded | Unsupported | Dominant cost |2727 |---|---|---|---|---|28-| Sheet size | A grid of 1,048,576 x 16,384 (A..XFD), with up to `max-cells` cells (ten million by default): navigation, drawing and every command cost what the cells cost, not the grid; ten million numbers take 205 MB and build in 2.2 s | Opening a `.012` file of ten million cells: 8 s and 2.7 GB at its peak; saving it: 3.7 s | More than `max-cells` cells: imports keep whole rows up to it and say what they dropped; larger pastes and fills are refused | JSON file format, decoded whole; heap per cell (about 20 B for numbers, 20 to 60 B for real data, 750 B for formulas) |28+| Sheet size | A grid of 1,048,576 x 16,384 (A..XFD), with up to `max-cells` cells (ten million by default): navigation, drawing and every command cost what the cells cost, not the grid; ten million numbers take 205 MB, build in 1.8 s and save in 0.54 s | Opening a `.012` file of ten million cells: 1.2 s, 311 MB of heap at its peak | More than `max-cells` cells: imports keep whole rows up to it and say what they dropped; larger pastes and fills are refused | Parsing each cell's entry from the file's JSON; heap per cell (about 20 B for numbers, 20 to 60 B for real data, 750 B for formulas) |2929 | Incremental recalc | A change that makes formulas read under about 500,000 cells in total (fan-out, chains and volatiles of 8192 cells: 1.3 to 1.9 ms); 1000 SUMs over a full column: 0.53 ms; 8192 running totals: 2.7 ms | 60 criteria functions (SUMIF, COUNTIFS, AVERAGEIF) over whole columns of 8192 rows: 29 ms an edit | | About 3 ns per cell read: an index into the column's block |3030 | Full recalc | Any sheet: numbers and text hold their values and cost nothing; 1000 full-column SUMs 0.5 ms; 8192 running totals 1.8 ms | 60 whole-column criteria functions over 8192 rows: 44 ms | | Same as above |3131 | Rendering | Any sheet at up to 200 x 60: 1 ms a frame; 400 x 120: 4.5 ms; a color scale on every cell shown adds 0.4 ms at 200 x 60 | 20 charts at 400 x 120: 7 ms | | View building styled strings, then Bubble Tea parsing and diffing them |3232 | Selection statistics | Any selection: extending one over all 2.1 M cells of 8192 x 256 costs 0.3 ms a key | | | Per column with data, 1024 or 64 rows at a time from an index on the blocks of filled cells, plus the rows at the selection's ends |3333 | Imports | CSV, SQLite, Parquet: 2 to 4 M cells/s (a million cells in 0.25 to 0.4 s); XLSX numbers or text: 0.8 to 1.4 M cells/s | XLSX with formulas: 0.4 to 0.6 M cells/s | Data past `max-cells` or the grid (dropped, with a note); XLSX files past the reader's limits (refused) | Building cells one at a time; XML decoding; XLSX formula translation |34-| Undo | One step of any size: undo costs what the edit cost | History capped at 100 steps and 256 MB of before-images: 100 whole-column steps hold 280 MB | | Whole-cell before-images, about 300 B per cell per step |34+| Undo | One step of any size: undo costs what the edit cost; clearing all ten million cells of a full sheet holds 193 MB | History capped at 100 steps and 256 MB of before-images: 100 whole-column steps hold 15 MB | A step past 1 GB (millions of formulas or long distinct texts at once): asks, and runs without undo if told to | Before-images in the store's form: a slot (20 B) per plain cell, a whole `Cell` per formula |3535 | JEV | 4000 JEV cells: 4 us of CPU per answer, 15 ms to answer them all | | | An answer recalculates the cells that asked it; answers within a frame recalculate together |3636 | Formula depth | 1000 nested parentheses or IFs: under 0.5 ms; chains of formulas through every cell of a sheet | | More than 1024 levels of nesting in one formula: a parse error | Recursive parser; evaluation puts off cells past 65,536 levels |3737 | Macros | Replaying 1000 recorded actions: 2.2 ms, one undo step; a script's call to the sheet: about 1.4 us | | Scripts past 10 M Starlark steps: stopped, with the line | One message per call to the sheet, served in batches on the UI goroutine |@@ -47,17 +47,19 @@ Hard limits: `sheet.MaxRows = 1,048,576`, `sheet.MaxCols = 16,384`
4747 ten million cells by default): imports keep whole rows up to it, pastes4848 and fills write at most that many cells at once. WK1 files keep their own4949 8192 x 256. The costs below grow with the cells; the grid itself costs50-nothing (see [The grid](#the-grid)). At ten million cells the file51-format is what degrades: a `.012` file is JSON decoded whole.50+nothing (see [The grid](#the-grid)). A `.012` file is read and written51+as a stream, cells straight into and out of the store ([The .01252+format](../files/format.md#reading-and-writing)), so opening costs about53+the memory the workbook then holds.5254 5355 | Measure | 8192 x 26 (213 k cells) | 8192 x 256 (2.1 M cells) | 1,000,000 x 10 (10 M cells) |5456 |---|---|---|---|55-| Build through `Load`, then full recalc | 48 ms | 0.47 s | 2.2 s (396 MB peak resident) |57+| Build through `Load`, then full recalc | 48 ms | 0.47 s | 1.8 s |5658 | Live heap | 4.4 MB | 43 MB (20.5 B per number cell) | 205 MB |57-| Save `.012` (JSON, one line per cell) | 77 ms, 4.6 MB | 0.83 s, 47 MB | 3.7 s, 236 MB |58-| Open `.012` | 108 ms (118 MB allocated) | 1.5 s (1.3 GB allocated) | 8.1 to 8.3 s (5.7 GB allocated, 2.7 GB peak resident) |59+| Save `.012` (JSON, one line per cell; heap used at the peak) | 12 ms, 4.6 MB (0.3 MB) | 0.14 s, 47 MB (0.1 MB) | 0.54 s, 236 MB (0.7 MB) |60+| Open `.012` (heap at the peak, the workbook included) | 24 ms (15 MB) | 0.24 s (60 MB) | 1.2 s (311 MB; 0.9 GB resident, the file's bytes and a sheet built beside it included) |5961 | Export CSV / TSV / XLSX / SQLite | 33 / 31 / 45 / 38 ms | 0.51 / 0.52 / 0.65 / 0.68 s | |60-| Edit a number, undo and redo it | 0.7 us, 1.2 us | | 0.7 us, 1.2 us |62+| Edit a number, undo and redo it | 0.6 us, 1.2 us | | 0.6 us, 1.2 us |6163 | Frame at 200 x 60, whole | 0.88 ms | 0.87 ms | |6264 6365 XLSX exports are written by 012 with archive/zip: 1.9 MB allocated for@@ -294,28 +296,41 @@ entities, so entity bombs fail as unknown entities.
294296 295297 ## Undo296298 297-Each step keeps a copy of every cell it changed, as a whole `Cell`298-(about 300 B, where the sheet keeps a number in 20). Undoing costs about299-what the change cost, since it recalculates the same cells. Clearing300-213 k cells and undoing it takes 144 ms; clearing all ten million cells301-of a full `max-cells` sheet would keep about 3 GB of before-images, the302-one step the history always keeps however large.299+Each step keeps the cells it changed as they were, in the form the300+sheet keeps them (`historyimage.go`): a plain cell as its 16-byte slot301+in columns of blocks, its text and formatting in the step's own tables,302+and a whole `Cell` only for a formula, a note or a derived cell. A step303+of one cell or a few keeps them in a list. Undoing costs about what the304+change cost, since it recalculates the same cells. Clearing 213 k cells305+and undoing it takes 86 ms; clearing all ten million cells of a full306+`max-cells` sheet of numbers holds 193 MB, and with its undo takes 6.2307+s. A deleted or replaced sheet is kept whole, so its step costs nothing308+more.303309 304310 The history is bounded by steps (`MaxUndo = 100`) and by an estimate of305311 the memory its before-images hold (`MaxUndoBytes = 256 MB`); past either,306312 the oldest steps are dropped first, and the newest step is kept however307-large it is. Each cell is counted as it is recorded (a fixed cost per308-cell plus its text, more for a formula, whose parsed tree stays alive),309-so the budget costs nothing to check; counting costs about 20 ns a310-cell, 3% of clearing a sheet and undoing it. The estimate follows the311-measured heap within about 10% (`BenchmarkHistoryFull`,312-`BenchmarkHistoryWide`):313+large it is. Each cell is counted as it is recorded (a slot, a block's314+share, its text once per step, more for a formula, whose parsed tree315+stays alive), so the budget costs nothing to check. The estimate follows316+the measured heap within about 10% (`BenchmarkHistoryFull`,317+`BenchmarkHistoryWide`, `BenchmarkClearMax`):313318 314319 | Benchmark | Result |315320 |---|---|316-| 100 edits rewriting a whole column (819 k before-images) | 281 MB held (252 MB estimated), all 100 kept |317-| 12 edits rewriting all of 8192 x 26 (2.6 M before-images) | 199 MB held (196 MB estimated), the last 3 kept |318-| Undo + redo of one cell (dense, names) | 1.2 us, 110 us |321+| 100 edits rewriting a whole column (819 k before-images) | 15.9 MB held (15.9 MB estimated), all 100 kept |322+| 12 edits rewriting all of 8192 x 26 (2.6 M before-images) | 49.3 MB held (49.4 MB estimated), all 12 kept |323+| Clearing all of 1,000,000 x 10 numbers | 193 MB held (193 MB estimated) |324+| Undo + redo of one cell (dense, names) | 1.2 us, 109 us |325+326+A change whose step would hold more than `MaxStepBytes` (1 GB), as327+`Sheet.UndoCost` estimates it before the change (a slot a cell, the328+formulas whole, the range's share of the strings), asks first on the329+context line: "This can't be undone: it would take N MB of undo330+history." Enter runs it `WithoutUndo`, recording nothing and forgetting331+the history, which could no longer be undone past it. Plain cells cost332+about 20 B each, so only millions of formulas or of long distinct texts333+changed at once come near it.319334 320335 ## JEV321336 @@ -496,26 +511,14 @@ computation.
496511 497512 ## What would raise the bounds498513 499-In order of value for effort; sizes and scheduling are in the500-[roadmap](../../ROADMAP.md#3-scale). `max-cells` stands at ten million,501-as Google Sheets' limit does; the sheet itself would hold several times502-that in a few GB, and these are what stand in the way.503-504-1. **Streaming, compact file format.** The `.012` file is JSON decoded505- whole: ten million cells take 8 s and 2.7 GB at the peak to open,506- 5.7 GB allocated, for a sheet that then holds 205 MB. A streaming507- decoder over the same format would roughly halve open time and cut508- allocation tenfold; a columnar or gzip-compressed variant would cut509- the size about fourfold. The format is `internal/sheet/file.go`,510- apart from the cell store.511-2. **Smaller undo steps.** A step keeps a whole `Cell` for each cell it512- changed, formatting-only changes included (see [Undo](#undo)).513- Keeping plain cells' before-images as slots, and formatting changes514- as diffs, would let a step over a full sheet cost what the sheet515- does.516-3. **Compact spilled and pivot cells.** What arrays and pivots write is517- kept as whole `Cell`s (516 k of them in the arrays stress shape);518- their values and inferred formats would fit in slots.514+Sizes and scheduling are in the [roadmap](../../ROADMAP.md#3-scale).515+`max-cells` stands at ten million, as Google Sheets' limit does; the516+sheet itself would hold several times that in a few GB, and opening,517+saving and undoing cost about what the cells do ([Sheet size](#sheet-size),518+[Undo](#undo)). What stands in the way is derived cells: what arrays519+and pivots write is kept as whole `Cell`s (516 k of them in the arrays520+stress shape), where their values and inferred formats would fit in521+slots.519522 520523 ## Measuring521524 @@ -9,7 +9,7 @@ sidebar_position: 5
99 make check # all of the below that must pass before a push: gofmt, vet, lint, test, oracle, e2e1010 make lint # go vet, staticcheck, cognitive complexity at most 25, Go files at most 500 lines, doclint and doccheck (see Docs below)1111 make test # engine, file formats and UI unit tests12-make fuzz # fuzz the formula parser and the CSV, .wk1 and XLSX readers12+make fuzz # fuzz the formula parser and the .012, CSV, .wk1 and XLSX readers1313 make oracle # compare formulas and number formats with excelize1414 make e2e # run the real binary in a terminal emulator (needs Zig and pkg-config)1515 make screens # rewrite the golden screens and build the review gallery@@ -19,7 +19,9 @@ make stress # benchmarks on synthetic and real data (see limits.md)
1919 ## Unit tests2020 2121 The engine is tested directly: parsing, every function, recalculation,22-undo, reference rewriting, file round trips. The UI is tested by sending22+undo, reference rewriting, file round trips. `FuzzRead` holds the23+streaming `.012` reader to encoding/json decoding the whole file: both24+refuse a file or both read it into workbooks that save the same bytes. The UI is tested by sending2325 Bubble Tea messages (keys, mouse, paste, window size) to the model and2426 reading what `View` renders, without a terminal.2527 @@ -34,6 +34,29 @@ file opens: formula results, [pivot tables](#pivot-tables)' results, and
3434 the arrays formulas [spill](../formulas/arrays.md), whose cells are kept3535 only for their formatting and notes.3636 37+## Reading and writing38+39+012 reads and writes the file as a stream (`internal/sheet/fileread.go`,40+`filescan.go`). Each cell goes into its sheet as its line is read, a41+number typed plainly straight into the 16-byte slot the sheet keeps it42+in, so opening a file takes about the memory of the workbook, however43+large the file; the other fields are small and decoded whole. Saving44+writes the cells in row-major order as it reads them from the sheet,45+straight into the file. Ten million numbers (236 MB of JSON) open in46+1.2 s and save in 0.5 s ([Bounds of support](../contributing/limits.md#sheet-size)).47+48+Any JSON with the same fields reads the same: the order of keys,49+spacing, and the case of field names don't matter. A cell given twice50+takes its last entry, though an invalid earlier one is refused; the51+`version` or the `sheets` list given twice around cells is refused.52+53+There is one format. A binary one that stored the slots themselves54+would open ten million numbers in about 0.3 s at best (storing ten55+million slots alone takes 0.17 s) where the JSON takes 1.2 s: a second56+saved on sheets near `max-cells`, and nothing a smaller sheet would57+notice, for the loss of what the JSON gives, diffs, merges, and a file58+anyone can read and fix.59+3760 ## Versions3861 3962 012 writes the lowest version that holds the workbook, so older builds@@ -54,6 +54,9 @@ typing, clearing, paste, fill, sort, insert and delete, formats, widths,
5454 names, charts, freeze, filters, pivot tables, and adding, deleting,5555 renaming, moving and duplicating sheets. A multi-cell change is one step,5656 and the context line says what was undone, e.g. `Undid: clear B3:B5`.57+A change whose undo would take more than 1 GB of memory (clearing58+millions of formulas at once) asks first: Enter makes it without undo,59+forgetting the steps before it, and Esc backs out.5760 5861 ## Rows and columns5962