A derived cell is a slot marked slotPivot or slotSpill: its value (an error's code in the string table), its look carrying a spill's inferred format, and its entry the value's text. The store counts the pivot's results and bounds where they are, so writeDerived finds them without the side table; spills write their values straight into slots. BenchmarkMemoryDerived, -count 6: pivot results 309.7 -> 20.6 B a cell spilled numbers 309.1 -> 20.5 B a cell (+32 B of the kept array) arrays-1000xFILTER8192: build 302 -> 181 ms, full recalc 215 -> 106 ms, edit 52.2 -> 52.5 ms (~), undo 104 -> 106 ms (+2%) BenchmarkPivot: rows-8 ~, rows-8x8-cols-8 +3.1%, 1000-unique ~, frequency ~
- commit
- 5fe016216a7cb163a80d67d4ce8c6d800d8c451d
- parent
- 9471af439d88bfe1f5f990ec1590e858a84452cb
- author
- Dami <dami.etoile@gmail.com>
- kind
- refactor risk score 2.3 of 3, Jev's estimate
| M | ROADMAP.md | 2 | +- |
| M | docs/ | 4 | +- |
| M | docs/ | 29 | +++--- |
| M | internal/ | 2 | +- |
| M | internal/ | 5 | +- |
| A | internal/ | 50 | ++++++++++ |
| A | internal/ | 131 | +++++++++++++++++++++++++++ |
| M | internal/ | 2 | +- |
| M | internal/ | 4 | +- |
| M | internal/ | 30 | ++++-- |
| M | internal/ | 115 | ++++++++++++++++++++----- |
| M | internal/ | 30 | ++++-- |
| M | internal/ | 28 | +++++- |
| A | internal/ | 143 | ++++++++++++++++++++++++++++++ |
@@ -36,7 +36,6 @@ rises in measured steps; see [Bounds of support](docs/contributing/limits.md#wha
3636 3737 | Item | Result | Size |3838 |---|---|---|39-| Compact spilled and pivot cells: derived cells in the compact slot form instead of whole `Cell`s | Large spills and pivots cost what plain cells cost ([Bounds of support](docs/contributing/limits.md#what-would-raise-the-bounds)) | M |4039 | Linked, paged read-only ranges over Parquet and SQLite that feed pivots and formulas by streaming | Sources too big for any grid | L |4140 4241 ### 4. Macros, keys and the terminal@@ -79,6 +78,7 @@ by a gateway dialing the iroh ticket.
7978 - An Excel-sized grid with a `max-cells` budget; operations cost the data, not the grid: [Bounds of support](docs/contributing/limits.md#sheet-size)8079 - Compact column storage and a ten-million-cell `max-cells` budget: [Bounds of support](docs/contributing/limits.md#sheet-size)8180 - Undo steps in the compact form: clearing a full ten-million-cell sheet holds about what the sheet does, and a step past 1 GB asks first: [Bounds of support](docs/contributing/limits.md#undo)81+- Spilled cells and pivot results in the compact form, about 20 B each: [Bounds of support](docs/contributing/limits.md#sheet-size)8282 8383 **Finding and using features**8484 @@ -118,7 +118,9 @@ style.
118118 its slot, with its text in a table of strings kept once each, its119119 formatting in a table of looks, and its input only when that isn't120120 the value's own text ("1.50" is kept as 1.5 printed with two121- decimals). Formulas, notes, and what pivots and spills write are whole121+ decimals). What pivots and spills write is a slot too, marked as122+ derived, holding the value and, in its look, a spill's inferred123+ format; its entry is the value's text. Formulas and notes are whole122124 `Cell`s in a side table, which recalculation updates in place. `get`123125 hands out a plain cell as a `Cell` made for the caller, a copy whose124126 changes reach nothing; `set` is the one way to change a cell. What@@ -38,7 +38,7 @@ it lags, and past a second it stalls.
3838 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |3939 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 8.7 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 120 fps pacing; per session, the terminal's cell buffers |4040 | Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 0.8 to 2.5 ms | | Results past the grid (the pivot shows #REF!) | Grouping the source rows |41-| Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 52 ms | Full recalculation of those 1000: 217 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |41+| Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 52 ms | Full recalculation of those 1000: 106 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |4242 4343 ## Sheet size4444 @@ -71,9 +71,12 @@ B (the parsed tree and reference lists); imported CSV, SQLite and
7171 Parquet 21 to 63 B; text about 60 B plus its length, once per distinct string7272 (a column of 90 texts repeated takes 27 B a cell). A plain cell (a7373 number, boolean or text as typed, with a format and style) is a 16-byte74-slot in its column's block of 1024 rows; formulas, notes, and the75-results pivots and spills write are whole `Cell`s, about 300 B each with76-their entry and input (the `Cell` itself is 216 B). How the store lays74+slot in its column's block of 1024 rows, and so is a derived cell, what75+a pivot or a spill writes: a pivot's results take 20.6 B a cell and76+spilled numbers 20.5 B, plus the 32 B a value of the array the anchor77+keeps to compare with the next one (`BenchmarkMemoryDerived`). Formulas78+and notes are whole `Cell`s, about 300 B each with their entry and79+input (the `Cell` itself is 216 B). How the store lays7780 them out is in [Architecture](architecture.md#the-engine). Reading a7881 cell's value is an index into its column's block: 2.8 ns7982 (`BenchmarkRead`), 8.6 ns a cell read by a SUM in a full recalculation.@@ -319,7 +322,7 @@ entities, so entity bombs fail as unknown entities.
319322 Each step keeps the cells it changed as they were, in the form the320323 sheet keeps them (`historyimage.go`): a plain cell as its 16-byte slot321324 in columns of blocks, its text and formatting in the step's own tables,322-and a whole `Cell` only for a formula, a note or a derived cell. A step325+and a whole `Cell` only for a formula or a note. A step323326 of one cell or a few keeps them in a list. Undoing costs about what the324327 change cost, since it recalculates the same cells. Clearing 213 k cells325328 and undoing it takes 86 ms; clearing all ten million cells of a full@@ -498,10 +501,10 @@ FILTERs, 63 of whose arrays change:
498501 499502 | Measure | Time | Allocated |500503 |---|---|---|501-| Edit a number | 52 ms | 161 MB, 7.9 k allocations |502-| Undo and redo it | 103 ms | |503-| Full recalculation | 217 ms | |504-| Build (load and recalculate) | 298 ms | |504+| Edit a number | 52 ms | 161 MB, 7.8 k allocations |505+| Undo and redo it | 106 ms | |506+| Full recalculation | 106 ms | |507+| Build (load and recalculate) | 181 ms | |505508 506509 Most of an edit is the FILTERs' own work: comparing 8192 categories each507510 (`B1:B8192="alpha"`, about 20 ns a value) and picking the rows that@@ -535,10 +538,10 @@ Sizes and scheduling are in the [roadmap](../../ROADMAP.md#3-scale).
535538 `max-cells` stands at ten million, as Google Sheets' limit does; the536539 sheet itself would hold several times that in a few GB, and opening,537540 saving and undoing cost about what the cells do ([Sheet size](#sheet-size),538-[Undo](#undo)). What stands in the way is derived cells: what arrays539-and pivots write is kept as whole `Cell`s (516 k of them in the arrays540-stress shape), where their values and inferred formats would fit in541-slots.541+[Undo](#undo)). Derived cells cost what plain cells cost, a slot each542+([Sheet size](#sheet-size)); what stays large is a formula, about 750 B543+with its parsed tree and reference lists, and the array a spilling544+formula keeps, 32 B a value.542545 543546 ## Measuring544547 @@ -143,7 +143,7 @@ func decodeBorders(fb *fileBorders) (Borders, error) {
143143 // its slot, without making a Cell.144144 func (st *cellStore) appendSaved(buf []byte, a Addr) ([]byte, bool, error) {145145 b, i := st.find(a)146- if sl := b.vals[i]; sl.kind != slotRich && sl.look == 0 {146+ if sl := b.vals[i]; sl.kind != slotRich && sl.kind&slotDerived == 0 && sl.look == 0 {147147 return st.appendInput(buf, sl), true, nil148148 }149149 c := st.get(a).saved()@@ -24,10 +24,7 @@ func (s *Sheet) DisplayFormat(a Addr) Format {
2424 return f2525 }2626 }27- if c := s.cells.richAt(a); c != nil { // only formulas infer formats28- return c.auto29- }30- return Format{}27+ return s.cells.autoAt(a) // formulas and spilled cells infer formats3128 }3229 3330 // SetFormat gives every cell in r the number format f, including blank@@ -0,0 +1,50 @@
1+//go:build stress2+3+package sheet_test4+5+import (6+ "runtime"7+ "testing"8+9+ "github.com/FelineStateMachine/012/internal/sheet"10+ "github.com/FelineStateMachine/012/internal/stress"11+)12+13+// BenchmarkMemoryDerived is the heap a derived cell holds: the cells a14+// formula spills (SEQUENCE over 8192 x 64, the array the anchor keeps15+// to compare with the next one reported apart) and a pivot's results16+// (a row per id of a 8191-row table, with a SUM and a COUNTA).17+func BenchmarkMemoryDerived(b *testing.B) {18+ b.Run("spilled-8192x64", func(b *testing.B) {19+ for b.Loop() {20+ before := heap()21+ s := sheet.New()22+ if err := s.Set(sheet.Addr{}, "=SEQUENCE(8192, 64)"); err != nil {23+ b.Fatal(err)24+ }25+ after := heap()26+ n := float64(s.Len())27+ const valueBytes = 32 // a Value in the array the anchor keeps28+ b.ReportMetric(float64(after-before)/n, "B/cell")29+ b.ReportMetric(float64(after-before)/n-valueBytes, "B/cell-stored")30+ runtime.KeepAlive(s)31+ }32+ })33+ b.Run("pivot-8191", func(b *testing.B) {34+ for b.Loop() {35+ src := stress.Table(stress.Rows-1, 8)36+ r := sheet.NewRect(sheet.Addr{}, sheet.Addr{Col: 7, Row: stress.Rows - 1})37+ p := sheet.NewPivot(src, r)38+ p.Rows = []sheet.PivotGroup{{Col: 0}}39+ p.Values = []sheet.PivotValue{{Col: 2, Summarize: sheet.SumBy}, {Col: 3, Summarize: sheet.CountABy}}40+ before := heap()41+ pv, err := src.Book().CreatePivot(src, r, "", p)42+ if err != nil {43+ b.Fatal(err)44+ }45+ after := heap()46+ b.ReportMetric(float64(after-before)/float64(pv.Len()), "B/cell")47+ runtime.KeepAlive(pv)48+ }49+ })50+}@@ -0,0 +1,131 @@
1+package sheet2+3+import (4+ "bytes"5+ "strings"6+ "testing"7+)8+9+// A derived cell of every value, format, inferred format and mark comes10+// back from the store as it went in, kept in a slot unless it has a11+// note, and the store counts the pivot's results it holds.12+func TestSlotDerived(t *testing.T) {13+ st := newCellStore()14+ a := Addr{Col: 2, Row: 700}15+ values := []Value{num(1.5), num(-0.25), boolean(true), boolean(false), {Kind: Text, Str: "north"},16+ {Kind: Text, Str: "=not a formula"}, {Kind: Text, Str: "12"}, ErrRef, ErrDiv0, {Kind: Text}}17+ for _, v := range values {18+ for _, pivot := range []bool{true, false} {19+ for _, note := range []string{"", "a note"} {20+ c := &Cell{Input: derivedInput(v), Value: v, Format: Preset(FmtCurrency), Note: note, derived: pivot, spilled: !pivot}21+ if !pivot {22+ c.auto = Preset(FmtPercent)23+ }24+ c.Style.Bold = pivot25+ checkDerived(t, &st, a, c)26+ }27+ }28+ }29+ st.delete(a)30+ if st.len() != 0 || st.pivots != 0 || len(st.strs.index) != 0 || len(st.rich) != len(st.richFree) {31+ t.Errorf("left: %d stored, %d pivot cells, %d strings, %d rich", st.len(), st.pivots, len(st.strs.index), len(st.rich)-len(st.richFree))32+ }33+}34+35+// checkDerived stores the derived cell c at a and reads it back.36+func checkDerived(t *testing.T, st *cellStore, a Addr, c *Cell) {37+ t.Helper()38+ want := c.clone()39+ st.set(a, c)40+ checkSame(t, st.get(a), want)41+ if rich := st.richAt(a) != nil; rich != (want.Note != "") {42+ t.Errorf("%+v: rich %v", want, rich)43+ }44+ if got := st.filledAt(a); got != !want.Blank() {45+ t.Errorf("%+v: filled %v", want, got)46+ }47+ if !st.holds(a, st.get(a)) {48+ t.Errorf("%+v: doesn't hold its own view", want)49+ }50+ if n := len(collect(st.pivotKeys())); n != st.pivots || (n == 1) != want.derived {51+ t.Errorf("%+v: %d pivot cells found, %d counted", want, n, st.pivots)52+ }53+}54+55+func collect(seq func(func(Addr) bool)) []Addr {56+ var out []Addr57+ seq(func(a Addr) bool { out = append(out, a); return true })58+ return out59+}60+61+// Spilled cells are slots, and read as they did as Cells: values of each62+// kind, the anchor's inferred format, values when copied, skipped by63+// Replace and left out of the file.64+func TestSpillCompact(t *testing.T) {65+ s := sheetOf(t, map[string]string{66+ "A1": "$5", "A2": "$6", "A3": "x",67+ "B1": "=ARRAYFORMULA(A1:A2*2)",68+ "C1": `=ARRAYFORMULA(IF(SEQUENCE(3)=2, "none", SEQUENCE(3)))`,69+ "D1": "=SEQUENCE(2)>1",70+ "E1": "=1/(SEQUENCE(2)-2)",71+ })72+ wantShown(t, s, map[string]string{"B2": "12", "C2": "none", "C3": "3", "D2": "TRUE", "E2": "#DIV/0!"})73+ for _, a := range []string{"B2", "C2", "C3", "D2", "E2"} {74+ if s.cells.richAt(at(a)) != nil {75+ t.Errorf("%s is a whole Cell", a)76+ }77+ if c := s.Cell(at(a)); !c.Spilled() || c.Input != derivedInput(c.Value) {78+ t.Errorf("%s = %+v", a, c)79+ }80+ }81+ if f := s.DisplayFormat(at("B2")); f.Kind != FmtCurrency {82+ t.Errorf("B2's format %v, want the anchor's currency", f)83+ }84+ found, err := s.Find("none", FindOptions{})85+ if err != nil || len(found) != 1 || found[0] != at("C2") {86+ t.Errorf("Find found %v %v, want the spilled C2", found, err)87+ }88+ if changed, err := s.Replace(at("C2"), "none", "x", FindOptions{}); changed || err != nil {89+ t.Errorf("Replace changed a spilled cell: %v %v", changed, err)90+ }91+ clip := s.Copy(rect("E1:E2"))92+ if _, err := s.Paste(clip, rect("G1:G1"), false); err != nil {93+ t.Fatal(err)94+ }95+ if c := s.Cell(at("G2")); c.Spilled() || c.Value.String() != "#DIV/0!" {96+ t.Errorf("pasted spilled error = %+v", c)97+ }98+ var buf bytes.Buffer99+ if err := s.Book().Write(&buf); err != nil {100+ t.Fatal(err)101+ }102+ if strings.Contains(buf.String(), `"B2"`) || strings.Contains(buf.String(), `"D2"`) {103+ t.Errorf("spilled cells saved:\n%s", buf.String())104+ }105+}106+107+// A pivot's results are slots, found again when they shrink.108+func TestPivotCompact(t *testing.T) {109+ src := sheetOf(t, map[string]string{"A1": "Region", "B1": "Sales", "A2": "north", "B2": "10", "A3": "south", "B3": "5", "A4": "east", "B4": "7"})110+ r := rect("A1:B4")111+ p := NewPivot(src, r)112+ p.Rows = []PivotGroup{{Col: 0}}113+ p.Values = []PivotValue{{Col: 1, Summarize: SumBy}}114+ pv, err := src.Book().CreatePivot(src, r, "", p)115+ if err != nil {116+ t.Fatal(err)117+ }118+ n := pv.cells.pivots119+ if n == 0 || len(pv.cells.rich) != len(pv.cells.richFree) {120+ t.Fatalf("%d pivot cells, %d rich", n, len(pv.cells.rich)-len(pv.cells.richFree))121+ }122+ if err := src.Set(at("A4"), "north"); err != nil {123+ t.Fatal(err)124+ }125+ if pv.cells.pivots >= n {126+ t.Errorf("%d pivot cells after a group went, had %d", pv.cells.pivots, n)127+ }128+ if got := len(collect(pv.cells.pivotKeys())); got != pv.cells.pivots {129+ t.Errorf("%d pivot cells found, %d counted", got, pv.cells.pivots)130+ }131+}@@ -385,7 +385,7 @@ func (s *Sheet) loadNum(a Addr, input []byte, f Format, st Style) bool {
385385 if err != nil {386386 return false387387 }388- lk, ok := s.cells.lookID(f, st)388+ lk, ok := s.cells.lookID(look{f: f, st: st})389389 if !ok {390390 return false391391 }@@ -6,7 +6,7 @@ import "slices"
66 // the form the sheet keeps them (store.go, slot.go): columns of blocks77 // of 16-byte slots, the text in a table of strings kept once each, the88 // formatting in a table of looks, and whole Cells only for rich cells9-// (formulas, notes, what pivots and spills write). A step that clears9+// (formulas, notes). A step that clears1010 // or rewrites a whole sheet so holds about what the sheet does, 20 B1111 // per number, rather than a Cell per cell. A cell that held nothing1212 // before the step is a bit in blank.@@ -178,7 +178,7 @@ func (img *image) look(src *cellStore, id uint16) (uint16, bool) {
178178 return img.looks[id] - 1, true179179 }180180 l := src.looks[id]181- lk, ok := img.store().lookID(l.f, l.st)181+ lk, ok := img.store().lookID(l)182182 if !ok {183183 return 0, false184184 }@@ -284,7 +284,7 @@ func (w *Workbook) refreshPivot(s *Sheet) []loc {
284284 s.pivot.blocked = extent(want)285285 s.pivot.err = err.Error()286286 want = []pivotCell{{v: ErrRef, bold: true}}287- if c := s.cells.get(Addr{}); c != nil && !c.derived && !c.Blank() {287+ if s.typedAt(Addr{}) {288288 want = nil // never write over what the user typed289289 }290290 }@@ -353,7 +353,7 @@ func (w *Workbook) computePivot(s *Sheet) ([]pivotCell, *pivotCalc, error) {
353353 return nil, c, fmt.Errorf("The pivot table doesn't fit on a sheet")354354 }355355 for _, pc := range l.cells {356- if old := s.cells.get(pc.a); old != nil && !old.derived && !old.Blank() {356+ if s.typedAt(pc.a) {357357 return l.cells, c, fmt.Errorf("The pivot table's results would overwrite data in %s", pc.a)358358 }359359 }@@ -387,16 +387,19 @@ func (s *Sheet) writeDerived(want []pivotCell) []loc {
387387 next[pc.a] = c388388 }389389 var changed []loc390- for a, c := range s.cells.richCells() {391- if c.derived && next[a] == nil {392- s.setDerived(a, nil)393- changed = append(changed, loc{s, a})390+ var gone []Addr391+ for a := range s.cells.pivotKeys() {392+ if next[a] == nil {393+ gone = append(gone, a)394394 }395395 }396+ for _, a := range gone {397+ s.setDerived(a, nil)398+ changed = append(changed, loc{s, a})399+ }396400 for a, c := range next {397- if old := s.cells.get(a); old != nil && old.derived && old.Input == c.Input && old.Value == c.Value &&398- old.Format == c.Format && old.Style == c.Style {399- continue401+ if v, l, kind := s.cells.derivedOf(a); kind == slotPivot && v == c.Value && l.f == c.Format && l.st == c.Style {402+ continue // its input is its value's400403 }401404 s.setDerived(a, c)402405 changed = append(changed, loc{s, a})@@ -404,6 +407,15 @@ func (s *Sheet) writeDerived(want []pivotCell) []loc {
404407 return changed405408 }406409 410+// typedAt reports whether the cell at a has contents that aren't the411+// pivot's results: what the pivot mustn't write over.412+func (s *Sheet) typedAt(a Addr) bool {413+ if !s.cells.filledAt(a) {414+ return false415+ }416+ return s.cells.derivedAt(a) != slotPivot417+}418+407419 // setDerived stores or removes a derived cell without recording it for408420 // undo. Derived cells have no formula, so there are no references to409421 // index.@@ -13,16 +13,21 @@ import (
1313 // number, a boolean or text, with at most a format and a style; their1414 // slot holds the value itself, the text in the store's string table, the1515 // format and style in its table of looks, and the input only when it16-// isn't the value's own text. Everything else (formulas, notes, what17-// pivots and spills write, text typed with a leading ') is rich: the18-// slot points at a whole Cell in the store's side table.16+// isn't the value's own text. What pivots and spills write (derived17+// cells) is a slot too, marked by its kind: the value alone, its entry18+// being the value's text (derivedInput), and a spill's inferred format19+// in its look. Everything else (formulas, notes, text typed with a20+// leading ') is rich: the slot points at a whole Cell in the store's21+// side table.1922 type slot struct {2023 num float64 // slotNum's value; slotBool's, 1 or 02124 // ref is slotText's text in strs; slotNum's or slotBool's input in2225 // strs when it isn't the value's text (0 when it is); slotRich's cell2326 // in rich.2427 ref uint3225- look uint16 // the format and style in looks; 0 for neither28+ look uint16 // the format, style and inferred format in looks; 0 for none29+ // kind is one of the kinds below, with slotPivot or slotSpill added30+ // for a derived cell.2631 kind uint82732 // dec is how a slotNum without ref prints its input: the value with2833 // dec-1 decimals, or its shortest text when 0.@@ -36,17 +41,30 @@ const (
3641 slotBool3742 slotText3843 slotRich44+ slotErr // a derived cell's error value, its code in strs3945 )4046 47+// Marks of a derived cell's slot, added to its kind: a pivot's result48+// (Cell.derived) or part of a spilled array (Cell.spilled).49+const (50+ slotPivot uint8 = 1 << 651+ slotSpill uint8 = 1 << 752+ slotDerived = slotPivot | slotSpill53+)54+55+// base is sl's kind without the marks of a derived cell.56+func (sl slot) base() uint8 { return sl.kind &^ slotDerived }57+4158 // maxDec bounds the decimals a number's input may have to be kept as a4259 // count rather than as text.4360 const maxDec = 404461 45-// look is a cell's format and style, kept once per store however many46-// cells share it.62+// look is a cell's format and style, and a spilled cell's inferred63+// format, kept once per store however many cells share it.4764 type look struct {48- f Format49- st Style65+ f Format66+ st Style67+ auto Format5068 }5169 5270 // strTable holds the text of plain cells, each distinct string once,@@ -97,8 +115,7 @@ func (t *strTable) release(i uint32) {
97115 98116 // lookID is the entry of l in the table of looks, added if new, and99117 // false when the table is full.100-func (st *cellStore) lookID(f Format, sty Style) (uint16, bool) {101- l := look{f, sty}118+func (st *cellStore) lookID(l look) (uint16, bool) {102119 if l == (look{}) {103120 return 0, true104121 }@@ -118,21 +135,29 @@ func (st *cellStore) lookID(f Format, sty Style) (uint16, bool) {
118135 return i, true119136 }120137 121-// lookOf is sl's format and style.122-func (st *cellStore) lookOf(sl slot) look {138+// lookOf is sl's look, which the caller mustn't change.139+func (st *cellStore) lookOf(sl slot) *look {123140 if sl.look == 0 {124- return look{}141+ return &noLook125142 }126- return st.looks[sl.look]143+ return &st.looks[sl.look]127144 }128145 146+// noLook is the look of a slot without one, never written.147+var noLook look148+129149 // plainSlot is c as a plain slot, or false if c is rich. It adds the130150 // strings it uses to the table.131151 func (st *cellStore) plainSlot(c *Cell) (slot, bool) {132- if c.Note != "" || c.derived || c.spilled || !c.auto.IsZero() || strings.HasPrefix(c.Input, "=") {152+ switch {153+ case c.Note != "":154+ return slot{}, false155+ case c.derived || c.spilled:156+ return st.derivedSlot(c)157+ case !c.auto.IsZero() || strings.HasPrefix(c.Input, "="):133158 return slot{}, false134159 }135- lk, ok := st.lookID(c.Format, c.Style)160+ lk, ok := st.lookID(look{f: c.Format, st: c.Style})136161 if !ok {137162 return slot{}, false138163 }@@ -171,6 +196,44 @@ func (st *cellStore) plainSlot(c *Cell) (slot, bool) {
171196 return sl, true172197 }173198 199+// derivedSlot is plainSlot for a derived cell without a note: its value,200+// marked with what derived it, its entry being the value's text.201+func (st *cellStore) derivedSlot(c *Cell) (slot, bool) {202+ if c.expr != nil {203+ return slot{}, false204+ }205+ lk, ok := st.lookID(look{c.Format, c.Style, c.auto})206+ if !ok {207+ return slot{}, false208+ }209+ return st.derivedValue(c.Value, lk, c.derived), true210+}211+212+// derivedValue is the slot of a derived cell showing v with look lk: a213+// pivot's result if pivot, else a spilled cell. It adds the string v214+// uses to the table.215+func (st *cellStore) derivedValue(v Value, lk uint16, pivot bool) slot {216+ sl := slot{look: lk, kind: slotSpill}217+ if pivot {218+ sl.kind = slotPivot219+ }220+ switch v.Kind {221+ case Number:222+ sl.kind |= slotNum223+ sl.num = v.Num224+ case Bool:225+ sl.kind |= slotBool226+ sl.num = v.Num227+ case Text:228+ sl.kind |= slotText229+ sl.ref = st.strs.add(v.Str)230+ case Error:231+ sl.kind |= slotErr232+ sl.ref = st.strs.add(v.Str)233+ }234+ return sl235+}236+174237 // numDec is how input prints v, as slot.dec, or false if it isn't v175238 // printed plainly.176239 func numDec(v float64, input string) (uint8, bool) {@@ -253,14 +316,16 @@ func boolText(b bool) string {
253316 return "FALSE"254317 }255318 256-// input is the entry of a plain slot.319+// input is the entry of a plain slot. A derived slot's is its value's320+// text, as derivedInput writes it: a number's shortest text, TRUE or321+// FALSE, the text or the error code.257322 func (st *cellStore) input(sl slot) string {258323 switch {259- case sl.kind == slotBlank:324+ case sl.base() == slotBlank:260325 return ""261326 case sl.ref != 0:262327 return st.strs.strs[sl.ref]263- case sl.kind == slotBool:328+ case sl.base() == slotBool:264329 return boolText(sl.num != 0)265330 }266331 return strconv.FormatFloat(sl.num, 'f', int(sl.dec)-1, 64)@@ -268,13 +333,15 @@ func (st *cellStore) input(sl slot) string {
268333 269334 // slotValue is the value of a plain slot.270335 func (st *cellStore) slotValue(sl slot) Value {271- switch sl.kind {336+ switch sl.base() {272337 case slotNum:273338 return Value{Kind: Number, Num: sl.num}274339 case slotBool:275340 return Value{Kind: Bool, Num: sl.num}276341 case slotText:277342 return Value{Kind: Text, Str: st.strs.strs[sl.ref]}343+ case slotErr:344+ return Value{Kind: Error, Str: st.strs.strs[sl.ref]}278345 }279346 return Value{}280347 }@@ -284,6 +351,10 @@ func (st *cellStore) slotValue(sl slot) Value {
284351 func (st *cellStore) view(sl slot) *Cell {285352 lk := st.lookOf(sl)286353 c := &Cell{Input: st.input(sl), Value: st.slotValue(sl), Format: lk.f, Style: lk.st}354+ if sl.kind&slotDerived != 0 { // a derived cell has no expression355+ c.auto, c.derived, c.spilled = lk.auto, sl.kind&slotPivot != 0, sl.kind&slotSpill != 0356+ return c357+ }287358 switch sl.kind {288359 case slotNum:289360 c.expr = formula.Num{V: sl.num}@@ -295,11 +366,11 @@ func (st *cellStore) view(sl slot) *Cell {
295366 296367 // releaseSlot counts one fewer use of what sl refers to.297368 func (st *cellStore) releaseSlot(sl slot) {298- switch sl.kind {369+ switch sl.base() {299370 case slotRich:300371 st.rich[sl.ref] = richCell{}301372 st.richFree = append(st.richFree, sl.ref)302- case slotNum, slotBool, slotText:373+ case slotNum, slotBool, slotText, slotErr:303374 st.strs.release(sl.ref)304375 }305376 }@@ -176,8 +176,11 @@ func (s *Sheet) spillArea(a Addr, arr *functions.Array) (Rect, string) {
176176 return area, "Array result was not expanded because it would overwrite merged cells in " + ms[0].String()177177 }178178 old := s.spills[a]179- for at, c := range s.cells.anyInRange(area) {180- if at == a || c.Blank() || c.spilled && old != nil && old.why == "" && old.area.Contains(at) {179+ for at := range s.cells.anyKeysIn(area) {180+ if at == a || !s.cells.filledAt(at) {181+ continue182+ }183+ if s.cells.derivedAt(at) == slotSpill && old != nil && old.why == "" && old.area.Contains(at) {181184 continue182185 }183186 return area, "Array result was not expanded because it would overwrite data in " + at.String()@@ -188,20 +191,25 @@ func (s *Sheet) spillArea(a Addr, arr *functions.Array) (Rect, string) {
188191 // writeSpilled makes the cell at a show v as a spilled cell, keeping its189192 // formatting and note, and reports whether it changed.190193 func (s *Sheet) writeSpilled(a Addr, v Value, auto Format) bool {191- old := s.cells.get(a)194+ was, l, kind := s.cells.derivedOf(a)195+ spilled := kind == slotSpill192196 if v.Kind == Empty {193- if old == nil || !old.spilled {197+ if !spilled {194198 return false195199 }196- s.setDerived(a, old.leftover())200+ s.setDerived(a, s.cells.get(a).leftover())197201 return true198202 }199- if old != nil && old.spilled && old.Value == v && old.auto == auto {203+ if spilled && was == v && l.auto == auto {200204 return false // its input is its value's201205 }202- c := &Cell{Input: derivedInput(v), Value: v, auto: auto, spilled: true}203- if old != nil {204- c.Format, c.Style, c.Note = old.Format, old.Style, old.Note206+ if s.cells.setSpilled(a, v, auto) { // formatting kept, no formula to unlink207+ s.version++208+ return true209+ }210+ c := &Cell{Input: derivedInput(v), Value: v, Format: l.f, Style: l.st, auto: auto, spilled: true}211+ if old := s.cells.richAt(a); old != nil {212+ c.Note = old.Note205213 }206214 s.setDerived(a, c)207215 return true@@ -212,8 +220,8 @@ func (s *Sheet) writeSpilled(a Addr, v Value, auto Format) bool {
212220 func (s *Sheet) clearSpilled(area, keep Rect) []loc {213221 var changed []loc214222 var gone []Addr215- for at, c := range s.cells.anyInRange(area) {216- if c.spilled && !keep.Contains(at) {223+ for at := range s.cells.anyKeysIn(area) {224+ if s.cells.derivedAt(at) == slotSpill && !keep.Contains(at) {217225 gone = append(gone, at)218226 }219227 }@@ -18,9 +18,10 @@ import (
1818 // occupancy index (occupancy.go) has a bitmap per block of the rows1919 // holding a cell, and each block keeps their slots (slot.go) in row2020 // order, so a cell is found by indexing a column, a block and a popcount,21-// without hashing. A plain number costs its 16-byte slot. Rich cells22-// (formulas, notes, pivots' and spills' results) are whole Cells in a23-// side table the slot points at.21+// without hashing. A plain number costs its 16-byte slot, and so does a22+// pivot's or a spill's result (a derived cell, marked in its slot). Rich23+// cells (formulas, notes) are whole Cells in a side table the slot24+// points at.2425 //2526 // get returns a rich cell's own Cell, which recalculation updates in2627 // place, as ever, and a plain cell as a Cell made for the caller (a@@ -47,6 +48,12 @@ type cellStore struct {
4748 lookIdx map[look]uint16 // looks' entries4849 rich []richCell // rich cells, by slot.ref4950 richFree []uint32 // entries of rich not in use51+52+ // pivots counts the pivot's results stored (Cell.derived), and53+ // pivotArea holds them all while there are any, so they are found54+ // without visiting every cell.55+ pivots int56+ pivotArea Rect5057 }5158 5259 func newCellStore() cellStore { return cellStore{} }@@ -111,8 +118,8 @@ func (st *cellStore) has(a Addr) bool { return st.stored.has(a) }
111118 // filledAt reports whether the cell at a has contents: !get(a).Blank().112119 func (st *cellStore) filledAt(a Addr) bool { return st.filled.has(a) }113120 114-// richAt returns the cell at a if it is rich (a formula, a note, a115-// pivot's or a spill's result), or nil.121+// richAt returns the cell at a if it is rich (a formula or a note, or a122+// cell a slot can't hold), or nil.116123 func (st *cellStore) richAt(a Addr) *Cell {117124 b, i := st.find(a)118125 if b == nil || b.vals[i].kind != slotRich {@@ -148,9 +155,15 @@ func (st *cellStore) holds(a Addr, c *Cell) bool {
148155 if sl.kind == slotRich {149156 return st.rich[sl.ref].c == c150157 }151- if l := st.lookOf(sl); c.Note != "" || c.derived || c.spilled || !c.auto.IsZero() || c.Format != l.f || c.Style != l.st {158+ if l := st.lookOf(sl); c.Note != "" || c.auto != l.auto || c.Format != l.f || c.Style != l.st {159+ return false160+ }161+ if c.derived != (sl.kind&slotPivot != 0) || c.spilled != (sl.kind&slotSpill != 0) {152162 return false153163 }164+ if sl.kind&slotDerived != 0 {165+ return c.expr == nil && c.Value == st.slotValue(sl)166+ }154167 switch sl.kind {155168 case slotText:156169 return c.expr == nil && c.Input == st.strs.strs[sl.ref]@@ -201,6 +214,7 @@ func (st *cellStore) set(a Addr, c *Cell) {
201214 b.vals = slices.Insert(b.vals, i, slot{kind: slotBlank})202215 }203216 old := b.vals[i]217+ st.countPivot(a, st.isPivot(old), c.derived)204218 sl, ok := st.plainSlot(c) // before releasing old, which may share its strings205219 switch {206220 case ok:@@ -226,6 +240,7 @@ func (st *cellStore) setSlot(a Addr, sl slot) {
226240 i, _ = b.index(a.Row & (blockRows - 1))227241 b.vals = slices.Insert(b.vals, i, slot{kind: slotBlank})228242 }243+ st.countPivot(a, st.isPivot(b.vals[i]), false)229244 st.releaseSlot(b.vals[i])230245 b.vals[i] = sl231246 st.filledAs(a, st.filled.has(a), sl.kind != slotBlank)@@ -251,6 +266,7 @@ func (st *cellStore) delete(a Addr) {
251266 }252267 st.touch(a)253268 old := b.vals[i]269+ st.countPivot(a, st.isPivot(old), false)254270 if st.filled.has(a) {255271 st.filled.unmark(a)256272 }@@ -0,0 +1,143 @@
1+package sheet2+3+import (4+ "iter"5+ "slices"6+)7+8+// The store's side of derived cells (slot.go): finding the pivot's9+// results, reading what a derived cell shows without making a Cell, and10+// writing a spilled value straight into its slot.11+12+// isPivot reports whether sl holds a pivot's result.13+func (st *cellStore) isPivot(sl slot) bool {14+ if sl.kind == slotRich {15+ return st.rich[sl.ref].c.derived16+ }17+ return sl.kind&slotPivot != 018+}19+20+// countPivot counts the cell at a, which was a pivot's result or not and21+// becomes one or not.22+func (st *cellStore) countPivot(a Addr, was, is bool) {23+ switch {24+ case is && !was:25+ if st.pivots++; st.pivots == 1 {26+ st.pivotArea = Rect{From: a, To: a}27+ } else {28+ st.pivotArea = union(st.pivotArea, Rect{From: a, To: a})29+ }30+ case was && !is:31+ st.pivots--32+ }33+}34+35+// pivotKeys yields the addresses of the pivot's results stored.36+func (st *cellStore) pivotKeys() iter.Seq[Addr] {37+ return func(yield func(Addr) bool) {38+ if st.pivots == 0 {39+ return40+ }41+ for a := range st.anyKeysIn(st.pivotArea) {42+ if b, i := st.find(a); b != nil && st.isPivot(b.vals[i]) && !yield(a) {43+ return44+ }45+ }46+ }47+}48+49+// derivedOf returns what the cell at a holds as a derived cell would:50+// its value, format, style and inferred format, and slotPivot or51+// slotSpill for a pivot's result or a spilled cell, 0 for any other52+// cell (none included). It makes no Cell, so the derived cells being53+// rewritten are compared cheaply; the look mustn't be changed.54+func (st *cellStore) derivedOf(a Addr) (v Value, l *look, kind uint8) {55+ b, i := st.find(a)56+ if b == nil {57+ return Value{}, &noLook, 058+ }59+ sl := b.vals[i]60+ if sl.kind != slotRich {61+ return st.slotValue(sl), st.lookOf(sl), sl.kind & slotDerived62+ }63+ c := st.rich[sl.ref].c64+ return c.Value, &look{c.Format, c.Style, c.auto}, richDerived(c)65+}66+67+// richDerived is derivedOf's kind for a rich cell.68+func richDerived(c *Cell) uint8 {69+ switch {70+ case c.derived:71+ return slotPivot72+ case c.spilled:73+ return slotSpill74+ }75+ return 076+}77+78+// derivedAt is derivedOf's kind alone.79+func (st *cellStore) derivedAt(a Addr) uint8 {80+ b, i := st.find(a)81+ switch {82+ case b == nil:83+ return 084+ case b.vals[i].kind == slotRich:85+ return richDerived(st.rich[b.vals[i].ref].c)86+ }87+ return b.vals[i].kind & slotDerived88+}89+90+// autoAt is the format the cell at a infers: a formula's or a spilled91+// cell's.92+func (st *cellStore) autoAt(a Addr) Format {93+ b, i := st.find(a)94+ if b == nil {95+ return Format{}96+ }97+ if sl := b.vals[i]; sl.kind == slotRich {98+ return st.rich[sl.ref].c.auto99+ } else {100+ return st.lookOf(sl).auto101+ }102+}103+104+// setSpilled makes the cell at a, blank, formatting only or spilled, a105+// spilled cell showing v with the inferred format auto, keeping its106+// formatting, straight in its slot. It reports false, changing nothing,107+// for a cell a slot can't hold (one with a note), which the caller108+// stores as a Cell.109+func (st *cellStore) setSpilled(a Addr, v Value, auto Format) bool {110+ b, i := st.find(a)111+ var old slot112+ if b != nil {113+ old = b.vals[i]114+ }115+ if old.kind == slotRich {116+ return false117+ }118+ lk := old.look119+ if l := st.lookOf(old); l.auto != auto {120+ var ok bool121+ if lk, ok = st.lookID(look{l.f, l.st, auto}); !ok {122+ return false123+ }124+ }125+ if b == nil {126+ b = st.stored.mark(a)127+ i, _ = b.index(a.Row & (blockRows - 1))128+ b.vals = slices.Insert(b.vals, i, slot{kind: slotBlank})129+ }130+ st.countPivot(a, st.isPivot(old), false)131+ sl := st.derivedValue(v, lk, false) // before releasing old, which may share its string132+ st.releaseSlot(old)133+ b.vals[i] = sl134+ st.filledAs(a, st.filled.has(a), !derivedBlank(v))135+ st.touch(a)136+ return true137+}138+139+// derivedBlank reports whether a derived cell showing v is blank: its140+// entry, the value's text, is empty.141+func derivedBlank(v Value) bool {142+ return v.Kind == Empty || v.Kind == Text && v.Str == ""143+}