docs/data.md explains pivots, the editor and frequency tables; files.md the version 5 pivot field and XLSX values; keys.md, architecture.md, observability.md and extending.md follow; the roadmap marks both done.
- commit
- 08ad294dbc0b1db4aa3a7d8bab995aa88184656e
- parent
- fbc938eff37337fbc4dd0506e41cc9743c914e9f
- author
- Dami <dami.etoile@gmail.com>
| M | README.md | 3 | +- |
| M | ROADMAP.md | 4 | +- |
| M | docs/ | 20 | ++++++++- |
| M | docs/ | 65 | +++++++++++++++++++++++++++++- |
| M | docs/ | 2 | +- |
| M | docs/ | 35 | +++++++++++++++- |
| M | docs/ | 2 | + |
| M | docs/ | 5 | ++- |
@@ -42,7 +42,8 @@ searches every command.
4242 while typing a formula, and Sheets' keys for moving between them.4343 - **Data tools.** Freeze rows and columns, multi-column sort, filters with4444 value pickers and conditions, find and replace with regular expressions,45- tracing precedents and dependents.45+ tracing precedents and dependents, live pivot tables and frequency46+ tables on sheets of their own.4647 - **Charts.** Column, bar, line and pie charts that float over the grid and4748 update live; real images in terminals with the kitty graphics protocol4849 (kitty, Ghostty, WezTerm), text elsewhere.@@ -39,7 +39,7 @@ quality keep pace with features: see [docs/UX.md](docs/UX.md).
3939 | Search (`n`/`N`) and non-destructive row filter | csvlens | | M |4040 | Fill down / fill series (1,2,3, Jan, Feb) | Excel, Sheets | needs Copy ref adjustment | S-M |4141 | Optional vim keymap (`hjkl`, counts, `:` commands) | sc-im, csvlens | `bubbles/v2/key` bindings | M |42-| Frequency table of a column as a derived sheet | VisiData Shift+F | | M |42+| Frequency table of a column as a derived sheet | VisiData Shift+F | a preset pivot table | M, done: Data > Frequency table, Alt+Shift+F |4343 4444 ## Phase 3: terminal-native features4545 @@ -74,7 +74,7 @@ Avoid in the main binary: DuckDB and automerge (both need cgo).
7474 ## Phase 5: beyond7575 7676 - Multiple sheets with references between them, in Sheets' style (`Sheet2!A1`) rather than 1-2-3 R3's `B:A1`, and sheet tabs. Done.77-- Pivot tables as derived sheets. L.77+- Pivot tables as derived sheets. Done: Data > Pivot table and its editor, live results the engine owns, saved as their definition (file version 5), values in XLSX; see [docs/data.md](docs/data.md#pivot-tables).7878 - Macros: replay the command log; Starlark (`go.starlark.net`) for scripts, with step limits.7979 - Serve over SSH with `charm.land/wish/v2`.8080 - Decimal mode for currency (`cockroachdb/apd/v3`), opt-in. Done: File > Settings > Decimal arithmetic, boundary in the README.@@ -85,6 +85,19 @@ style.
8585 the cells, widths, names, charts, view state and sheet list they change,8686 on any sheet, so a step can be reversed exactly; multi-cell operations8787 are one step. A deleted sheet keeps its cells, so undo brings it back.88+- **Pivot tables.** A sheet may hold a pivot (`pivot.go`): a definition89+ naming its source by sheet name, as formulas do, and a region of90+ derived cells from A1 that the engine owns. When a recalculation marks91+ a cell of a pivot's source range, or its definition changes, the pivot92+ is recomputed after the formulas (`pivotcalc.go` groups the source93+ rows into a tree of accumulators, `pivotlayout.go` lays out Sheets'94+ table) and only the result cells that changed are rewritten, outside95+ the undo history, then what reads them recalculates, pivots of pivots96+ included (bounded, as a loop can't be defined). Undo restores the97+ source and the definition, and the results follow. Derived cells are98+ constants to formulas, values to copies and exports, refused by `Set`,99+ and never saved: the file keeps the definition (`pivotfile.go`). A100+ frequency table is a pivot with a preset definition.88101 - **JEV.** The engine never touches the network. JEV functions describe a89102 question and look up the answer in the workbook's `RemoteSource`, set90103 with `SetRemote`; `internal/jev` answers from a cache and queues new@@ -124,7 +137,10 @@ context menus, the command palette, the help overlay and mouse gestures
124137 that stand for one (double-clicking a tab renames it through125138 `sheet.rename`), so they can't disagree. `runCommand` is the one place a126139 command runs: telemetry times it there, and a command log or macro127-recorder would attach there.140+recorder would attach there. A command that writes cells declares the141+range it writes (`edits`), and `runCommand` refuses it over a pivot142+table's results, so a new editing command is guarded by saying what it143+edits.128144 129145 **Model and components.** `Model` (`model.go`) is the root: it holds the130146 file, the mode and the note on the context line, owns one component for@@ -138,7 +154,7 @@ draw. The components:
138154 | edit line | `lineEdit` | the one-line editor shared by cell entries, prompts and search fields (`line.go`) |139155 | cell entry | `entry`, `assist` | typing into a cell, pointing at references, other sheets while pointing, formula suggestions and signatures (`entry.go`, `assist.go`) |140156 | prompt | `prompt` | a question on the context line, typed or pointed at (`prompt.go`) |141-| overlays | `overlay` | whatever has taken over input: menus (`menuoverlay.go`), the palette and pickers (`palette.go`, `names.go`), the filter picker, the find, sort and choice bars, the chart editor and selection, the shortcuts |157+| overlays | `overlay` | whatever has taken over input: menus (`menuoverlay.go`), the palette and pickers (`palette.go`, `names.go`), the filter picker, the find, sort and choice bars, the chart editor and selection, the pivot editor (`pivoteditor.go`, `pivotactions.go`), the shortcuts |142158 | sheet tabs | `tabStrip` | where each sheet was left, the tab strip's scroll, layout and clicks (`tabstrip.go`) |143159 | mouse | `mouseState` | drags, hover, double clicks, the fill handle (`mouse.go`, `fill.go`) |144160 | import | `transfer` | the import in progress, its progress display and cancelling (`transfer.go`) |@@ -27,8 +27,8 @@ while typing enters the same entry into every selected cell.
2727 Ctrl+Z undoes and Ctrl+Y or Ctrl+Shift+Z redoes, 100 steps deep, across2828 every sheet: undo shows the sheet a step changed. Every change is covered:2929 typing, clearing, paste, fill, sort, insert and delete, formats, widths,30-names, charts, freeze, filters, and adding, deleting, renaming, moving and31-duplicating sheets. A multi-cell change is one step,30+names, charts, freeze, filters, pivot tables, and adding, deleting,31+renaming, moving and duplicating sheets. A multi-cell change is one step,3232 and the context line says what was undone, e.g. `Undid: clear B3:B5`.3333 3434 ## Rows and columns@@ -62,6 +62,67 @@ as "greater than" or "text contains". Filtered-out rows are hidden, not
6262 deleted: row numbers show the gap, navigation skips them, formulas still6363 count them, and the status line says how many rows are hidden.6464 65+## Pivot tables66+67+Data > Pivot table summarizes the selection, or the table around the68+active cell, on a new sheet named `Pivot Table 1`, as Sheets' Insert >69+Pivot table does. The data's first row names the fields. The pivot editor70+opens docked at the right of the grid, with the results updating behind it:71+72+- **Rows** group the data by a field's values, one row per value; with two73+ or more, the groups nest and each outer group gets a subtotal row.74+- **Columns** spread the groups across columns by a field's values.75+- **Values** summarize a field for each group: SUM, COUNTA, COUNT,76+ COUNTUNIQUE, AVERAGE, MAX or MIN (Sheets' "Summarize by"), shown as they77+ are or as a share of their row, column or grand total ("Show as").78+ A new value is summed when its field holds numbers, and counted79+ otherwise.80+- **Filters** leave out rows by a field's values or a condition, in the81+ same picker as a filter's column.82+- **Grand total row** and **Grand total column** add Sheets' totals.83+84+Up and Down pick a line. Space adds a field to the section it's on (pick85+it from the fields of the data, type to narrow the list), opens a filter's86+values, or flips a toggle; Space on the data range points at a new one on87+its sheet. On a row or column field, Left and Right order its groups A to88+Z, Z to A, or by a value's total, smallest or largest first, and89+Shift+Up and Shift+Down move it before or after the others, which changes90+how the groups nest. On a value, Left and Right change how it's91+summarized and S how it's shown. Del removes a field. Every change is an92+undo step; Enter keeps them, Esc undoes them, and removes a pivot just93+created. Data > Edit pivot table opens the editor again.94+95+Groups follow Sheets: text ignoring case (`east` joins `East`), numbers96+and dates by value, with the source's format, so dates show as dates.97+Numbers and text never share a group: `1` and the text `'1` are two.98+Groups sort numbers first, then text, booleans and errors, and blank99+cells, shown as `(blank)`, last. Rows blank across the whole range are100+left out, so a range can reach past the data. SUM, AVERAGE, MIN and MAX101+keep the column's number format; an error in a summed column shows as that102+error, as SUM would.103+104+The results are live: any change to the data, typed, pasted, filled or105+computed by a formula, recomputes the pivot, and rows or columns inserted106+inside the range widen it. Formulas can read the results (`='Pivot Table107+1'!B5`), and copying them pastes their values. The results themselves108+can't be edited, as in Sheets: typing, pasting, clearing, formatting,109+sorting or inserting rows or columns over them is refused with a note on110+the context line. A pivot shows `#REF!` in A1, explained on the context111+line, when its data's sheet is deleted or when its results would112+overwrite a cell typed next to them; undo or moving the cell brings it113+back. The sheet's columns widen to fit the results until you set a width114+yourself.115+116+## Frequency tables117+118+Data > Frequency table (column stats), or Alt+Shift+F (VisiData's119+Shift+F; plain Shift+F types an F), counts each value of the active120+column of the table around the active cell on a new sheet, `Frequency of121+<field>`: the value, how many rows have it and their percentage, most122+frequent first, with a Grand Total. Blank cells are counted too. It is a123+pivot table like any other, so it stays live, and the pivot editor124+changes it.125+65126 ## Find and replace66127 67128 Ctrl+F opens a find bar on the context line: matches highlight as you type,@@ -87,7 +87,7 @@ whole sheet per keystroke or per frame needs a cache or an index.
8787 8888 The `.012` format only gains optional fields, and the version rises only when8989 a file uses a feature older builds can't read (names, freeze and filters in90-v3, several sheets in v4). Old files always load. Settings that older90+v3, several sheets in v4, pivot tables in v5). Old files always load. Settings that older9191 builds can safely ignore (decimal arithmetic) don't raise the version.9292 9393 ## Keeping it honest@@ -10,7 +10,7 @@ bar; Esc cancels. Saving an imported sheet asks whether to save it as a
1010 | Format | Import | Download |1111 |---|---|---|1212 | CSV, TSV | Delimiter (`,` `;` tab `\|`), UTF-8 BOM, UTF-16 and Windows-1252 detected; entries become numbers, dates, currency and percentages as if typed; formulas stay text | Values as shown, as Sheets' Download does |13-| Excel `.xlsx` | Every sheet, opening on the one Excel showed: values, formulas (references between sheets too), workbook named ranges, number formats, bold, italic, underline, strikethrough, alignment, column widths. Formulas 012 can't read (unknown functions) keep their values | Every sheet, the same, with formulas in Excel's syntax and their results cached. JEV functions and `#AND#` save as values |13+| Excel `.xlsx` | Every sheet, opening on the one Excel showed: values, formulas (references between sheets too), workbook named ranges, number formats, bold, italic, underline, strikethrough, alignment, column widths. Formulas 012 can't read (unknown functions) keep their values; Excel pivot tables come in as the values they showed | Every sheet, the same, with formulas in Excel's syntax and their results cached. JEV functions and `#AND#` save as values, and so do pivot tables: Excel gets the results, not a pivot |1414 | SQLite | Pick a table or view, or type a query; a header row names the columns | The sheet shown or the selection as a table, first row as column names; a table of that name is replaced |1515 | Parquet | Every column, with dates and timestamps; lists joined with commas | |1616 | Lotus 1-2-3 `.wk1`, `.wks` | Numbers, labels with their alignment, formats, column widths, formulas translated (references, operators, `@SUM`, `@AVG`, `@IF`, `@ROUND`, `@PMT` and 60 more) or kept as values | |@@ -35,7 +35,8 @@ read naturally and files merge reasonably in version control:
3535 A cell without formatting is just what was typed; a formatted cell is a3636 small object. Version 3 adds named ranges, frozen panes and a filter, and is3737 only written when a sheet uses one of them, so older builds of 012 can open38-everything else. Charts are an optional `charts` field that older builds38+everything else (version 4 adds several sheets and version 5 pivot tables,39+below). Charts are an optional `charts` field that older builds3940 ignore. Saves are atomic: 012 writes a temporary file and renames it.4041 4142 Other formats import as one sheet named after the file (or the SQLite@@ -74,3 +75,33 @@ arithmetic, the sheet shown when saved) stay at the top:
7475 ```7576 7677 Older builds refuse version 4 files rather than lose sheets.78+79+## Pivot tables80+81+A workbook with a pivot table is version 5: version 4 with a `pivot`82+field, on one line after the cells of the pivot's sheet, holding its83+definition. The results are never saved; they are computed again when84+the file opens, so the file stays small and can't disagree with its85+data. Builds that know only version 4 refuse the file rather than show an86+empty sheet; a workbook without pivots is still written as version 4.87+88+```json89+{90+ "name": "Pivot Table 1",91+ "cells": {},92+ "pivot": {"source":"Sales!A1:D200","rows":[{"column":"B"},{"column":"A","order":"desc","sortBy":1}],"columns":[{"column":"C"}],"values":[{"column":"D","summarize":"sum"},{"column":"D","summarize":"counta","showAs":"percent_of_total","name":"Share"}],"filters":[{"column":"A","hidden":["North"]}],"rowTotals":true,"columnTotals":false}93+}94+```95+96+- `source` is the data, with its sheet; `Sales!#REF!` once the range was97+ deleted. The sheet is by name, as in formulas, and follows renames.98+- `rows` and `columns` are fields by column letter on the source sheet,99+ with `order` `desc` for Z to A and `sortBy` the value, counting from 1,100+ whose totals order the groups.101+- `values` summarize a column: `sum`, `counta`, `count`, `countunique`,102+ `average`, `max`, `min`, or `rows` (every row, blank or not, which103+ frequency tables use); `showAs` is `percent_of_row`,104+ `percent_of_column` or `percent_of_total`; `name` replaces the header.105+- `filters` take a filter column's criteria: `hidden` values and a106+ `condition` with its `value`.107+- `rowTotals` and `columnTotals` are the grand total row and column.@@ -28,6 +28,8 @@ Inside the grid, 012 works like Google Sheets.
2828 | View > Freeze | Keep rows or columns on screen while the rest scrolls |2929 | Data > Sort sheet, Sort range | Sort by the active column A to Z or Z to A, or pick columns and order on a bar (Left/Right column, Space order, Alt+A add, Alt+H header row) |3030 | Alt+Down, click ▾ in a header | With a filter (Data > Create a filter): pick the column's values (Space checks, type to search) or a condition |31+| Data > Pivot table | Summarize the table on a new sheet; in the pivot editor Up/Down pick a line, Space adds a field (or opens a filter, flips a total, changes the data range), Left/Right change a field's order or summary, S its "show as", Shift+Up/Down move it, Del removes it, Enter keeps, Esc undoes. Data > Edit pivot table reopens it |32+| Alt+Shift+F | Frequency table of the active column on a new sheet (Data > Frequency table), as VisiData's Shift+F |3133 | Ctrl+Alt+= / Ctrl+Alt+- | Insert rows above / delete the selected rows (columns when whole columns are selected) |3234 | Ctrl+Enter while typing | Enter the same entry in every selected cell |3335 | F4 while typing a formula | Cycle the reference at the caret through A1, $A$1, A$1, $A1 |@@ -54,6 +54,8 @@ same, as an attribute for queries), `service` (`012`) and `dur_ms`.
5454 | `frame` | every frame, at debug level | `key_ms` |5555 | `command` | every registered command | `id`, e.g. `data.sort` |5656 | `sort`, `filter`, `find`, `replace`, `fill` | the operation itself | `rows`, `cols`, `keys`; `hidden`; `matches`; `replaced`; `cells` |57+| `pivot` | every pivot table recomputation (a change to its data or definition) | `records` (source rows summarized), `groups` (row groups at every depth), `cells` (results), `failed` (shows `#REF!`) |58+| `frequency` | making a frequency table | `rows` (data rows counted) |5759 | `import`, `export` | file transfers (`internal/fileio`) | `format`, `bytes`, `rows`, `cells`, `notes` |5860 | `save`, `open` | the native `.012` file | `cells`, `bytes` (save times serializing, on the UI goroutine) |5961 | `jev` | each question sent | `kind`, `queued` (waiting when sent), `outcome` (`ok` or `failed`) |@@ -62,7 +64,8 @@ same, as an attribute for queries), `service` (`012`) and `dur_ms`.
6264 Failures are logged at `WARN` with an `error` message.6365 6466 The engine doesn't log: `sheet.OnRecalc` hands each recalculation's65-counts to `cmd/012`, which logs them. The call sites use67+counts to `cmd/012`, which logs them, and `sheet.OnPivot` each pivot68+table's. The call sites use6669 `internal/telemetry` (`Start`/`End` spans, `Event`, `Frame`, `Set`), so6770 the backends (the JSON file, OTLP) sit behind it without touching them.6871