- commit
- 7f52fda7e706a4dfee4a2a0b6b3115e03532abb7
- parent
- a8fea1c6dca23447fe190a69cf0bf0861c619e00
- parent
- 07dc4228e10d21a249f6894d1cf6074546f66f10
- author
- Dami <dami.etoile@gmail.com>
@@ -49,6 +49,10 @@ rest: building from a clone, importing files, `012 serve` and settings.
4949 nushell's NUON, XLSX, SQLite, Parquet and Lotus 1-2-3, export to CSV,5050 TSV, JSON, NUON, XLSX, SQLite and a self-contained web page with its5151 charts as SVG ([files](docs/files/README.md)).52+- **Tables too big for any grid**: a Parquet file or a SQLite table of53+ tens of millions of rows links as a source on a tab of its own,54+ scrolled, sorted and filtered in place, with SUMIFS, XLOOKUP and pivot55+ tables streaming over every row ([linked sources](docs/files/sources.md)).5256 - **For agents**: `012 describe`, `get`, `set --dry-run` and `recalc`5357 with JSON results, a Claude Code skill, and `012 mcp`, an MCP server5458 whose writes go through the same checks as yours, with views@@ -86,6 +90,8 @@ rest: building from a clone, importing files, `012 serve` and settings.
8690 | Arrays that spill | Nushell notebooks |8791 |  |  |8892 | JEV functions in formulas | Macros |93+|  | |94+| Linked sources, too big for any grid | |8995 9096 Each guide shows its own recordings too. They are9197 [VHS](https://github.com/charmbracelet/vhs) tapes in [`demos/`](demos),@@ -20,10 +20,7 @@ Sizes: S (a day or two), M (about a week), L (weeks).
2020 The grid is Excel's, 1,048,576 x 16,384, and memory is bounded by the2121 `max-cells` budget (ten million cells) rather than the grid. The budget2222 rises in measured steps; see [Bounds of support](docs/contributing/limits.md#what-would-raise-the-bounds).23-24-| Item | Result | Size |25-|---|---|---|26-| Linked, paged read-only ranges over Parquet and SQLite that feed pivots and formulas by streaming | Sources too big for any grid | L |23+Tables past it are read in place as [linked sources](docs/files/sources.md).2724 2825 ### 2. A TUI Jupyter for nushell2926 @@ -130,6 +127,7 @@ needs cgo.
130127 - Commands for scripts without the screen: `012 get` (text, CSV, TSV, JSON or NUON), `012 set`, `012 recalc` and `012 export`, running notebooks and JEV only behind flags: [Scripts](docs/files/scripts.md)131128 - `012 diff` cell by cell, as git's diff command or textconv, and `012 merge-driver` merging cell by cell with conflicts noted on the cells: [Diff and merge in git](docs/files/git.md)132129 - Following files: a linked region follows a CSV, TSV, JSON lines or NUON file as it grows and any importable file as it's rewritten, keeping every row or the last ones, its rows arriving as the change stream: [Following files](docs/files/following.md)130+- Linked sources: a Parquet file or a SQLite table or query read in place on a tab of its own, scrolled, sorted and filtered however many rows it has, formulas and pivot tables streaming over it: [Linked sources](docs/files/sources.md)133131 - The `.012` format written down with a version, and each release's workbooks kept as fixtures that every later build opens and saves unchanged: [The .012 format](docs/files/format.md#versions), [Releasing](docs/contributing/releasing.md#file-fixtures)134132 - Web pages in 012's look: a sheet, a range or a chart as one self-contained `.html` file, charts as SVG, from File > Download and `012 export`: [Files](docs/files/README.md#web-pages)135133 - A crash keeps unsaved work for recovery, restores the terminal and writes a report, locally and in `012 serve`: [Saving](docs/files/saving.md#if-012-crashes)@@ -0,0 +1,37 @@
1+// Command gensales writes the sales the linked sources demo reads2+// (sources.tape): ten million rows of Parquet, an id, one of eight3+// categories, an amount and a date each, the same every time4+// (internal/stress's SalesParquet), into the file named on its command5+// line.6+package main7+8+import (9+ "flag"10+ "fmt"11+ "os"12+ "path/filepath"13+14+ "github.com/FelineStateMachine/012/internal/stress"15+)16+17+func main() {18+ rows := flag.Int("rows", 10_000_000, "how many rows to write")19+ flag.Parse()20+ if flag.NArg() != 1 {21+ fmt.Fprintln(os.Stderr, "usage: gensales [-rows n] out.parquet")22+ os.Exit(2)23+ }24+ out := flag.Arg(0)25+ dir, err := os.MkdirTemp(filepath.Dir(out), "gensales")26+ if err == nil {27+ defer os.RemoveAll(dir)28+ var name string29+ if name, err = stress.SalesParquet(dir, *rows); err == nil {30+ err = os.Rename(name, out)31+ }32+ }33+ if err != nil {34+ fmt.Fprintln(os.Stderr, "gensales:", err)35+ os.Exit(1)36+ }37+}@@ -0,0 +1,48 @@
1+# Linked sources: ten million rows of Parquet (made by gensales) linked2+# as a source (Data > Linked file > Link a source, here from the3+# palette), sorted by amount Z to A in the background, its rows4+# numbered past the grid's last, and read whole by a SUMIFS on5+# Sheet1.6+Output out/sources.gif7+Source lib/setup.tape8+9+Hide10+Type "go run ./gensales out/sales.parquet && clear && ../bin/012" Enter11+Wait+Screen@90s /Help/12+Show13+14+Sleep 500ms15+Ctrl+K16+Type@80ms "link a source"17+Sleep 400ms18+Enter19+Sleep 500ms20+Type@40ms "out/sales.parquet"21+Enter22+Sleep 800ms23+Down24+Up25+Sleep 1s26+Screenshot out/stills/sources-linked.png27+Sleep 400ms28+Right 229+Ctrl+K30+Type@80ms "sort sheet z to a"31+Sleep 400ms32+Enter33+Wait+Screen@30s /sorted by amount Z to A/34+Sleep 1.2s35+Screenshot out/stills/sources-sorted.png36+Sleep 400ms37+Ctrl+K38+Type@80ms "previous sheet"39+Sleep 400ms40+Enter41+Sleep 400ms42+Type "North"43+Tab44+Type@40ms `=SUMIFS(sales[amount],sales[cat],"north")`45+Enter46+Sleep 1.5s47+Screenshot out/stills/sources-formulas.png48+Sleep 1s@@ -20,6 +20,7 @@ internal/locale the locales: separators, date order, currency, formula separato
2020 internal/fileio import and export: CSV, TSV, XLSX, SQLite, Parquet, Lotus .wk1, JSON, NUON; tables on streams2121 internal/nuon nushell's object notation: typed values, tables read a row at a time, written back2222 internal/live the sources of linked regions: files followed as they grow or are rewritten23+internal/paged linked sources read in place: the host answering formulas and pivots by streaming them, the pages a tab shows2324 internal/headless workbook files without the screen: references, get, set, recalc, notebooks and JEV when asked, atomic saves2425 internal/diff workbooks compared and merged cell by cell, from their files' fields2526 internal/chart chart layout, text rendering and kitty image encoding@@ -43,6 +44,7 @@ internal/ui the Bubble Tea model: modes, menus, overlays, rendering
4344 picker the searchable list behind the palette and every picker4445 cmdline the : command line and its completions4546 nbview a notebook tab as Jupyter's: its toolbar, cells, outputs, command and edit modes, mouse, the code editor and its language providers47+ srcview a linked source's tab: its rows a page at a time, the row numbers and scrollbar over every row4648 findbar find and replace, a bar on the context line4749 evalview Data > Evaluate formula: a formula stepped through part by part4850 themepicker File > Settings > Theme, previewing as it moves@@ -72,7 +74,7 @@ flowchart TD
7274 direction LR7375 ui[internal/ui] ~~~ fileio[internal/fileio] ~~~ live[internal/live] ~~~ nushell[internal/nushell]7476 chart[internal/chart] ~~~ jev[internal/jev] ~~~ stress[internal/stress] ~~~ oracle[oracle]75- headless[internal/headless] ~~~ diff[internal/diff] ~~~ room[internal/room]77+ headless[internal/headless] ~~~ diff[internal/diff] ~~~ room[internal/room] ~~~ paged[internal/paged]7678 end7779 callers --> sheet[internal/sheet]7880 sheet --> functions[internal/functions]@@ -260,6 +262,21 @@ style.
260262 table as `nu.name`, resolved when evaluated as named ranges are. The261263 file keeps the definitions (`regionfile.go`), and converts the command262264 regions of earlier files into notebook cells (`notebookfile.go`).265+- **Linked sources.** A [linked source](../files/sources.md) is a tab266+ whose one region is `Paged` (`source.go`): a Parquet file or a SQLite267+ table named as a table is (`sales[amount]`, `nu.sales`), holding no268+ cells, whose table may run past the grid's last row. The engine never269+ reads it. What its host found on opening it (the columns, their270+ formats, the rows) is kept outside the undo history271+ (`SetSourceShape`), and everything that reads it asks the workbook's272+ `SourceHost` (`sourceask.go`): a function given one of its ranges is273+ asked whole as a `functions.StreamCall`, a cell or a range read some274+ other way as its values, and a pivot table over it as its groups275+ (`pivotdata.go`, where a pivot reads a sheet's cells or a source's276+ rows streaming past through one interface). An answer not known yet277+ is `Pending`, the formula or pivot waiting for it, and278+ `RecalcSourced` recomputes what waited once it's known;279+ `SourceChanged` recomputes what read a source whose file changed.263280 - **Notebooks.** A notebook tab is a sheet whose `regionState` holds264281 cells (`notebook.go`), so adding, editing, moving and deleting cells265282 are undo steps as any change is, and the tab is named, moved, hidden@@ -324,6 +341,36 @@ values and names nothing but its region, so applying the same ops in the
324341 same order to the same workbook makes the same cells, whoever applies325342 it. `OnLive` times each op for telemetry.326343 344+### Questions to linked sources345+346+A formula reading a linked source asks, and the answer comes back in347+the background, as a JEV question's does:348+349+```mermaid350+sequenceDiagram351+ participant E as the engine (a recalculation)352+ participant H as paged.Host353+ participant J as a job, on a goroutine354+ participant F as fileio.Source355+ E->>H: SUM(sales[amount]) as a StreamCall356+ H-->>E: not known: queued, the cell Loading…357+ H->>J: the call, and the source as it was opened358+ J->>F: the column, streamed in batches359+ J->>J: functions.EvalStream over a Book that reads the source360+ J-->>H: the answer, kept until the file changes361+ H->>E: RecalcSourced: the cells that waited362+```363+364+The job runs the function library's own function (`EvalStream`) over365+a `Book` that reads the source (`paged/book.go`): an aggregate folds366+the column as it streams, a lookup walks it, and the functions over367+aligned ranges read them a window of rows at a time when the `Reader`368+says it is streaming (`functions/streamed.go`), so every function369+means what it means on a sheet. A function that holds what it reads is370+given a source's range only up to `max-cells` cells. Pivot tables371+gather their groups the same way (`sheet.GatherPivot`), and the engine372+lays them out when the answer arrives.373+327374 ### Functions328375 329376 `internal/functions` holds the function library: the `FuncDef` table@@ -351,6 +398,8 @@ the engine implements once per sheet (`reader` in `recalc.go`):
351398 | `RangeAgg(sheet, r)` | the running aggregate SUM-like functions share within a recalculation |352399 | `Fold(sheet, r, agg)` | a range added to a SUM-like function's aggregate, as the engine reads it |353400 | `Ask(call)` | a JEV question's answer, `Pending` or `ErrNoRemote` |401+| `Paged(sheet)` | whether a sheet is a linked source's tab, whose ranges a function is given whole |402+| `Stream(call)` | the answer to a call over linked sources (`StreamCall`), `Pending` while it's worked out |354403 355404 Sheets are named as references write them, so resolving names, missing356405 sheets (`#REF!`), cycles and the depth limit stay the engine's. The@@ -495,6 +544,7 @@ draw. The components:
495544 | mouse | `mouseState` | drags, hover, double clicks, the fill handle (`mouse.go`, `fill.go`) |496545 | import | `transfer.Transfer`, `pipeState` | the import in progress, its progress display and cancelling (package `transfer`); choosing and placing imports (`transfer.go`, `importplace.go`); standard input read as a sheet, and what a pipeline gets on quitting (`pipe.go`) |497546 | linked files | `followState` | a source (`live.Source`) for each linked region, polled one poll at a time on a command every `live.Interval` and reconciled with the workbook after every update, so undo, opening a file and unlinking need nothing of their own; trust in files outside a workbook's folder (`follow.go`); Data > Linked file and Import's Follow the file (`linked.go`) |547+| linked sources | `sourceState` | the workbook's `paged.Host` (a room's in `012 serve`, run by whoever keeps the room) and its jobs, run a few at a time on commands, their answers applied as they come; each file looked at every `live.Interval` (`sources.go`); the tab shown, package `srcview`, its pages read as it scrolls, and the commands that act on a source (`srcscreen.go`, `srccmds.go`) |498548 | notebooks | `nbview.View`, `nbState` | a view of each notebook tab, its selection, mode, scroll and editor (package `nbview`); the tab on the screen (`nbscreen.go`); the notebook's commands and keys (`notebook.go`); cells running one at a time in the background, their queue, stale outputs, trust (`nbrun.go`); what a run reads (`nbjob.go`); outputs sent to sheets (`nbsend.go`) |499549 | macros | `recorder`, `macroState` | a recording in progress (`macrorec.go`); a macro running, trust in the file's macros (`macrorun.go`); what scripts act on (`macrohost.go`, `macrohostnav.go`); Data > Macros and the manager (`macro.go`, `macromanage.go`) |500550 | others | `clipboard`, `trace`, `traceView`, `chartState`, `jevRunner`, `terminal`, `session` | what Ctrl+V pastes, a trace being stepped through and tracing that stays on (`traceview.go`), chart commands' target, JEV questions in flight, what the terminal supports and the chart images sent to it, what outlasts the file open (the `:` history, whether keys can be held: `keyboard.go`) |@@ -524,6 +574,7 @@ methods off `ui.Model`'s exported API:
524574 | `filterpick` | `filterpick.Host` | theme, size, the edit line, close, the locale (5); what applying and cancelling do are callbacks, as the sheet's filter and a pivot's differ |525575 | `cmdline` | `cmdline.Host` | theme, size, the edit line, close, the commands to complete, run a line, fail, the session's history (8) |526576 | `nbview` | `nbview.Host` | theme, locale, the cells, a cell's output, how its run stands, run a command, keep a cell's new source (7); the language's highlighter, completer and checker are `nbview.Providers`, small interfaces of their own |577+| `srcview` | `srcview.Host` | theme, locale, the source's columns, how many rows the tab shows, a row, ask for rows (6); the rows come from `paged.Pages`, read a page at a time |527578 | `suggest` | `suggest.Host` | theme, size, the edit line, whether an entry is being typed, the entry's sheet, the formula as parsed, where the formula bar's text starts (7) |528579 | `review` | `review.Host` | theme, size, close, the agents' suggestions waiting, accept, reject, show a cell (7) |529580 | `themepicker` | `themepicker.Host` | a picker's host, and the current theme, the themes directory, preview, keep, whether keys can be held (10) |@@ -570,7 +621,7 @@ hints) are what every overlay is drawn with.
570621 571622 **Packages.** `theme`, `rowtext`, `formula`, `overlay` and `lineedit`572623 depend on nothing in `ui`, so they can be tested and measured alone. The573-components in `picker`, `cmdline`, `nbview`, `themepicker`, `findbar`, `rules`,624+components in `picker`, `cmdline`, `nbview`, `srcview`, `themepicker`, `findbar`, `rules`,574625 `sortbar`, `filterpick`, `choicebar`, `shortcuts`, `evalview`, `suggest`, `review`, `tabstrip`575626 and `transfer` build on them and reach the model only through their576627 hosts, with unit tests of their own against fake hosts. A component moves out of package@@ -620,6 +671,18 @@ cell run as a stream (`live.Stream`), whose pipeline nu runs printing
620671 each value as NUON on a line of its own (`nushell.Stream`), read by the621672 same `Tail` as it arrives.622673 674+A `Source` (`source.go`) is a Parquet file or a SQLite table or query675+read in place, for [linked sources](../files/sources.md): it streams676+the columns asked for (Parquet reads only their column chunks, from the677+row group holding the first row), fetches rows by number, and orders678+them as a `SourceView`, pushed down to SQL for SQLite (the rowids in679+order in a work database of its own, the source opened read-only) and680+streamed for Parquet (a filter's matches found in one pass, a sort681+merged from sorted runs in temporary files, either kept as a file of682+row numbers a page reads). Values come out typed as an import types683+them (`sourcecell.go`), and every method is safe from several684+goroutines at once.685+623686 XLSX is read by 012's own SpreadsheetML reader on `archive/zip` and624687 `encoding/xml`: the workbook, shared strings (kept end to end in one625688 buffer) and styles first, then each worksheet a token at a time, a row@@ -183,8 +183,9 @@ fails it.
183183 184184 The `.012` format only gains optional fields, and the version rises only when185185 a file uses a feature older builds can't read (names, freeze and filters in186-v3, several sheets in v4, pivot tables in v5). Old files always load. Settings that older187-builds can safely ignore (decimal arithmetic) don't raise the version.186+v3, several sheets in v4, pivot tables in v5, tables in v6, linked187+sources in v7). Old files always load. Settings that older builds can188+safely ignore (decimal arithmetic) don't raise the version.188189 189190 ## Keeping it honest190191 @@ -39,7 +39,9 @@ it lags, and past a second it stalls.
3939 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |4040 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 8.7 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 120 fps pacing; per session, the terminal's cell buffers |4141 | Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 0.8 to 2.5 ms | | Results past the grid (the pivot shows #REF!) | Grouping the source rows |42+| Linked sources | Ten million rows of Parquet or SQLite: opened in 0.4 ms or 0.65 s, a frame scrolling them 0.9 ms at 200 x 60, a page anywhere 0.2 to 12 ms; SUM of a column 0.42 s (Parquet) or 1.4 s (SQLite), COUNTIFS 1.2 or 2.3 s, a pivot table 2.5 or 3.9 s, each in a few MB | Sorting ten million rows: 3.5 s (Parquet) or 19 s (SQLite), in the background | Functions that hold what they read, over more than `max-cells` cells: `#VALUE!`, saying why | Reading the column from the file; sorting |4243 | Following files | A log growing by 10,000 rows a second, formulas over it: each update (about 2,500 rows, four times a second) applied in 7 ms keeping every row, 1.7 to 2 ms keeping the last 1000, 14 ms the last 10,000; frames 2.5 to 3.6 ms at 200 x 60 | | Rows past `max-cells`: left out, with a note | Writing the rows and recalculating what reads them; a window rewrites its rows on every update |44+| Notebook streams (`TestStreamGridAtFrameSpeed`) | 10,000 rows arriving in batches of 250, the output's grid on screen: 2.3 ms a batch through to its frame at 200 x 60 (3.6 ms at worst) | Past 20,000 rows in one run: the grid is read again from the 10,000 the output keeps, about 60 ms once every 10,000 rows | | Reading the batch's rows into the grid and measuring their text; the output's NUON copied once a batch |4345 | Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 52 ms | Full recalculation of those 1000: 106 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |4446 4547 ## Sheet size@@ -373,6 +375,54 @@ change; a growing file is read a megabyte a poll at most, so a large
373375 file loads over several frames. Rows are held once, in the region's374376 cells; a window reads the rows it keeps back from them.375377 378+## Linked sources379+380+A [linked source](../files/sources.md) is read in place: its tab keeps381+at most 48 pages of 128 rows, and what formulas and pivot tables ask382+of it is worked out by streaming the file in the background, a few383+questions at a time, each answer kept until the file changes. None of384+it costs what the source's size does in memory. Measured on ten million385+rows of sales (an id, one of eight categories, an amount and a date;386+`stress.SalesParquet` and `stress.SalesSQLite`, 204 MB and 385 MB), one387+run each, the peak heap above what was live before:388+389+| Reading the file (`BenchmarkSource`, `internal/fileio`) | Parquet | SQLite |390+|---|---|---|391+| Open | 0.39 ms, 0.2 MB | 0.65 s (counting the rows) |392+| Scan one column | 0.35 s (28.5 M rows/s), 4.5 MB | 1.36 s (7.4 M rows/s), 3.4 MB |393+| A page of 60 rows deep in, in the source's order | 0.28 ms | 0.19 ms |394+| Sorting by a column, Z to A | 3.5 s, 113 MB | 19.1 s, in SQLite's own memory and files |395+| A page deep in, sorted | 12 ms (the rows are scattered) | 0.41 ms |396+| Filtering to one category (1,248,497 rows) | 1.26 s, 6.7 MB | 1.19 s |397+| A page deep in, filtered | 0.23 ms | 0.23 ms |398+399+| Answering a question (`BenchmarkSourceFormulas`, `internal/paged`) | Parquet | SQLite |400+|---|---|---|401+| `SUM`, `AVERAGE` of a column | 0.42 s, 3.8 MB | 1.42 s, 2.6 MB |402+| `COUNTIFS`, `SUMIFS` over two columns | 1.2 s, 8.3 MB | 2.3 s, 7.5 MB |403+| `SUMPRODUCT` of two columns | 0.44 s, 9 MB | 1.56 s, 7.7 MB |404+| `XLOOKUP` of the last id | 0.62 s, 5.9 MB | 1.28 s, 4.3 MB |405+| `MATCH` of the middle id | 0.31 s, 5.9 MB | 0.63 s, 3.8 MB |406+| `MEDIAN`, which holds every value: ten million cells are within `max-cells` | 1.15 s, 248 MB | 2.15 s, 247 MB |407+| A pivot table: SUM and COUNT of the amounts by category | 2.45 s, 5.9 MB | 3.9 s, 2.7 MB |408+409+SQLite's own memory, through modernc's libc, isn't in the Go heap.410+Functions over aligned ranges (the criteria functions, `SUMPRODUCT`)411+read their ranges a window of 2048 rows at a time, each column412+streamed by a goroutine of its own in batches of 4096 rows. A function413+that holds what it reads is refused past `max-cells` cells before it414+reads any, so `MEDIAN` of a hundred million values says so at once415+rather than holding them.416+417+Scrolling (`BenchmarkSourceScroll`, `internal/ui`) draws a frame in418+0.26 ms at 80 x 24 and 0.9 ms at 200 x 60 within the pages read, as a419+sheet's frame costs. A jump anywhere, as dragging the scrollbar makes,420+reads the pages it shows and a window either side, in the background421+while the frames go on showing `…`: 0.64 ms at 80 x 24 and 1.5 ms at422+200 x 60, the pages read and the frame drawn.423+The speed gate holds a frame on a 200,000-row source and a SUM over it424+to their baselines (`frame/source-200000x4`, `source/sum-200000`).425+376426 ## Undo377427 378428 Each step keeps the cells it changed as they were, in the form the@@ -33,8 +33,9 @@ between features: from each of a thousand seeds it builds a workbook
3333 of two sheets and a notebook tab, makes up to 40 edits drawn from the3434 seed (entries, pastes, fills, series, sorts, inserted and deleted rows3535 and columns, formats, borders, merges, moves, notes, rules, names,36-sheets, frozen panes, filters, notebook cells, and outputs sent to37-sheets, frozen, deleted and sent rows, with tables standing in for36+sheets, frozen panes, filters, tables, linked sources and their order,37+notebook cells, and outputs sent to sheets, frozen, deleted and sent38+rows, with tables standing in for3839 nushell's), and checks that undoing every step gives back the file and3940 the values it started with, redoing gives back the end, saving and4041 reopening gives the same file and values (a recalculation from@@ -107,6 +107,7 @@ there.
107107 | Entries failing validation | A dotted underline, and `Invalid:` with the rule on the context line |108108 | Values an array spilled | Italic, and `Spilled from B2` on the context line |109109 | Linked files' rows, following, paused or failing | Italic; `●`, `‖` or `!` beside the tab's name and on the context line, with the state in words (`Following`, `Paused`, the error) |110+| A linked source's tab, its scrollbar, rows on their way | `▦` before the tab's name and on the context line, `!` after it when the file can't be read; the scrollbar's thumb `┃` on a track of `│`; `…` in place of a row being read |110111 | A notebook's cells and outputs | Marks and words, listed in [Notebooks](../nushell/notebooks.md#reading-a-cell): the bar left of the active cell, a heavy box while it's edited, the prompt's and the box's words for its state, `×` before an error, `OUTPUT` in an output's grid, an output sent to a sheet in italic |111112 | A table's header row | Bold and underlined, when the table styles its header; `Table Sales, column Amount` on the context line |112113 | Pivot table results | Their headings (`SUM of Units`, `Grand Total`), and a note in words when an edit is refused |@@ -115,7 +116,7 @@ there.
115116 | Others in a shared workbook (012 serve) | Their pointer's cell double-underlined, their initial on its row's header, their name on the status line, `▘` in a cell they changed in the last 30 seconds, and `Changed by`, `is here` or their warnings in words on the context line |116117 | Agents in the workbook (live mode) | A `◆` for their initial on the row header and before their name on the status line, `agent` in Who's here; a `◇` in a cell their suggestion would set, what they suggest in words on the context line, and the count on the status line |117118 | Macro recording | `REC` beside the mode indicator |118-| Mode | The mode indicator's word (`READY`, `ENTER`, `POINT`, `MENU`, and on a notebook `NOTEBOOK`, `EDIT` or `OUTPUT`) |119+| Mode | The mode indicator's word (`READY`, `ENTER`, `POINT`, `MENU`, on a notebook `NOTEBOOK`, `EDIT` or `OUTPUT`, and on a linked source's tab `SOURCE`) |119120 | Search matches, traced cells | Reverse video, with a count or the list on the context line; dependents also bold |120121 | Evaluate formula | The part computed next underlined and bold, values in its place italic, and `Next` and `Value` in words |121122 | The copied range | A dashed underline |@@ -38,7 +38,8 @@ SQLite table). Formats that keep no column widths (all but XLSX and
3838 1-2-3) have their columns fitted to the header and the first 200 rows,3939 at most 30 characters wide, as are tables read from standard input. A file can also be followed instead, its table in a4040 linked region that takes in new rows as the file grows or is rewritten:41-see [Following files](following.md). A table can also come in on standard input, and go back41+see [Following files](following.md). A Parquet file or SQLite table too big to import links as a42+[source](sources.md), read in place. A table can also come in on standard input, and go back4243 out on standard output: see [Pipelines](../nushell/pipelines.md).4344 4445 File > Import asks where the data goes, as Sheets' Import location does@@ -98,6 +99,7 @@ line how many rows it left out.
9899 | [Excel files](excel.md) | What comes in from and goes out to `.xlsx` |99100 | [Saving](saving.md) | Atomic saves, overwrite checks, unsaved work under `012 serve` |100101 | [Following files](following.md) | Linked regions that follow a file as it grows or is rewritten, like `tail -f` |102+| [Linked sources](sources.md) | Parquet files and SQLite tables too big for any grid, read in place on a tab of their own, with formulas and pivot tables streaming over them |101103 | [The .012 format](format.md) | The JSON format, its versions and fields |102104 | [Scripts](scripts.md) | `012 get`, `set`, `recalc` and `export`: workbooks read and changed without the screen |103105 | [Diff and merge in git](git.md) | `012 diff` cell by cell, and git's diff and merge drivers for `.012` files |@@ -22,6 +22,10 @@ region recalculate as rows arrive.
2222 file. On a new, empty spreadsheet File > Import opens the file instead,2323 as it always does; link it with Data > Linked file there.2424 25+A linked file's rows are cells, up to `max-cells`. A Parquet file or a26+SQLite table too big for that links as a [source](sources.md) instead,27+read in place a page at a time.28+2529 For a SQLite database with several tables, a picker asks which one. Then2630 a question on the context line asks how many rows to keep:2731 @@ -7,7 +7,7 @@ sidebar_position: 4
77 88 `.012` files are JSON with one line per cell, keyed by address, so diffs99 read naturally and files merge reasonably in version control. This page10-is the format at version 5, the newest; [Versions](#versions) says what10+is the format at version 7, the newest; [Versions](#versions) says what1111 each version adds and what 012 promises about old files. A cell1212 without formatting is just what was typed; a formatted cell is a small1313 object:@@ -77,6 +77,7 @@ still load.
7777 | 4 | Several sheets, or a formula naming a sheet | `sheets`, a list; see [Several sheets](#several-sheets) |7878 | 5 | A pivot table | a sheet's `pivot`; see [Pivot tables](#pivot-tables) |7979 | 6 | A table | a sheet's `tables`; see [Tables](#tables) |80+| 7 | A linked source | a sheet whose region is `paged`; see [Linked sources](#linked-sources) |8081 8182 What stays the same from release to release:8283 @@ -241,6 +242,37 @@ earlier versions wrote (with `input`, the range it read as `$in`, and
241242 opens as a code cell of the workbook's notebook, its table sent where it242243 was ([Notebook sheets](../nushell/notebooks.md#notebook-sheets)).243244 245+## Linked sources246+247+A [linked source](sources.md)'s tab is a sheet with no cells whose one248+region, at `A1`, is `paged`: the file keeps what the source reads and249+how its tab orders the rows, never the rows.250+251+```json252+ {253+ "name": "trips",254+ "cells": {},255+ "regions": [256+ {"name":"trips","at":"A1","path":"trips.parquet","paged":true,"order":{"sort":[{"column":"C","desc":true}],"filter":[{"column":"B","condition":"eq","value":"card"}]}}257+ ]258+ }259+```260+261+- `name` is the table formulas read (`trips[fare]`, `nu.trips`); the262+ tab is named the same when linked, and keeps its own name when263+ renamed.264+- `path`, `format`, `table` and `query` are what a linked file's are265+ ([Regions](#regions)).266+- `paged` marks the region as a source, read in place.267+- `order` is how the tab sorts and filters the rows: `sort`, the columns268+ by letter, `desc` for Z to A, and `filter`, each a column by letter269+ with a `condition` and its `value`, as a filter's column has270+ ([Filter](../sheets/sort-filter.md#filter)). Formulas read the271+ source in its own order whatever the tab shows.272+273+Sources need version 7: a build without them would read the region as274+a linked file and load the whole source into cells.275+244276 ## Notebooks245277 246278 A notebook tab is a sheet with `"tab": "notebook"`, `"reactive": true`@@ -0,0 +1,181 @@
1+---2+title: "Linked sources"3+sidebar_position: 84+---5+6+# Linked sources7+8+A Parquet file, or a SQLite table or query, too big for any grid links9+as a source: a read-only table on a tab of its own, read in place10+rather than imported. The tab shows a window of the rows as it11+scrolls, with the row numbers and the scrollbar over every row, and12+formulas and pivot tables read the whole source by streaming it. A13+source of ten million rows opens at once and takes a few MB.14+15+16+17+18+## Linking a source19+20+21+22+**Data > Linked file > Link a source** lists the Parquet files and23+SQLite databases in the folder, or takes a path. For a database a24+second list asks which table or view to read, or **A query…** to read25+what a `SELECT` returns.26+27+The source gets a tab after the one shown, named after the file28+(`sales` for `sales.parquet`), with `▦` before its name. The source is29+a [table](../sheets/tables.md) of that name, which formulas on any sheet30+read as a table's:31+32+| Formula | Reads |33+|---|---|34+| `=SUM(sales[amount])` | a column's values, every row |35+| `=ROWS(sales)` | the rows under the header |36+| `=COUNTA(nu.sales)` | the whole table, header row included |37+| `=sales!C5` | one cell: row 1 is the header, so row 5 is the source's fourth row |38+39+Renaming the tab keeps the table's name. Undo takes the link back. A40+source's rows may run past the grid's last row, 1,048,576: its table's41+name reaches them all, where a reference typed as `C5` stops at the42+grid's edge.43+44+## The tab45+46+Row 1 is the header, the columns' names, and stays put as the rows47+scroll under it. The rows are numbered as the tab's rows, so the48+source's first row is row 2, and `sales!C2` is its first amount. The49+scrollbar at the right edge is as long as the share of the rows shown,50+and as far down as they are.51+52+| Key or mouse | Does |53+|---|---|54+| Arrows, Tab | Move the active cell |55+| Page Up, Page Down, the wheel | Scroll |56+| Home, End | The first or last column |57+| Ctrl+Up, Ctrl+Down | The first or last row |58+| Ctrl+Home, Ctrl+End | The first cell, the last |59+| A click on the scrollbar, or dragging its thumb | Jumps there |60+| Ctrl+G, F5 | Goes to a row, or a cell such as `C5000000` |61+| Ctrl+C | Copies the active cell |62+63+The rows come from the file a page (128 rows) at a time, in the64+background, the pages either side of the window read ahead, so65+scrolling finds them there. A row on its way shows `…`. The name box66+shows the active cell as the tab names it, and the formula bar its67+value; the context line says what the source is:68+`▦ sales.parquet 10,000,000 rows, 4 columns`. The mode is `SOURCE`,69+and typing, pasting or formatting there says the tab is read-only.70+71+## Sorting and filtering72+73+The tab's rows sort and filter as a sheet's do, without changing the74+file:75+76+- **Data > Sort sheet A to Z** (or **Z to A**) sorts by the active77+ column; blanks go last either way.78+- **Data > Create a filter** or **Filter by column** asks for a79+ condition on the active column: the conditions of a sheet's80+ [filter](../sheets/sort-filter.md#filter). There is no list of values81+ to tick: a source's are too many to list. Each column's condition82+ applies at once.83+- **Data > Remove filter** drops the conditions, and **Data > Linked84+ file > Original order** shows the source's own rows in its own order.85+86+Each is an undo step and is saved with the workbook. Formulas read the87+source in its own order, every row, as formulas read the rows a sheet's88+filter hides.89+90+Ordering reads the whole source once, in the background, while the91+context line says `sorting and filtering…`; the tab then shows the92+first of the rows. SQLite does the work: the sort and the conditions93+become SQL, the rows' order is kept in a temporary database beside the94+system's temporary files, and the database itself is never written. A95+Parquet file is streamed: a filter is one pass over the columns it96+tests, and a sort writes sorted runs to temporary files and merges97+them, so memory stays a few MB and up to about 100 MB while sorting,98+whatever the size. Either way a page anywhere in the ordered rows99+costs what one at the top does:100+101+```mermaid102+flowchart TD103+ order[a sort or a filter] -->|once, in the background| view[the rows' order: SQL into a temporary table, or streamed into temporary files]104+ scroll[the tab scrolls] --> want[the pages around the window]105+ view --> want106+ want -->|a page not read| read[read 128 rows at their positions]107+ read --> kept[kept, the least used let go past 48 pages]108+ kept --> frame[the next frame draws them]109+```110+111+In SQLite a text condition compares the text as SQLite writes it, so a112+number reads as `1e+20` where the sheet shows `1E+20`, a date held as113+text (`2026-09-26`) sorts and compares as that text, and letters beyond114+ASCII are compared by case.115+116+## Formulas over a source117+118+119+120+121+A function given a source's range is worked out in the background, by122+reading the source; its cell shows `Loading…` until then. The answer123+is kept until the file changes, so another formula asking the same, or124+the same formula recalculating, costs nothing.125+126+These read a source of any size in one pass, holding only what they127+compute:128+129+| Kind | Functions |130+|---|---|131+| Aggregates | SUM, AVERAGE, COUNT, COUNTA, MIN, MAX, PRODUCT |132+| Criteria | COUNTIF, COUNTIFS, SUMIF, SUMIFS, AVERAGEIF, AVERAGEIFS, COUNTBLANK, SUMPRODUCT |133+| Lookups | MATCH, XLOOKUP, VLOOKUP, HLOOKUP, INDEX, ROWS, COLUMNS |134+135+Any other function (MEDIAN, SORT, FILTER, UNIQUE, TEXTJOIN) holds what136+it reads, and is given a source's range as long as that holds no more137+than `max-cells` cells ([Configuration](../reference/config.md#max-cells));138+past it the formula shows `#VALUE!`, and the context line says what it139+would have read. So does an operator over a source's range140+(`=SUM(sales[amount]*2)`) and a function whose other arguments are141+ranges of a sheet or arrays: `SUMPRODUCT((sales[cat]="north")*sales[amount])`142+computes an array as long as the source, where143+`SUMIFS(sales[amount],sales[cat],"north")` streams.144+145+## Pivot tables146+147+**Data > Pivot table** on a source's tab makes a148+[pivot table](../sheets/pivots.md) over the whole source, and **Data >149+Frequency table (column stats)** one of the active column. The source's rows are150+grouped in the background, `Loading…` showing in the pivot table's151+first cell until they are, and again whenever the file changes. A152+filter of a pivot table over a source takes a condition, not values153+ticked.154+155+## When the file changes156+157+012 looks at each source's file four times a second, with the158+operating system's file information alone. Once the file has changed159+and held still for 0.3 s, the source is read again: the tab, and every160+formula and pivot table reading it. **Data > Linked file > Read the161+source again** does the same at once. A file that isn't there shows162+`!` after the tab's name and the reason on the context line, and the163+formulas reading it show `#REF!` with it.164+165+## Saving and sharing166+167+The workbook keeps the link, never the rows: the file's path relative168+to the workbook's folder, the table or query, and the tab's order169+([The .012 format](format.md#linked-sources)). Opening the workbook170+reads the sources again. A workbook from another computer asks before171+reading files outside its own folder, as for172+[linked files](following.md#files-from-elsewhere).173+174+`012 get`, `recalc` and `export` ([Scripts](scripts.md)) read the175+sources in the workbook's folder or below, to the end, before they176+answer. Under `012 serve` sources are read in the served folder, and in177+a [shared workbook](../terminal/ssh.md#sharing-a-workbook) the room reads them, as it follows linked178+files, while each participant scrolls a tab of their own.179+180+What sources cost, measured on ten million rows, is in181+[Bounds of support](../contributing/limits.md#linked-sources).Binary file.
Binary file.
Binary file.
Binary file.
Binary file.
Binary file.
Binary file.
@@ -357,6 +357,14 @@ keeps every row, up to `max-cells`. The rows arrive as the change
357357 stream does for a [followed file](../files/following.md), outside the358358 undo history.359359 360+A table's grid takes the rows as they arrive, under those it shows, so361+what you've done in it stays: the pointer, the selection, a filter362+(which covers the new rows) and the rows as sorted, the new ones under363+them. Its columns widen to fit longer rows, never narrowing, and wait364+while you're in the grid so nothing shifts under the pointer; a column365+you resized keeps its width. Past 20,000 rows the grid starts again366+from the rows the output keeps.367+360368 ## Names and $name361369 362370 A cell holds one statement or several, a line each (or apart by `;`),@@ -158,7 +158,7 @@ How new sheets and files without a locale of their own are typed and shown: deci
158158 159159 #### `max-cells`160160 161-The most cells an import keeps, and a paste or fill writes at once. Numbers and text take 20 to 60 bytes a cell and formulas about 750, so the default of ten million cells of data is 200 to 600 MB. Imports keep whole rows up to the budget and say how many they left out; larger pastes and fills are refused. The grid itself is 1,048,576 rows by 16,384 columns (A to XFD) whatever this is.161+The most cells an import keeps, and a paste or fill writes at once. Numbers and text take 20 to 60 bytes a cell and formulas about 750, so the default of ten million cells of data is 200 to 600 MB. Imports keep whole rows up to the budget and say how many they left out; larger pastes and fills are refused. A function that holds what it reads (MEDIAN, SORT) is given no more of a linked source. The grid itself is 1,048,576 rows by 16,384 columns (A to XFD) whatever this is.162162 163163 | | |164164 |---|---|@@ -152,6 +152,13 @@ With agents working in the session ([Live mode](../agents/live.md)):
152152 | Ctrl+Alt+Shift+0 to 9 | Run the macro with that shortcut ([Macros](../sheets/macros.md#running)) |153153 | Esc | Stop a macro while it runs (the mode indicator says CMD) |154154 155+## Linked sources156+157+On a [linked source's](../files/sources.md#the-tab) tab (`SOURCE`) the158+keys move and scroll as on a sheet, over every row of the source; Ctrl+G159+and F5 go to a row, the sort and filter keys order the rows the tab160+shows, and keys that would change cells say the tab is read-only.161+155162 ## Notebooks156163 157164 On a [notebook's](../nushell/notebooks.md#keys) tab, as in Jupyter:@@ -9,13 +9,23 @@ require (
99 )1010 1111 require (12+ github.com/andybalholm/brotli v1.1.1 // indirect13+ github.com/google/uuid v1.6.0 // indirect14+ github.com/klauspost/compress v1.17.9 // indirect15+ github.com/parquet-go/bitpack v1.0.0 // indirect16+ github.com/parquet-go/jsonlite v1.0.0 // indirect17+ github.com/parquet-go/parquet-go v0.32.0 // indirect18+ github.com/pierrec/lz4/v4 v4.1.21 // indirect1219 github.com/richardlehane/mscfb v1.0.7 // indirect1320 github.com/richardlehane/msoleps v1.0.6 // indirect1421 github.com/tiendc/go-deepcopy v1.7.2 // indirect22+ github.com/twpayne/go-geom v1.6.1 // indirect1523 github.com/xuri/efp v0.0.1 // indirect1624 github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 // indirect1725 golang.org/x/crypto v0.53.0 // indirect1826 golang.org/x/image v0.45.0 // indirect1927 golang.org/x/net v0.56.0 // indirect28+ golang.org/x/sys v0.47.0 // indirect2029 golang.org/x/text v0.41.0 // indirect30+ google.golang.org/protobuf v1.34.2 // indirect2131 )@@ -1,7 +1,21 @@
1+github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=2+github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=13 github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=24 github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=35 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=46 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=7+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=8+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=9+github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=10+github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=11+github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA=12+github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=13+github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=14+github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=15+github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM=16+github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=17+github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=18+github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=519 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=620 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=721 github.com/richardlehane/mscfb v1.0.7 h1:oeoiM0WE79vHwE8RpIYYvIAc8ajTH2mb6UZm55/+EB0=@@ -12,12 +26,15 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
1226 github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=1327 github.com/tiendc/go-deepcopy v1.7.2 h1:Ut2yYR7W9tWjTQitganoIue4UGxZwCcJy3orjrrIj44=1428 github.com/tiendc/go-deepcopy v1.7.2/go.mod h1:4bKjNC2r7boYOkD2IOuZpYjmlDdzjbpTRyCx+goBCJQ=29+github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4=30+github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028=1531 github.com/xuri/efp v0.0.1 h1:fws5Rv3myXyYni8uwj2qKjVaRP30PdjeYe2Y6FDsCL8=1632 github.com/xuri/efp v0.0.1/go.mod h1:ybY/Jr0T0GTCnYjKqmdwxyxn2BQf2RcQIIvex5QldPI=1733 github.com/xuri/excelize/v2 v2.11.0 h1:HxaEFl6sRN2+8J5a8HaKq+0M4FsjBGMnWWtjOCPSG88=1834 github.com/xuri/excelize/v2 v2.11.0/go.mod h1:jxFLbzaIwGQ5ufFNvYfUOHqXhfPaNmP14KWfmNz2Uak=1935 github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 h1:+C0TIdyyYmzadGaL/HBLbf3WdLgC29pgyhTjAT/0nuE=2036 github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9/go.mod h1:WwHg+CVyzlv/TX9xqBFXEZAuxOPxn2k1GNHwG41IIUQ=37+github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=2138 go.mitchellh.com/libghostty v0.0.0-20260920220152-31b65cdc24cf h1:PgfZ7mRcXk8n3+B+46oo9WxDi6xK1z6feOQEKNRWCBs=2239 go.mitchellh.com/libghostty v0.0.0-20260920220152-31b65cdc24cf/go.mod h1:sx8yDvnDrQC/PzsrFfYx7ROmH8my91nz7lOxQb6sbo0=2340 golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=@@ -28,9 +45,13 @@ golang.org/x/image v0.45.0 h1:FMb1nTbH5H9vF55SriQHgFw5GnNL9Jg6L25BwXKzhB0=
2845 golang.org/x/image v0.45.0/go.mod h1:n62x/7RqlwXDvGsSU4u6IUTUf6KghUZ9Bt7cG/T9Fx4=2946 golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=3047 golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=48+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=49+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=3150 golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=3251 golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=3352 golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=3453 golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=54+google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=55+google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=3556 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=3657 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=@@ -133,6 +133,8 @@ var notebookScreens = []screen{
133133 {name: "notebook-stream", setup: func(s *session) {134134 filesNotebook(s)135135 s.keys("<enter>", "log = tail -f app.log | lines | parse '{time} {level} {msg}'", "<esc>", "f")136+ // The head counts the rows the output has, so the grid is fitted137+ // to all three: only the first shows.136138 s.waitFor("● live, 3 rows")137139 }},138140 {name: "notebook-sent", setup: func(s *session) {@@ -0,0 +1,102 @@
1+package e2e2+3+import (4+ "path/filepath"5+ "strings"6+ "testing"7+8+ ghostty "go.mitchellh.com/libghostty"9+)10+11+// Golden screens for linked sources: two million trips on a tab of12+// their own, sorted by fare and scrolled halfway down by the scrollbar;13+// formulas over them on a sheet, one past max-cells saying why; the14+// condition a source's column is filtered by; and a source whose file15+// isn't there. Each on a light terminal too, the tab in the high-contrast16+// theme as well.17+18+func tripsFile(t *testing.T, dir string) { writeTrips(t, dir, tripRows) }19+20+// sortedTrips links trips.parquet, sorts it by fare Z to A and drags21+// the scrollbar halfway down.22+func sortedTrips(s *session) {23+ s.linkSource("trips.parquet")24+ s.waitFor("2,000,000 rows, 5 columns")25+ s.keys("<right>", "<right>")26+ s.keys("<ctrl+k>", "Sort sheet Z to A", "<enter>")27+ s.waitFor("sorted by fare Z to A")28+ s.waitFor("92.99")29+ bar := int(s.cols) - 130+ top, bottom := 7, int(s.rows)-231+ s.click(ghostty.MouseButtonLeft, bar, (top+bottom)/2)32+ s.waitFor("┃")33+ s.eventually("the rows halfway down", func() bool {34+ return strings.Contains(s.line(1), " 48") && !strings.Contains(s.screen(), " … ")35+ })36+}37+38+// tripFormulas reads the trips from a sheet: counts, sums, a lookup,39+// and a MEDIAN past max-cells (a million, in the screen's config),40+// the pointer on it.41+func tripFormulas(s *session) {42+ s.linkSource("trips.parquet")43+ s.keys("<ctrl+pgup>")44+ s.waitFor("Sheet1")45+ for _, row := range [][2]string{46+ {"Trips", "=ROWS(trips)"},47+ {"Fares", "=SUM(trips[fare])"},48+ {"By card", `=SUMIFS(trips[fare],trips[payment],"card")`},49+ {"Vouchers", `=COUNTIF(trips[payment],"voucher")`},50+ {"Longest", "=MAX(trips[miles])"},51+ {"Trip 1M", "=XLOOKUP(1000000,trips[trip],trips[fare])"},52+ {"Median", "=MEDIAN(trips[fare])"},53+ } {54+ s.keys(row[0], "<tab>", row[1], "<enter>")55+ }56+ s.waitFor("285714")57+ s.waitFor("#VALUE!")58+ s.keys("<up>", "<right>")59+ s.waitFor("MEDIAN would hold")60+}61+62+var sourceScreens = []screen{63+ {name: "source-tab", files: tripsFile, setup: sortedTrips},64+ {name: "source-formulas", opts: options{config: "max-cells = 1000000\n"}, files: tripsFile, setup: tripFormulas},65+ {name: "source-filter", files: tripsFile, setup: func(s *session) {66+ s.linkSource("trips.parquet")67+ s.waitFor("card")68+ s.keys("<right>")69+ s.keys("<ctrl+k>", "Filter by column", "<enter>")70+ s.waitFor("Filter B payment")71+ for range 7 {72+ s.keys("<down>")73+ }74+ s.keys("card")75+ s.waitFor("Text is exactly")76+ s.waitFor("card ")77+ }},78+ {name: "source-missing", files: func(t *testing.T, dir string) {79+ writeText(t, filepath.Join(dir, "rides.012"),80+ `{"version": 7, "sheets": [{"name": "Sheet1", "cells": {"A1": "=SUM(trips[fare])"}}, `+81+ `{"name": "trips", "cells": {}, "regions": [{"name": "trips", "at": "A1", "path": "trips.parquet", "paged": true}]}]}`)82+ }, args: []string{"rides.012"}, setup: func(s *session) {83+ s.waitFor("#REF!")84+ s.keys("<ctrl+pgdown>")85+ s.waitFor("! trips.parquet: the file isn't there")86+ }},87+}88+89+func init() {90+ for _, sc := range sourceScreens {91+ light := sc92+ light.name += "-light"93+ light.opts.light = true94+ screens = append(screens, sc, light)95+ if sc.name == "source-tab" {96+ hc := sc97+ hc.name += "-high-contrast"98+ hc.opts.config = highContrastConfig99+ screens = append(screens, hc)100+ }101+ }102+}@@ -0,0 +1,113 @@
1+package e2e2+3+import (4+ "fmt"5+ "os"6+ "path/filepath"7+ "strings"8+ "testing"9+ "time"10+11+ "github.com/parquet-go/parquet-go"12+)13+14+// trip is a row of the Parquet file the linked sources' tests read: a15+// ride, how it was paid, its fare, its miles and its day.16+type trip struct {17+ Trip int64 `parquet:"trip"`18+ Payment string `parquet:"payment,dict"`19+ Fare float64 `parquet:"fare"`20+ Miles float64 `parquet:"miles"`21+ Day int32 `parquet:"day,date"`22+}23+24+// payments are the ways trips are paid, in the order rows take them.25+var payments = []string{"card", "cash", "card", "app", "card", "cash", "voucher"}26+27+// tripAt is trip i, the same whenever it's asked for.28+func tripAt(i int) trip {29+ day := int32(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC).Unix()/86400) + int32(i%270)30+ return trip{Trip: int64(i + 1), Payment: payments[i%len(payments)],31+ Fare: float64((i*7919)%9000)/100 + 3, Miles: float64((i*104729)%400) / 10, Day: day}32+}33+34+// writeTrips writes rows trips to trips.parquet in dir.35+func writeTrips(t *testing.T, dir string, rows int) {36+ t.Helper()37+ f, err := os.Create(filepath.Join(dir, "trips.parquet"))38+ if err != nil {39+ t.Fatal(err)40+ }41+ defer f.Close()42+ w := parquet.NewGenericWriter[trip](f, parquet.MaxRowsPerRowGroup(1<<18))43+ batch := make([]trip, 1<<14)44+ for i := 0; i < rows; i += len(batch) {45+ n := min(len(batch), rows-i)46+ for j := range n {47+ batch[j] = tripAt(i + j)48+ }49+ if _, err := w.Write(batch[:n]); err != nil {50+ t.Fatal(err)51+ }52+ }53+ if err := w.Close(); err != nil {54+ t.Fatal(err)55+ }56+}57+58+// tripRows is how many rows the screens' source has: past the grid's59+// last row.60+const tripRows = 2_000_00061+62+// linkSource links name as a source through the palette, and waits for63+// its rows.64+func (s *session) linkSource(name string) {65+ s.t.Helper()66+ s.keys("<ctrl+k>", "Link a source")67+ s.waitFor("Link a source")68+ s.keys("<enter>")69+ s.waitFor("Type to filter, or a path")70+ s.keys(name, "<enter>")71+ s.waitFor("Linked " + name)72+ s.keys("<down>", "<up>") // the note gives way to the source's line73+ s.waitFor("▦ " + name)74+}75+76+func TestLinkedSource(t *testing.T) {77+ dir := t.TempDir()78+ writeTrips(t, dir, tripRows)79+ s := start(t, dir)80+ s.linkSource("trips.parquet")81+ s.waitFor("2,000,000 rows, 5 columns")82+ s.waitFor("card")83+84+ // The last row, numbered past the grid's last row.85+ s.keys("<ctrl+down>")86+ s.waitFor("2000001 ")87+88+ // Sorted by fare, Z to A, in the background.89+ s.keys("<ctrl+home>", "<right>", "<right>")90+ s.keys("<ctrl+k>", "Sort sheet Z to A", "<enter>")91+ s.waitFor("sorted by fare Z to A")92+ s.eventually("the top fare", func() bool { return strings.Contains(s.line(1), "92.99") })93+94+ // A formula on another sheet reads the whole source.95+ s.keys("<ctrl+pgup>")96+ s.waitFor("Sheet1")97+ s.keys("=COUNTIF(trips[payment],\"voucher\")", "<enter>")98+ s.waitFor(fmt.Sprint(tripRows / 7))99+100+ // Saved, the workbook keeps the link, and reads the source again.101+ s.keys("<ctrl+s>")102+ s.waitFor("Save as:")103+ s.keys("rides", "<enter>")104+ s.waitFor("rides.012")105+ data, err := os.ReadFile(filepath.Join(dir, "rides.012"))106+ if err != nil || !strings.Contains(string(data), `"paged":true`) || !strings.Contains(string(data), `"version": 7`) {107+ t.Fatalf("the file: %v\n%s", err, data)108+ }109+ s.keys("<ctrl+q>")110+ s.waitExit()111+ s = start(t, dir, "rides.012")112+ s.waitFor(fmt.Sprint(tripRows / 7))113+}@@ -41,6 +41,8 @@ var docScreens = []docScreen{
4141 {name: "live-suggestion", from: "live-suggestion", cols: 80, rows: 12},4242 {name: "live-review", from: "live-review", cols: 100, rows: 12},4343 {name: "live-ask", from: "live-ask", cols: 80, rows: 10},44+ {name: "source-tab", from: "source-tab", cols: 80, rows: 20},45+ {name: "source-formulas", from: "source-formulas", cols: 80, rows: 14},4446 }4547 4648 // stillsVersion changes when the drawing does, so every still is@@ -147,6 +147,13 @@ var symbols = map[rune]func(y symbol){
147147 '✓': func(y symbol) { y.line(.12, .14, .54, .4, .8, .88, .2) },148148 '✗': func(y symbol) { y.line(.12, .2, .2, .8, .8); y.line(.12, .8, .2, .2, .8) },149149 '❯': func(y symbol) { y.line(.16, .3, .16, .72, .5, .3, .84) },150+ '▦': func(y symbol) {151+ y.box(.1, .1, .9, .9, .08)152+ y.line(.06, .1, .37, .9, .37)153+ y.line(.06, .1, .63, .9, .63)154+ y.line(.06, .37, .1, .37, .9)155+ y.line(.06, .63, .1, .63, .9)156+ },150157 '⧉': func(y symbol) { y.box(.08, .08, .64, .64, .08); y.box(.36, .36, .92, .92, .08) },151158 '⎘': func(y symbol) { y.box(.16, .16, .84, .94, .08); y.rect(.34, .04, .66, .26) },152159 '↻': reload,@@ -14,5 +14,5 @@
1414 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as JSON <span style="color:var(--vt-palette-0)">File</span> <span style="color:var(--vt-palette-0)">›</span> <span style="color:var(--vt-palette-0)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as NUON <span style="color:var(--vt-palette-0)">File</span> <span style="color:var(--vt-palette-0)">›</span> <span style="color:var(--vt-palette-0)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1616 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as HTML <span style="color:var(--vt-palette-0)">File</span> <span style="color:var(--vt-palette-0)">›</span> <span style="color:var(--vt-palette-0)">Download</span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-0)">228</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-0)">230</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 Start a new, empty sheet <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Up/Down</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">move</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Enter</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">run</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Esc</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">close</span></pre>\ No newline at end of file@@ -14,5 +14,5 @@
1414 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as JSON <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as NUON <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1616 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-8)">│</span> Download as HTML <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 Start a new, empty sheet <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Up/Down</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">move</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Enter</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">run</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Esc</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">close</span></pre>\ No newline at end of file@@ -0,0 +1,14 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">READY</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">B7</span><span style="background:var(--vt-palette-7)"> </span> =MEDIAN(trips[fare])3+<span style="color:var(--vt-palette-1)">#VALUE!</span> <span style="color:var(--vt-palette-0)">MEDIAN</span> <span style="color:var(--vt-palette-0)">would</span> <span style="color:var(--vt-palette-0)">hold</span> <span style="color:var(--vt-palette-0)">2,000,000</span> <span style="color:var(--vt-palette-0)">cells</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">trips,</span> <span style="color:var(--vt-palette-0)">past</span> <span style="color:var(--vt-palette-0)">max-cells</span> <span style="color:var(--vt-palette-0)">(1,000,000):</span>4+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A</span><span style="background:var(--vt-palette-7)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">C</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">D</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">E</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">F</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">G</span><span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span> Trips 20000006+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">2</span><span style="background:var(--vt-palette-7)"> </span> Fares 959900007+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">3</span><span style="background:var(--vt-palette-7)"> </span> By card 411385638+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">4</span><span style="background:var(--vt-palette-7)"> </span> Vouchers 2857149+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">5</span><span style="background:var(--vt-palette-7)"> </span> Longest 39.910+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">6</span><span style="background:var(--vt-palette-7)"> </span> Trip 1M 3.8111+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">7</span><span style="background:var(--vt-palette-6)"> </span> Median <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);text-decoration:underline wavy var(--vt-palette-1)">#VALUE!</span><span style="background:var(--vt-palette-6)"> </span>12+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">8</span><span style="background:var(--vt-palette-7)"> </span>13+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">9</span><span style="background:var(--vt-palette-7)"> </span>14+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">Sheet1</span><span style="background:var(--vt-palette-6)"> </span> ▦ trips <span style="color:var(--vt-palette-0)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-0)">modified</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+K</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">search</span></pre>\ No newline at end of file@@ -0,0 +1,14 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">READY</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">B7</span><span style="background:var(--vt-palette-8)"> </span> =MEDIAN(trips[fare])3+<span style="color:var(--vt-palette-1)">#VALUE!</span> <span style="color:var(--vt-palette-8)">MEDIAN</span> <span style="color:var(--vt-palette-8)">would</span> <span style="color:var(--vt-palette-8)">hold</span> <span style="color:var(--vt-palette-8)">2,000,000</span> <span style="color:var(--vt-palette-8)">cells</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">trips,</span> <span style="color:var(--vt-palette-8)">past</span> <span style="color:var(--vt-palette-8)">max-cells</span> <span style="color:var(--vt-palette-8)">(1,000,000):</span>4+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A</span><span style="background:var(--vt-palette-8)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">C</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">D</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">E</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">F</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">G</span><span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span> Trips 20000006+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">2</span><span style="background:var(--vt-palette-8)"> </span> Fares 959900007+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">3</span><span style="background:var(--vt-palette-8)"> </span> By card 411385638+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">4</span><span style="background:var(--vt-palette-8)"> </span> Vouchers 2857149+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">5</span><span style="background:var(--vt-palette-8)"> </span> Longest 39.910+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">6</span><span style="background:var(--vt-palette-8)"> </span> Trip 1M 3.8111+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">7</span><span style="background:var(--vt-palette-6)"> </span> Median <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);text-decoration:underline wavy var(--vt-palette-1)">#VALUE!</span><span style="background:var(--vt-palette-6)"> </span>12+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">8</span><span style="background:var(--vt-palette-8)"> </span>13+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">9</span><span style="background:var(--vt-palette-8)"> </span>14+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">Sheet1</span><span style="background:var(--vt-palette-6)"> </span> ▦ trips <span style="color:var(--vt-palette-8)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-8)">modified</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+K</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">search</span></pre>\ No newline at end of file@@ -0,0 +1,20 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">C1912502</span><span style="background:var(--vt-palette-7)"> </span> 483+<span style="color:var(--vt-palette-0)">▦</span> <span style="font-weight:bold">trips.parquet</span> <span style="color:var(--vt-palette-0)">2,000,000</span> <span style="color:var(--vt-palette-0)">rows,</span> <span style="color:var(--vt-palette-0)">5</span> <span style="color:var(--vt-palette-0)">columns;</span> <span style="color:var(--vt-palette-0)">sorted</span> <span style="color:var(--vt-palette-0)">by</span> <span style="color:var(--vt-palette-0)">fare</span> <span style="color:var(--vt-palette-0)">Z</span> <span style="color:var(--vt-palette-0)">to</span> <span style="color:var(--vt-palette-0)">A</span>4+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">B</span><span style="background:var(--vt-palette-7)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">C</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">D</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">E</span><span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">payment</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">fare</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">miles</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">day</span><span style="text-decoration:underline"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">1912502</span><span style="background:var(--vt-palette-6)"> </span>1912501 card <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">48</span><span style="background:var(--vt-palette-6)"> </span> 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>8+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1921502</span><span style="background:var(--vt-palette-7)"> </span>1921501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1930502</span><span style="background:var(--vt-palette-7)"> </span>1930501 cash 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1939502</span><span style="background:var(--vt-palette-7)"> </span>1939501 app 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1948502</span><span style="background:var(--vt-palette-7)"> </span>1948501 cash 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1957502</span><span style="background:var(--vt-palette-7)"> </span>1957501 voucher 48 30 1/1/2026 <span style="font-weight:bold">┃</span>13+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1966502</span><span style="background:var(--vt-palette-7)"> </span>1966501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1975502</span><span style="background:var(--vt-palette-7)"> </span>1975501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1984502</span><span style="background:var(--vt-palette-7)"> </span>1984501 card 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1993502</span><span style="background:var(--vt-palette-7)"> </span>1993501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">8823</span><span style="background:var(--vt-palette-7)"> </span> 8822 cash 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>18+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">17823</span><span style="background:var(--vt-palette-7)"> </span> 17822 voucher 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">26823</span><span style="background:var(--vt-palette-7)"> </span> 26822 card 47.99 10.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>20+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-0)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-0)">modified</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+K</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">search</span></pre>\ No newline at end of file@@ -0,0 +1,20 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">C1912502</span><span style="background:var(--vt-palette-8)"> </span> 483+<span style="color:var(--vt-palette-8)">▦</span> <span style="font-weight:bold">trips.parquet</span> <span style="color:var(--vt-palette-8)">2,000,000</span> <span style="color:var(--vt-palette-8)">rows,</span> <span style="color:var(--vt-palette-8)">5</span> <span style="color:var(--vt-palette-8)">columns;</span> <span style="color:var(--vt-palette-8)">sorted</span> <span style="color:var(--vt-palette-8)">by</span> <span style="color:var(--vt-palette-8)">fare</span> <span style="color:var(--vt-palette-8)">Z</span> <span style="color:var(--vt-palette-8)">to</span> <span style="color:var(--vt-palette-8)">A</span>4+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">B</span><span style="background:var(--vt-palette-8)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">C</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">D</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">E</span><span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">payment</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">fare</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">miles</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">day</span><span style="text-decoration:underline"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">1912502</span><span style="background:var(--vt-palette-6)"> </span>1912501 card <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">48</span><span style="background:var(--vt-palette-6)"> </span> 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>8+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1921502</span><span style="background:var(--vt-palette-8)"> </span>1921501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1930502</span><span style="background:var(--vt-palette-8)"> </span>1930501 cash 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1939502</span><span style="background:var(--vt-palette-8)"> </span>1939501 app 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1948502</span><span style="background:var(--vt-palette-8)"> </span>1948501 cash 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1957502</span><span style="background:var(--vt-palette-8)"> </span>1957501 voucher 48 30 1/1/2026 <span style="font-weight:bold">┃</span>13+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1966502</span><span style="background:var(--vt-palette-8)"> </span>1966501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1975502</span><span style="background:var(--vt-palette-8)"> </span>1975501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1984502</span><span style="background:var(--vt-palette-8)"> </span>1984501 card 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1993502</span><span style="background:var(--vt-palette-8)"> </span>1993501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">8823</span><span style="background:var(--vt-palette-8)"> </span> 8822 cash 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>18+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">17823</span><span style="background:var(--vt-palette-8)"> </span> 17822 voucher 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">26823</span><span style="background:var(--vt-palette-8)"> </span> 26822 card 47.99 10.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>20+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-8)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-8)">modified</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+K</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">search</span></pre>\ No newline at end of file@@ -10,7 +10,7 @@
1010 <span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-3);font-weight:bold">S</span>ort range Z to <span style="color:var(--vt-palette-3);font-weight:bold">A</span> <span style="color:var(--vt-palette-8)">│</span>1111 <span style="color:var(--vt-palette-8)">│</span> Sort <span style="color:var(--vt-palette-3);font-weight:bold">s</span>heet <span style="color:var(--vt-palette-3);font-weight:bold">A</span> to Z <span style="color:var(--vt-palette-8)">│</span>1212 <span style="color:var(--vt-palette-8)">│</span> Sort <span style="color:var(--vt-palette-3);font-weight:bold">s</span>heet Z to <span style="color:var(--vt-palette-3);font-weight:bold">A</span> <span style="color:var(--vt-palette-8)">│</span>13+<span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-8)">Read</span> <span style="color:var(--vt-palette-8)">the</span> <span style="color:var(--vt-palette-8)">source</span> <span style="color:var(--vt-palette-8)">again</span> <span style="color:var(--vt-palette-8)">│</span>1314 <span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-8)">Ask</span> <span style="color:var(--vt-palette-8)">JEV</span> <span style="color:var(--vt-palette-8)">again</span> <span style="color:var(--vt-palette-8)">│</span>14-<span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-3);font-weight:bold">S</span>ort r<span style="color:var(--vt-palette-3);font-weight:bold">a</span>nge <span style="color:var(--vt-palette-8)">│</span>15-<span style="color:var(--vt-palette-8)">└──────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">66</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>15+<span style="color:var(--vt-palette-8)">└──────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">68</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1616 Save the sheet u… <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Up/Down</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">move</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Enter</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">run</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Esc</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">close</span></pre>\ No newline at end of file@@ -14,7 +14,7 @@
1414 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">10</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">│</span> Reset <span style="color:var(--vt-palette-4);font-weight:bold">col</span>umn width <span style="color:var(--vt-palette-0)">Format</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">11</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">│</span> Insert <span style="color:var(--vt-palette-4);font-weight:bold">col</span>umns right <span style="color:var(--vt-palette-0)">Insert</span> <span style="color:var(--vt-palette-8)">│</span>1616 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">12</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">│</span> Delete rows or <span style="color:var(--vt-palette-4);font-weight:bold">col</span>umns <span style="color:var(--vt-palette-0)">Edit</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+Alt+-</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">13</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-0)">48</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">13</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-0)">48</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">14</span><span style="background:var(--vt-palette-7)"> </span>1919 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">15</span><span style="background:var(--vt-palette-7)"> </span>2020 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">16</span><span style="background:var(--vt-palette-7)"> </span>@@ -14,7 +14,7 @@
1414 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Reset <span style="color:var(--vt-palette-3);font-weight:bold">col</span>umn width <span style="color:var(--vt-palette-8)">Format</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">11</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Insert <span style="color:var(--vt-palette-3);font-weight:bold">col</span>umns right <span style="color:var(--vt-palette-8)">Insert</span> <span style="color:var(--vt-palette-8)">│</span>1616 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Delete rows or <span style="color:var(--vt-palette-3);font-weight:bold">col</span>umns <span style="color:var(--vt-palette-8)">Edit</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+Alt+-</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">48</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">48</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">14</span><span style="background:var(--vt-palette-8)"> </span>1919 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">15</span><span style="background:var(--vt-palette-8)"> </span>2020 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">16</span><span style="background:var(--vt-palette-8)"> </span>@@ -13,8 +13,8 @@
1313 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">9</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-3);font-weight:bold">S</span>av<span style="color:var(--vt-palette-3);font-weight:bold">e</span> <span style="color:var(--vt-palette-8)">Fi</span><span style="color:var(--vt-palette-3);font-weight:bold">l</span><span style="color:var(--vt-palette-8)">e</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+S</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span>1414 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-3);font-weight:bold">S</span>av<span style="color:var(--vt-palette-3);font-weight:bold">e</span> as <span style="color:var(--vt-palette-8)">Fi</span><span style="color:var(--vt-palette-3);font-weight:bold">l</span><span style="color:var(--vt-palette-8)">e</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">11</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> <span style="color:var(--vt-palette-8)">Code</span> <span style="color:var(--vt-palette-8)">Data</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Shell</span> <span style="color:var(--vt-palette-8)">y</span> <span style="color:var(--vt-palette-8)">│</span>16-<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> In<span style="color:var(--vt-palette-3);font-weight:bold">se</span>rt rows be<span style="color:var(--vt-palette-3);font-weight:bold">l</span>ow <span style="color:var(--vt-palette-8)">Insert</span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">61</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>16+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Link a <span style="color:var(--vt-palette-3);font-weight:bold">s</span>ourc<span style="color:var(--vt-palette-3);font-weight:bold">e</span> <span style="color:var(--vt-palette-8)">Data</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-3);font-weight:bold">L</span><span style="color:var(--vt-palette-8)">inked</span> <span style="color:var(--vt-palette-8)">file</span> <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└────────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">63</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">14</span><span style="background:var(--vt-palette-8)"> </span>1919 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">15</span><span style="background:var(--vt-palette-8)"> </span>2020 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">16</span><span style="background:var(--vt-palette-8)"> </span>@@ -14,7 +14,7 @@
1414 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Download as JSON <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1515 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">11</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Download as NUON <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>1616 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">│</span> Download as HTML <span style="color:var(--vt-palette-8)">File</span> <span style="color:var(--vt-palette-8)">›</span> <span style="color:var(--vt-palette-8)">Download</span> <span style="color:var(--vt-palette-8)">│</span>17-<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">228</span> <span style="color:var(--vt-palette-8)">─┘</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">└───────────────────────────────────────────────────────────────</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">230</span> <span style="color:var(--vt-palette-8)">─┘</span>1818 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">14</span><span style="background:var(--vt-palette-8)"> </span>1919 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">15</span><span style="background:var(--vt-palette-8)"> </span>2020 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">16</span><span style="background:var(--vt-palette-8)"> </span>@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">FILTER</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">B2</span><span style="background:var(--vt-palette-7)"> </span> card3+4+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A</span><span style="background:var(--vt-palette-7)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">C</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">D</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">E</span><span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-8)">┌─</span> <span style="font-weight:bold">Filter</span> <span style="font-weight:bold">B</span> <span style="font-weight:bold">payment</span> <span style="color:var(--vt-palette-8)">────────────────┐</span>6+<span style="color:var(--vt-palette-8)">───────────────│</span> <span style="color:var(--vt-palette-0)">If</span> <span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">‹</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">Text</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">is</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">exactly</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">›</span> card <span style="color:var(--vt-palette-8)">│───────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">2</span><span style="background:var(--vt-palette-6)"> </span> 1 <span style="color:var(--vt-palette-8)">└────────────────────────────────────┘</span> <span style="font-weight:bold">┃</span>8+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">3</span><span style="background:var(--vt-palette-7)"> </span> 2 cash 82.19 32.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">4</span><span style="background:var(--vt-palette-7)"> </span> 3 card 71.38 25.8 1/3/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">5</span><span style="background:var(--vt-palette-7)"> </span> 4 app 60.57 18.7 1/4/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">6</span><span style="background:var(--vt-palette-7)"> </span> 5 card 49.76 11.6 1/5/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">7</span><span style="background:var(--vt-palette-7)"> </span> 6 cash 38.95 4.5 1/6/2026 <span style="color:var(--vt-palette-8)">│</span>13+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">8</span><span style="background:var(--vt-palette-7)"> </span> 7 voucher 28.14 37.4 1/7/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">9</span><span style="background:var(--vt-palette-7)"> </span> 8 card 17.33 30.3 1/8/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">10</span><span style="background:var(--vt-palette-7)"> </span> 9 cash 6.52 23.2 1/9/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">11</span><span style="background:var(--vt-palette-7)"> </span> 10 card 85.71 16.1 1/10/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">12</span><span style="background:var(--vt-palette-7)"> </span> 11 app 74.9 9 1/11/2026 <span style="color:var(--vt-palette-8)">│</span>18+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">13</span><span style="background:var(--vt-palette-7)"> </span> 12 card 64.09 1.9 1/12/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">14</span><span style="background:var(--vt-palette-7)"> </span> 13 cash 53.28 34.8 1/13/2026 <span style="color:var(--vt-palette-8)">│</span>20+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">15</span><span style="background:var(--vt-palette-7)"> </span> 14 voucher 42.47 27.7 1/14/2026 <span style="color:var(--vt-palette-8)">│</span>21+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">16</span><span style="background:var(--vt-palette-7)"> </span> 15 card 31.66 20.6 1/15/2026 <span style="color:var(--vt-palette-8)">│</span>22+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">17</span><span style="background:var(--vt-palette-7)"> </span> 16 cash 20.85 13.5 1/16/2026 <span style="color:var(--vt-palette-8)">│</span>23+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">18</span><span style="background:var(--vt-palette-7)"> </span> 17 card 10.04 6.4 1/17/2026 <span style="color:var(--vt-palette-8)">│</span>24+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">19</span><span style="background:var(--vt-palette-7)"> </span> 18 app 89.23 39.3 1/18/2026 <span style="color:var(--vt-palette-8)">│</span>25+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">20</span><span style="background:var(--vt-palette-7)"> </span> 19 card 78.42 32.2 1/19/2026 <span style="color:var(--vt-palette-8)">│</span>26+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">21</span><span style="background:var(--vt-palette-7)"> </span> 20 cash 67.61 25.1 1/20/2026 <span style="color:var(--vt-palette-8)">│</span>27+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">22</span><span style="background:var(--vt-palette-7)"> </span> 21 voucher 56.8 18 1/21/2026 <span style="color:var(--vt-palette-8)">│</span>28+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">23</span><span style="background:var(--vt-palette-7)"> </span> 22 card 45.99 10.9 1/22/2026 <span style="color:var(--vt-palette-8)">│</span>29+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">24</span><span style="background:var(--vt-palette-7)"> </span> 23 cash 35.18 3.8 1/23/2026 <span style="color:var(--vt-palette-8)">│</span>30+Rows must meet the condition <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Up/Down</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">condition</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Enter</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">apply</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Esc</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">cancel</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">FILTER</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">B2</span><span style="background:var(--vt-palette-8)"> </span> card3+4+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A</span><span style="background:var(--vt-palette-8)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">C</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">D</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">E</span><span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-8)">┌─</span> <span style="font-weight:bold">Filter</span> <span style="font-weight:bold">B</span> <span style="font-weight:bold">payment</span> <span style="color:var(--vt-palette-8)">────────────────┐</span>6+<span style="color:var(--vt-palette-8)">───────────────│</span> <span style="color:var(--vt-palette-8)">If</span> <span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">‹</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">Text</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">is</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">exactly</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">›</span> card <span style="color:var(--vt-palette-8)">│───────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">2</span><span style="background:var(--vt-palette-6)"> </span> 1 <span style="color:var(--vt-palette-8)">└────────────────────────────────────┘</span> <span style="font-weight:bold">┃</span>8+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">3</span><span style="background:var(--vt-palette-8)"> </span> 2 cash 82.19 32.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">4</span><span style="background:var(--vt-palette-8)"> </span> 3 card 71.38 25.8 1/3/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">5</span><span style="background:var(--vt-palette-8)"> </span> 4 app 60.57 18.7 1/4/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">6</span><span style="background:var(--vt-palette-8)"> </span> 5 card 49.76 11.6 1/5/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">7</span><span style="background:var(--vt-palette-8)"> </span> 6 cash 38.95 4.5 1/6/2026 <span style="color:var(--vt-palette-8)">│</span>13+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">8</span><span style="background:var(--vt-palette-8)"> </span> 7 voucher 28.14 37.4 1/7/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">9</span><span style="background:var(--vt-palette-8)"> </span> 8 card 17.33 30.3 1/8/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">10</span><span style="background:var(--vt-palette-8)"> </span> 9 cash 6.52 23.2 1/9/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">11</span><span style="background:var(--vt-palette-8)"> </span> 10 card 85.71 16.1 1/10/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span> 11 app 74.9 9 1/11/2026 <span style="color:var(--vt-palette-8)">│</span>18+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span> 12 card 64.09 1.9 1/12/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">14</span><span style="background:var(--vt-palette-8)"> </span> 13 cash 53.28 34.8 1/13/2026 <span style="color:var(--vt-palette-8)">│</span>20+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">15</span><span style="background:var(--vt-palette-8)"> </span> 14 voucher 42.47 27.7 1/14/2026 <span style="color:var(--vt-palette-8)">│</span>21+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">16</span><span style="background:var(--vt-palette-8)"> </span> 15 card 31.66 20.6 1/15/2026 <span style="color:var(--vt-palette-8)">│</span>22+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">17</span><span style="background:var(--vt-palette-8)"> </span> 16 cash 20.85 13.5 1/16/2026 <span style="color:var(--vt-palette-8)">│</span>23+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">18</span><span style="background:var(--vt-palette-8)"> </span> 17 card 10.04 6.4 1/17/2026 <span style="color:var(--vt-palette-8)">│</span>24+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">19</span><span style="background:var(--vt-palette-8)"> </span> 18 app 89.23 39.3 1/18/2026 <span style="color:var(--vt-palette-8)">│</span>25+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">20</span><span style="background:var(--vt-palette-8)"> </span> 19 card 78.42 32.2 1/19/2026 <span style="color:var(--vt-palette-8)">│</span>26+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">21</span><span style="background:var(--vt-palette-8)"> </span> 20 cash 67.61 25.1 1/20/2026 <span style="color:var(--vt-palette-8)">│</span>27+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">22</span><span style="background:var(--vt-palette-8)"> </span> 21 voucher 56.8 18 1/21/2026 <span style="color:var(--vt-palette-8)">│</span>28+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">23</span><span style="background:var(--vt-palette-8)"> </span> 22 card 45.99 10.9 1/22/2026 <span style="color:var(--vt-palette-8)">│</span>29+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">24</span><span style="background:var(--vt-palette-8)"> </span> 23 cash 35.18 3.8 1/23/2026 <span style="color:var(--vt-palette-8)">│</span>30+Rows must meet the condition <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Up/Down</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">condition</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Enter</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">apply</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Esc</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">cancel</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">READY</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">B7</span><span style="background:var(--vt-palette-7)"> </span> =MEDIAN(trips[fare])3+<span style="color:var(--vt-palette-1)">#VALUE!</span> <span style="color:var(--vt-palette-0)">MEDIAN</span> <span style="color:var(--vt-palette-0)">would</span> <span style="color:var(--vt-palette-0)">hold</span> <span style="color:var(--vt-palette-0)">2,000,000</span> <span style="color:var(--vt-palette-0)">cells</span> <span style="color:var(--vt-palette-0)">of</span> <span style="color:var(--vt-palette-0)">trips,</span> <span style="color:var(--vt-palette-0)">past</span> <span style="color:var(--vt-palette-0)">max-cells</span> <span style="color:var(--vt-palette-0)">(1,000,000):</span> <span style="color:var(--vt-palette-0)">SUMIFS,</span> <span style="color:var(--vt-palette-0)">XLOOKUP</span> <span style="color:var(--vt-palette-0)">and</span>4+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A</span><span style="background:var(--vt-palette-7)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">C</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">D</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">E</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">F</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">G</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">H</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">I</span><span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span> Trips 20000006+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">2</span><span style="background:var(--vt-palette-7)"> </span> Fares 959900007+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">3</span><span style="background:var(--vt-palette-7)"> </span> By card 411385638+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">4</span><span style="background:var(--vt-palette-7)"> </span> Vouchers 2857149+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">5</span><span style="background:var(--vt-palette-7)"> </span> Longest 39.910+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">6</span><span style="background:var(--vt-palette-7)"> </span> Trip 1M 3.8111+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">7</span><span style="background:var(--vt-palette-6)"> </span> Median <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);text-decoration:underline wavy var(--vt-palette-1)">#VALUE!</span><span style="background:var(--vt-palette-6)"> </span>12+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">8</span><span style="background:var(--vt-palette-7)"> </span>13+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">9</span><span style="background:var(--vt-palette-7)"> </span>14+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">10</span><span style="background:var(--vt-palette-7)"> </span>15+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">11</span><span style="background:var(--vt-palette-7)"> </span>16+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">12</span><span style="background:var(--vt-palette-7)"> </span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">13</span><span style="background:var(--vt-palette-7)"> </span>18+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">14</span><span style="background:var(--vt-palette-7)"> </span>19+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">15</span><span style="background:var(--vt-palette-7)"> </span>20+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">16</span><span style="background:var(--vt-palette-7)"> </span>21+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">17</span><span style="background:var(--vt-palette-7)"> </span>22+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">18</span><span style="background:var(--vt-palette-7)"> </span>23+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">19</span><span style="background:var(--vt-palette-7)"> </span>24+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">20</span><span style="background:var(--vt-palette-7)"> </span>25+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">21</span><span style="background:var(--vt-palette-7)"> </span>26+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">22</span><span style="background:var(--vt-palette-7)"> </span>27+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">23</span><span style="background:var(--vt-palette-7)"> </span>28+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">24</span><span style="background:var(--vt-palette-7)"> </span>29+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">25</span><span style="background:var(--vt-palette-7)"> </span>30+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">Sheet1</span><span style="background:var(--vt-palette-6)"> </span> ▦ trips <span style="color:var(--vt-palette-0)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-0)">modified</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+K</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">search</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+/</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">shortcuts</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">F10</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">READY</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">B7</span><span style="background:var(--vt-palette-8)"> </span> =MEDIAN(trips[fare])3+<span style="color:var(--vt-palette-1)">#VALUE!</span> <span style="color:var(--vt-palette-8)">MEDIAN</span> <span style="color:var(--vt-palette-8)">would</span> <span style="color:var(--vt-palette-8)">hold</span> <span style="color:var(--vt-palette-8)">2,000,000</span> <span style="color:var(--vt-palette-8)">cells</span> <span style="color:var(--vt-palette-8)">of</span> <span style="color:var(--vt-palette-8)">trips,</span> <span style="color:var(--vt-palette-8)">past</span> <span style="color:var(--vt-palette-8)">max-cells</span> <span style="color:var(--vt-palette-8)">(1,000,000):</span> <span style="color:var(--vt-palette-8)">SUMIFS,</span> <span style="color:var(--vt-palette-8)">XLOOKUP</span> <span style="color:var(--vt-palette-8)">and</span>4+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A</span><span style="background:var(--vt-palette-8)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">B</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">C</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">D</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">E</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">F</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">G</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">H</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">I</span><span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span> Trips 20000006+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">2</span><span style="background:var(--vt-palette-8)"> </span> Fares 959900007+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">3</span><span style="background:var(--vt-palette-8)"> </span> By card 411385638+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">4</span><span style="background:var(--vt-palette-8)"> </span> Vouchers 2857149+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">5</span><span style="background:var(--vt-palette-8)"> </span> Longest 39.910+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">6</span><span style="background:var(--vt-palette-8)"> </span> Trip 1M 3.8111+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">7</span><span style="background:var(--vt-palette-6)"> </span> Median <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);text-decoration:underline wavy var(--vt-palette-1)">#VALUE!</span><span style="background:var(--vt-palette-6)"> </span>12+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">8</span><span style="background:var(--vt-palette-8)"> </span>13+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">9</span><span style="background:var(--vt-palette-8)"> </span>14+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">10</span><span style="background:var(--vt-palette-8)"> </span>15+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">11</span><span style="background:var(--vt-palette-8)"> </span>16+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">12</span><span style="background:var(--vt-palette-8)"> </span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">13</span><span style="background:var(--vt-palette-8)"> </span>18+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">14</span><span style="background:var(--vt-palette-8)"> </span>19+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">15</span><span style="background:var(--vt-palette-8)"> </span>20+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">16</span><span style="background:var(--vt-palette-8)"> </span>21+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">17</span><span style="background:var(--vt-palette-8)"> </span>22+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">18</span><span style="background:var(--vt-palette-8)"> </span>23+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">19</span><span style="background:var(--vt-palette-8)"> </span>24+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">20</span><span style="background:var(--vt-palette-8)"> </span>25+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">21</span><span style="background:var(--vt-palette-8)"> </span>26+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">22</span><span style="background:var(--vt-palette-8)"> </span>27+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">23</span><span style="background:var(--vt-palette-8)"> </span>28+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">24</span><span style="background:var(--vt-palette-8)"> </span>29+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">25</span><span style="background:var(--vt-palette-8)"> </span>30+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">Sheet1</span><span style="background:var(--vt-palette-6)"> </span> ▦ trips <span style="color:var(--vt-palette-8)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-8)">modified</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+K</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">search</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+/</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">shortcuts</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">F10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A1</span><span style="background:var(--vt-palette-7)"> </span>3+<span style="color:var(--vt-palette-3)">!</span> <span style="color:var(--vt-palette-3)">trips.parquet:</span> <span style="color:var(--vt-palette-3)">the</span> <span style="color:var(--vt-palette-3)">file</span> <span style="color:var(--vt-palette-3)">isn't</span> <span style="color:var(--vt-palette-3)">there</span>4+<span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>8+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>9+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>10+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>11+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>12+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>13+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>14+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>15+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>16+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>17+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>18+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>19+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>20+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>21+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>22+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>23+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>24+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>25+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>26+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>27+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>28+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>29+<span style="background:var(--vt-palette-7)"> </span> <span style="font-weight:bold">┃</span>30+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">!</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-0)">+</span> <span style="color:var(--vt-palette-8)">│</span> rides.012 <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+K</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">search</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+/</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">shortcuts</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">F10</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A1</span><span style="background:var(--vt-palette-8)"> </span>3+<span style="color:var(--vt-palette-3)">!</span> <span style="color:var(--vt-palette-3)">trips.parquet:</span> <span style="color:var(--vt-palette-3)">the</span> <span style="color:var(--vt-palette-3)">file</span> <span style="color:var(--vt-palette-3)">isn't</span> <span style="color:var(--vt-palette-3)">there</span>4+<span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>8+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>9+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>10+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>11+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>12+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>13+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>14+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>15+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>16+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>17+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>18+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>19+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>20+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>21+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>22+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>23+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>24+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>25+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>26+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>27+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>28+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>29+<span style="background:var(--vt-palette-8)"> </span> <span style="font-weight:bold">┃</span>30+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">!</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-8)">+</span> <span style="color:var(--vt-palette-8)">│</span> rides.012 <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+K</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">search</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+/</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">shortcuts</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">F10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">F</span><span style="color:#ffffff;background:#1f1f1f">ile</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">E</span><span style="color:#ffffff;background:#1f1f1f">dit</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">V</span><span style="color:#ffffff;background:#1f1f1f">iew</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">I</span><span style="color:#ffffff;background:#1f1f1f">nsert</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f">F</span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">o</span><span style="color:#ffffff;background:#1f1f1f">rmat</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">D</span><span style="color:#ffffff;background:#1f1f1f">ata</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f;text-decoration:underline">H</span><span style="color:#ffffff;background:#1f1f1f">elp</span><span style="background:#1f1f1f"> </span><span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff;font-weight:bold">SOURCE</span><span style="background:#40e0ff"> </span>2+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">C1867502</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span>3+<span style="color:#959595;background:#000000">▦</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000;font-weight:bold">trips.parquet</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">2,000,000</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">rows,</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">5</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">columns;</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">sorted</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">by</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">fare</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">Z</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">to</span><span style="background:#000000"> </span><span style="color:#959595;background:#000000">A</span><span style="background:#000000"> </span>4+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">A</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">B</span><span style="background:#3a3a3a"> </span><span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff;font-weight:bold">C</span><span style="background:#40e0ff"> </span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">D</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">E</span><span style="background:#3a3a3a"> </span>5+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1</span><span style="background:#3a3a3a"> </span><span style="background:#000000;text-decoration:underline"> </span><span style="color:#40e0ff;background:#000000;font-weight:bold;text-decoration:underline">trip</span><span style="background:#000000;text-decoration:underline"> </span><span style="color:#40e0ff;background:#000000;font-weight:bold;text-decoration:underline">payment</span><span style="background:#000000;text-decoration:underline"> </span><span style="color:#40e0ff;background:#000000;font-weight:bold;text-decoration:underline">fare</span><span style="background:#000000;text-decoration:underline"> </span><span style="color:#40e0ff;background:#000000;font-weight:bold;text-decoration:underline">miles</span><span style="background:#000000;text-decoration:underline"> </span><span style="color:#40e0ff;background:#000000;font-weight:bold;text-decoration:underline">day</span><span style="background:#000000;text-decoration:underline"> </span><span style="background:#000000"> </span>6+<span style="color:#757575;background:#000000">────────────────────────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff;font-weight:bold">1867502</span><span style="background:#40e0ff"> </span><span style="color:#ffffff;background:#000000">1867501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff">48</span><span style="background:#40e0ff"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">6/30/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>8+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1876502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1876501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">app</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>9+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1885502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1885501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>10+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1894502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1894501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">voucher</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">6/30/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>11+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1903502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1903501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>12+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1912502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1912501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>13+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1921502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1921501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">6/30/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>14+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1930502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1930501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>15+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1939502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1939501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">app</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>16+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1948502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1948501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">6/30/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>17+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1957502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1957501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">voucher</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/1/2026</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000;font-weight:bold">┃</span>18+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1966502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1966501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>19+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1975502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1975501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">6/30/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>20+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1984502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1984501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>21+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">1993502</span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#000000">1993501</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">48</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>22+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">8823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">8822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">7/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>23+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">17823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">17822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">voucher</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/2/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>24+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">26823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">26822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/2/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>25+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">35823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">35822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">7/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>26+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">44823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">44822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">card</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/2/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>27+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">53823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">53822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">4/2/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>28+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">62823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">62822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">app</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">10.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">7/1/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>29+<span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">71823</span><span style="background:#3a3a3a"> </span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">71822</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">cash</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">47.99</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">30.9</span><span style="background:#000000"> </span><span style="color:#ffffff;background:#000000">1/2/2026</span><span style="background:#000000"> </span><span style="color:#757575;background:#000000">│</span>30+<span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f">Sheet1</span><span style="background:#1f1f1f"> </span><span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff;font-weight:bold">▦</span><span style="background:#40e0ff"> </span><span style="color:#000000;background:#40e0ff;font-weight:bold">trips</span><span style="background:#40e0ff"> </span><span style="background:#1f1f1f"> </span><span style="color:#959595;background:#1f1f1f">+</span><span style="background:#1f1f1f"> </span><span style="color:#757575;background:#1f1f1f">│</span><span style="background:#1f1f1f"> </span><span style="color:#ffffff;background:#1f1f1f">untitled</span><span style="background:#1f1f1f"> </span><span style="color:#959595;background:#1f1f1f">modified</span><span style="background:#1f1f1f"> </span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">Ctrl+K</span><span style="background:#3a3a3a"> </span><span style="background:#1f1f1f"> </span><span style="color:#959595;background:#1f1f1f">search</span><span style="background:#1f1f1f"> </span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">Ctrl+/</span><span style="background:#3a3a3a"> </span><span style="background:#1f1f1f"> </span><span style="color:#959595;background:#1f1f1f">shortcuts</span><span style="background:#1f1f1f"> </span><span style="background:#3a3a3a"> </span><span style="color:#ffffff;background:#3a3a3a">F10</span><span style="background:#3a3a3a"> </span><span style="background:#1f1f1f"> </span><span style="color:#959595;background:#1f1f1f">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">C1867502</span><span style="background:var(--vt-palette-7)"> </span> 483+<span style="color:var(--vt-palette-0)">▦</span> <span style="font-weight:bold">trips.parquet</span> <span style="color:var(--vt-palette-0)">2,000,000</span> <span style="color:var(--vt-palette-0)">rows,</span> <span style="color:var(--vt-palette-0)">5</span> <span style="color:var(--vt-palette-0)">columns;</span> <span style="color:var(--vt-palette-0)">sorted</span> <span style="color:var(--vt-palette-0)">by</span> <span style="color:var(--vt-palette-0)">fare</span> <span style="color:var(--vt-palette-0)">Z</span> <span style="color:var(--vt-palette-0)">to</span> <span style="color:var(--vt-palette-0)">A</span>4+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">A</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">B</span><span style="background:var(--vt-palette-7)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">C</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">D</span><span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">E</span><span style="background:var(--vt-palette-7)"> </span>5+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1</span><span style="background:var(--vt-palette-7)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">payment</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">fare</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">miles</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-4);font-weight:bold;text-decoration:underline">day</span><span style="text-decoration:underline"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">1867502</span><span style="background:var(--vt-palette-6)"> </span>1867501 cash <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">48</span><span style="background:var(--vt-palette-6)"> </span> 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>8+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1876502</span><span style="background:var(--vt-palette-7)"> </span>1876501 app 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1885502</span><span style="background:var(--vt-palette-7)"> </span>1885501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1894502</span><span style="background:var(--vt-palette-7)"> </span>1894501 voucher 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1903502</span><span style="background:var(--vt-palette-7)"> </span>1903501 card 48 30 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1912502</span><span style="background:var(--vt-palette-7)"> </span>1912501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>13+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1921502</span><span style="background:var(--vt-palette-7)"> </span>1921501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1930502</span><span style="background:var(--vt-palette-7)"> </span>1930501 cash 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1939502</span><span style="background:var(--vt-palette-7)"> </span>1939501 app 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1948502</span><span style="background:var(--vt-palette-7)"> </span>1948501 cash 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1957502</span><span style="background:var(--vt-palette-7)"> </span>1957501 voucher 48 30 1/1/2026 <span style="font-weight:bold">┃</span>18+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1966502</span><span style="background:var(--vt-palette-7)"> </span>1966501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1975502</span><span style="background:var(--vt-palette-7)"> </span>1975501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>20+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1984502</span><span style="background:var(--vt-palette-7)"> </span>1984501 card 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>21+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">1993502</span><span style="background:var(--vt-palette-7)"> </span>1993501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>22+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">8823</span><span style="background:var(--vt-palette-7)"> </span> 8822 cash 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>23+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">17823</span><span style="background:var(--vt-palette-7)"> </span> 17822 voucher 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>24+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">26823</span><span style="background:var(--vt-palette-7)"> </span> 26822 card 47.99 10.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>25+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">35823</span><span style="background:var(--vt-palette-7)"> </span> 35822 card 47.99 30.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>26+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">44823</span><span style="background:var(--vt-palette-7)"> </span> 44822 card 47.99 10.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>27+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">53823</span><span style="background:var(--vt-palette-7)"> </span> 53822 cash 47.99 30.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>28+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">62823</span><span style="background:var(--vt-palette-7)"> </span> 62822 app 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>29+<span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">71823</span><span style="background:var(--vt-palette-7)"> </span> 71822 cash 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>30+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-0)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-0)">modified</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+K</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">search</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">Ctrl+/</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">shortcuts</span> <span style="background:var(--vt-palette-7)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-7)">F10</span><span style="background:var(--vt-palette-7)"> </span> <span style="color:var(--vt-palette-0)">menu</span></pre>\ No newline at end of file@@ -0,0 +1,30 @@
1+<pre class="screen"> <span style="text-decoration:underline">F</span>ile <span style="text-decoration:underline">E</span>dit <span style="text-decoration:underline">V</span>iew <span style="text-decoration:underline">I</span>nsert F<span style="text-decoration:underline">o</span>rmat <span style="text-decoration:underline">D</span>ata <span style="text-decoration:underline">H</span>elp <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">SOURCE</span><span style="background:var(--vt-palette-6)"> </span>2+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">C1867502</span><span style="background:var(--vt-palette-8)"> </span> 483+<span style="color:var(--vt-palette-8)">▦</span> <span style="font-weight:bold">trips.parquet</span> <span style="color:var(--vt-palette-8)">2,000,000</span> <span style="color:var(--vt-palette-8)">rows,</span> <span style="color:var(--vt-palette-8)">5</span> <span style="color:var(--vt-palette-8)">columns;</span> <span style="color:var(--vt-palette-8)">sorted</span> <span style="color:var(--vt-palette-8)">by</span> <span style="color:var(--vt-palette-8)">fare</span> <span style="color:var(--vt-palette-8)">Z</span> <span style="color:var(--vt-palette-8)">to</span> <span style="color:var(--vt-palette-8)">A</span>4+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">A</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">B</span><span style="background:var(--vt-palette-8)"> </span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">C</span><span style="background:var(--vt-palette-6)"> </span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">D</span><span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">E</span><span style="background:var(--vt-palette-8)"> </span>5+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1</span><span style="background:var(--vt-palette-8)"> </span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">trip</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">payment</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">fare</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">miles</span><span style="text-decoration:underline"> </span><span style="color:var(--vt-palette-6);font-weight:bold;text-decoration:underline">day</span><span style="text-decoration:underline"> </span>6+<span style="color:var(--vt-palette-8)">────────────────────────────────────────────────────────────────────────────────────────────────────</span>7+<span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">1867502</span><span style="background:var(--vt-palette-6)"> </span>1867501 cash <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6)">48</span><span style="background:var(--vt-palette-6)"> </span> 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>8+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1876502</span><span style="background:var(--vt-palette-8)"> </span>1876501 app 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>9+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1885502</span><span style="background:var(--vt-palette-8)"> </span>1885501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>10+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1894502</span><span style="background:var(--vt-palette-8)"> </span>1894501 voucher 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>11+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1903502</span><span style="background:var(--vt-palette-8)"> </span>1903501 card 48 30 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>12+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1912502</span><span style="background:var(--vt-palette-8)"> </span>1912501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>13+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1921502</span><span style="background:var(--vt-palette-8)"> </span>1921501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>14+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1930502</span><span style="background:var(--vt-palette-8)"> </span>1930501 cash 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>15+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1939502</span><span style="background:var(--vt-palette-8)"> </span>1939501 app 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>16+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1948502</span><span style="background:var(--vt-palette-8)"> </span>1948501 cash 48 10 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>17+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1957502</span><span style="background:var(--vt-palette-8)"> </span>1957501 voucher 48 30 1/1/2026 <span style="font-weight:bold">┃</span>18+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1966502</span><span style="background:var(--vt-palette-8)"> </span>1966501 card 48 10 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>19+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1975502</span><span style="background:var(--vt-palette-8)"> </span>1975501 card 48 30 6/30/2026 <span style="color:var(--vt-palette-8)">│</span>20+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1984502</span><span style="background:var(--vt-palette-8)"> </span>1984501 card 48 10 1/1/2026 <span style="color:var(--vt-palette-8)">│</span>21+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">1993502</span><span style="background:var(--vt-palette-8)"> </span>1993501 cash 48 30 4/1/2026 <span style="color:var(--vt-palette-8)">│</span>22+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">8823</span><span style="background:var(--vt-palette-8)"> </span> 8822 cash 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>23+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">17823</span><span style="background:var(--vt-palette-8)"> </span> 17822 voucher 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>24+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">26823</span><span style="background:var(--vt-palette-8)"> </span> 26822 card 47.99 10.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>25+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">35823</span><span style="background:var(--vt-palette-8)"> </span> 35822 card 47.99 30.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>26+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">44823</span><span style="background:var(--vt-palette-8)"> </span> 44822 card 47.99 10.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>27+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">53823</span><span style="background:var(--vt-palette-8)"> </span> 53822 cash 47.99 30.9 4/2/2026 <span style="color:var(--vt-palette-8)">│</span>28+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">62823</span><span style="background:var(--vt-palette-8)"> </span> 62822 app 47.99 10.9 7/1/2026 <span style="color:var(--vt-palette-8)">│</span>29+<span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">71823</span><span style="background:var(--vt-palette-8)"> </span> 71822 cash 47.99 30.9 1/2/2026 <span style="color:var(--vt-palette-8)">│</span>30+ Sheet1 <span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">▦</span><span style="background:var(--vt-palette-6)"> </span><span style="color:var(--vt-palette-0);background:var(--vt-palette-6);font-weight:bold">trips</span><span style="background:var(--vt-palette-6)"> </span> <span style="color:var(--vt-palette-8)">+</span> <span style="color:var(--vt-palette-8)">│</span> untitled <span style="color:var(--vt-palette-8)">modified</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+K</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">search</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">Ctrl+/</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">shortcuts</span> <span style="background:var(--vt-palette-8)"> </span><span style="color:var(--vt-palette-15);background:var(--vt-palette-8)">F10</span><span style="background:var(--vt-palette-8)"> </span> <span style="color:var(--vt-palette-8)">menu</span></pre>\ No newline at end of file@@ -116,6 +116,7 @@ var Options = []Option{
116116 "take 20 to 60 bytes a cell and formulas about 750, so the default of ten million cells of " +117117 "data is 200 to 600 MB. Imports keep whole rows " +118118 "up to the budget and say how many they left out; larger pastes and fills are refused. " +119+ "A function that holds what it reads (MEDIAN, SORT) is given no more of a linked source. " +119120 "The grid itself is 1,048,576 rows by 16,384 columns (A to XFD) whatever this is."},120121 121122 {Name: "shell", Kind: Enum, Group: GroupShell, Default: "ask", Values: []string{"off", "ask", "on"}, Env: []string{"O12_SHELL"}, Live: true,@@ -94,6 +94,7 @@ type tableCells struct {
9494 cols map[string]int9595 zone *time.Location9696 dated map[int]bool // columns widened for dates97+ fixed bool // widths are the sheet's own: dates widen nothing9798 }9899 99100 // dateTimeWidth fits a date and time in the Date time format, in any@@ -148,7 +149,7 @@ func (t *tableCells) value(a sheet.Addr, v nuon.Value) {
148149 default:149150 b.text(a, c.V.Str, c.F, none)150151 }151- if v.Kind == nuon.Date && !t.dated[a.Col] && a.Valid() {152+ if v.Kind == nuon.Date && !t.fixed && !t.dated[a.Col] && a.Valid() {152153 t.dated[a.Col] = true153154 b.s.LoadColWidth(a.Col, dateTimeWidth)154155 }@@ -3,6 +3,8 @@ package fileio
33 import (44 "bytes"55 "context"6+ "errors"7+ "io"68 "strings"79 810 "github.com/FelineStateMachine/012/internal/nuon"@@ -33,3 +35,35 @@ func NUONRows(ctx context.Context, data []byte, maxCells int) (TailRows, string,
3335 // value's own, and its format (a file size's Size, a date's Date time,3436 // in the local time zone), as importing it would store it.3537 func NUONCell(v nuon.Value) sheet.LiveCell { return nuonCell(v, zone()) }38+39+// AppendTable reads data, a NUON table, into s as more rows of the table40+// importing a table put there: under its row last, each value in the41+// column its header row (row 0, cols columns) names, a column named for42+// the first time added after the others with its name in the header,43+// the cells as importing stores them. It leaves the widths alone and44+// recalculates nothing, the cells being values, and returns the rows45+// and the columns the table has now.46+func AppendTable(ctx context.Context, s *sheet.Sheet, last, cols int, data []byte) (rows, allCols int, err error) {47+ b := &builder{s: s, maxCells: -1}48+ t := tableCells{b: b, cols: make(map[string]int, cols), zone: zone(), dated: map[int]bool{}, fixed: true}49+ for c := range cols {50+ t.cols[s.ShownText(sheet.Addr{Col: c})] = c51+ }52+ r := nuon.NewReader(bytes.NewReader(data))53+ for ; ; rows++ {54+ if rows%256 == 0 {55+ if err := ctx.Err(); err != nil {56+ return rows, len(t.cols), err57+ }58+ }59+ fields, err := r.Next()60+ if errors.Is(err, io.EOF) {61+ return rows, len(t.cols), nil62+ }63+ if err != nil {64+ return rows, len(t.cols), err65+ }66+ t.header(r.Header())67+ t.row(last+rows+1, fields)68+ }69+}@@ -20,6 +20,9 @@ import (
2020 type parquetColumn struct {2121 logical format.LogicalTypeValue2222 kind parquet.Kind23+ // repeated is set on a list's column, whose rows hold several24+ // values; sources read it a row at a time (sourceparquet.go).25+ repeated bool2326 }2427 2528 // importParquet reads every leaf column, named by its path, into a@@ -134,78 +137,29 @@ func putParquetRow(b *builder, row int, r parquet.Row, cols []parquetColumn) (re
134137 b.fits(sheet.Addr{Row: row})135138 return false136139 }137- // Values arrive grouped by column; a repeated column has several.140+ eachParquetColumn(r, func(col int, vals []parquet.Value) {141+ if col < 0 || col >= len(cols) {142+ return143+ }144+ c, rep := parquetCells(vals, cols[col])145+ c.put(b, sheet.Addr{Col: col, Row: row})146+ repeated = repeated || rep147+ })148+ return repeated149+}150+151+// eachParquetColumn calls fn with each column's values in a row: they152+// arrive grouped by column, and a repeated column has several.153+func eachParquetColumn(r parquet.Row, fn func(col int, vals []parquet.Value)) {138154 for i := 0; i < len(r); {139155 col := r[i].Column()140156 j := i + 1141157 for j < len(r) && r[j].Column() == col {142158 j++143159 }144- if col < 0 || col >= len(cols) {145- i = j146- continue147- }148- a := sheet.Addr{Col: col, Row: row}149- vals := r[i:j]150- if len(vals) > 1 {151- parts := make([]string, 0, len(vals))152- for _, v := range vals {153- if !v.IsNull() {154- parts = append(parts, parquetText(v, cols[col]))155- }156- }157- if len(parts) > 0 {158- b.text(a, strings.Join(parts, ", "), sheet.Format{}, sheet.Style{})159- repeated = true160- }161- } else if !vals[0].IsNull() {162- putParquetValue(b, a, vals[0], cols[col])163- }160+ fn(col, r[i:j])164161 i = j165162 }166- return repeated167-}168-169-func putParquetValue(b *builder, a sheet.Addr, v parquet.Value, c parquetColumn) {170- switch lt := c.logical.(type) {171- case *format.DateType:172- b.number(a, float64(v.Int32())+serialOf(time.Unix(0, 0).UTC()), sheet.Format{Kind: sheet.FmtDate}, sheet.Style{})173- return174- case *format.TimestampType:175- t := timestamp(v.Int64(), lt.Unit)176- b.number(a, serialOf(t), sheet.Format{Kind: sheet.FmtDateTime}, sheet.Style{})177- return178- case *format.TimeType:179- d := timeUnit(lt.Unit) * time.Duration(timeInt(v))180- b.number(a, d.Hours()/24, sheet.Format{Kind: sheet.FmtTime}, sheet.Style{})181- return182- case *format.DecimalType:183- if x, ok := decimal(v, lt.Scale); ok {184- b.number(a, x, sheet.Format{}, sheet.Style{})185- return186- }187- }188- switch c.kind {189- case parquet.Boolean:190- b.boolean(a, v.Boolean(), sheet.Style{})191- case parquet.Int32:192- b.number(a, float64(v.Int32()), sheet.Format{}, sheet.Style{})193- case parquet.Int64:194- b.number(a, float64(v.Int64()), sheet.Format{}, sheet.Style{})195- case parquet.Float:196- b.number(a, float64(v.Float()), sheet.Format{}, sheet.Style{})197- case parquet.Double:198- b.number(a, v.Double(), sheet.Format{}, sheet.Style{})199- case parquet.Int96:200- // Legacy timestamps: nanoseconds of the day, then the Julian day.201- i := v.Int96()202- nanos := int64(i[1])<<32 | int64(i[0])203- days := int64(i[2]) - 2440588 // the Julian day of 1970-01-01204- t := time.Unix(days*86400, nanos).UTC()205- b.number(a, serialOf(t), sheet.Format{Kind: sheet.FmtDateTime}, sheet.Style{})206- default:207- textOrDate(b, a, parquetText(v, c))208- }209163 }210164 211165 // parquetText is a value as text, for byte arrays and joined lists.@@ -0,0 +1,164 @@
1+package fileio2+3+import (4+ "cmp"5+ "context"6+ "errors"7+ "fmt"8+ "os"9+ "slices"10+11+ "github.com/FelineStateMachine/012/internal/sheet"12+)13+14+// Sources: a Parquet file, or a SQLite table or query, read in place a15+// window of rows at a time rather than imported, for tables too big for16+// any grid. A Source streams its rows in its own order for what reads17+// them all (aggregates, pivot tables) and fetches the rows a screen18+// shows; a SourceView is its rows as a sort and a filter order them,19+// pushed down to SQL for SQLite and streamed for Parquet. Values come20+// out typed as an import types them.21+22+// SourceSpec names what a source reads.23+type SourceSpec struct {24+ // Path is the file.25+ Path string26+ // Format is "Parquet" or "SQLite", or "" to tell by the extension.27+ Format string28+ // Table is a SQLite table or view to read, or Query a query to run;29+ // both empty reads a database's only table.30+ Table, Query string31+ // TempDir is where views keep what they build (sorted row orders,32+ // SQLite's temporary tables); "" is the system's.33+ TempDir string34+}35+36+// SourceColumn is one of a source's columns.37+type SourceColumn struct {38+ Name string39+ // Format is what its values show in when the column's type says (a40+ // Parquet DATE column's FmtDate); a value's own may differ.41+ Format sheet.Format42+ // Numeric is set when its type holds numbers, dates and times43+ // included.44+ Numeric bool45+}46+47+// SourceSort is a column a view is sorted by.48+type SourceSort struct {49+ Col int50+ Desc bool51+}52+53+// SourceFilter is a condition a view's rows meet on a column.54+type SourceFilter struct {55+ Col int56+ Cond sheet.Condition57+}58+59+// SourceOrder is how a view sorts and filters a source's rows.60+type SourceOrder struct {61+ Sort []SourceSort62+ Filter []SourceFilter63+}64+65+// IsZero reports whether the order is the source's own, every row.66+func (o SourceOrder) IsZero() bool { return len(o.Sort) == 0 && len(o.Filter) == 0 }67+68+// Source is a table read in place. It is safe for use by several69+// goroutines at once.70+type Source interface {71+ // Columns are the source's columns, left to right.72+ Columns() []SourceColumn73+ // Rows counts its rows, the header not among them.74+ Rows() int6475+ // Scan streams the rows from row from on (counting from 0) in the76+ // source's order, calling fn with each row's number and the values77+ // of cols (every column when cols is nil, in that order; the slice78+ // is reused) until fn returns false, the rows end or ctx is done.79+ Scan(ctx context.Context, from int64, cols []int, fn func(row int64, vals []sheet.LiveCell) bool) error80+ // Fetch reads the rows numbered rows, in the order given, with the81+ // values of cols (every column when nil).82+ Fetch(ctx context.Context, rows []int64, cols []int) ([][]sheet.LiveCell, error)83+ // View orders the rows as o says. Building it may read the whole84+ // source once, which the caller does in the background.85+ View(ctx context.Context, o SourceOrder) (SourceView, error)86+ // Close lets the file go, and what its views built.87+ Close() error88+}89+90+// SourceView is a source's rows as a sort and a filter order them.91+type SourceView interface {92+ // Rows counts the rows the view shows.93+ Rows() int6494+ // Page reads n rows from position from of the view: each one's95+ // number in the source and its values.96+ Page(ctx context.Context, from int64, n int) ([]int64, [][]sheet.LiveCell, error)97+ // Close lets go of what the view built.98+ Close() error99+}100+101+// ErrNotSource is what OpenSource says of a file that is neither102+// Parquet nor SQLite.103+var ErrNotSource = errors.New("only Parquet files and SQLite tables or queries can be linked as sources")104+105+// SourceKind is the format spec reads, Parquet or SQLite, by its Format106+// or else the file's extension.107+func SourceKind(spec SourceSpec) (Kind, error) {108+ k, ok := KindNamed(spec.Format)109+ if spec.Format == "" {110+ k, ok = KindOf(spec.Path)111+ }112+ if !ok || k != Parquet && k != SQLite {113+ return 0, ErrNotSource114+ }115+ return k, nil116+}117+118+// OpenSource opens what spec names.119+func OpenSource(ctx context.Context, spec SourceSpec) (Source, error) {120+ k, err := SourceKind(spec)121+ if err != nil {122+ return nil, err123+ }124+ if spec.TempDir == "" {125+ spec.TempDir = os.TempDir()126+ }127+ if k == Parquet {128+ return openParquetSource(spec)129+ }130+ return openSQLiteSource(ctx, spec)131+}132+133+// allCols is cols, or every one of n columns when cols is nil.134+func allCols(cols []int, n int) []int {135+ if cols != nil {136+ return cols137+ }138+ out := make([]int, n)139+ for i := range out {140+ out[i] = i141+ }142+ return out143+}144+145+// checkCols reports a column past the source's n.146+func checkCols(cols []int, n int) error {147+ for _, c := range cols {148+ if c < 0 || c >= n {149+ return fmt.Errorf("the source has no column %d", c+1)150+ }151+ }152+ return nil153+}154+155+// fetchOrder is the positions of rows sorted by row number, so a fetch156+// reads forward, the answers going back to the order asked.157+func fetchOrder(rows []int64) []int {158+ idx := make([]int, len(rows))159+ for i := range idx {160+ idx[i] = i161+ }162+ slices.SortFunc(idx, func(a, b int) int { return cmp.Compare(rows[a], rows[b]) })163+ return idx164+}@@ -0,0 +1,121 @@
1+//go:build stress2+3+// Stress benchmarks of sources: a Parquet file and a SQLite table of ten4+// million rows, generated once into $STRESS_DIR/generated (or a5+// temporary directory), read in place: opening, summing a column as SUM6+// does, a page deep in the file, and building and paging a sorted and a7+// filtered view, each with the peak heap it took. Timed under the bench8+// lock (/tmp/012-bench.lock), as the speed gate is.9+package fileio10+11+import (12+ "context"13+ "math/rand/v2"14+ "testing"15+16+ "github.com/FelineStateMachine/012/internal/sheet"17+ "github.com/FelineStateMachine/012/internal/stress"18+)19+20+// bigRows is the rows of the generated sources (stress.SalesParquet).21+const bigRows = 10_000_00022+23+// BenchmarkSource reads ten million rows of each kind of source.24+func BenchmarkSource(b *testing.B) {25+ dir := stress.GeneratedDir(b.TempDir())26+ pq, err := stress.SalesParquet(dir, bigRows)27+ if err != nil {28+ b.Fatal(err)29+ }30+ db, err := stress.SalesSQLite(dir, bigRows)31+ if err != nil {32+ b.Fatal(err)33+ }34+ files := map[string]SourceSpec{"parquet": {Path: pq}, "sqlite": {Path: db}}35+ defer stress.BenchLock(b)()36+ for _, kind := range []string{"parquet", "sqlite"} {37+ spec := files[kind]38+ spec.TempDir = b.TempDir()39+ b.Run(kind, func(b *testing.B) { benchSource(b, spec) })40+ }41+}42+43+// sortByAmount and northOnly are the views measured.44+var (45+ sortByAmount = SourceOrder{Sort: []SourceSort{{Col: 2, Desc: true}}}46+ northOnly = SourceOrder{Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondExactly, Arg: "north"}}}}47+)48+49+func benchSource(b *testing.B, spec SourceSpec) {50+ ctx := context.Background()51+ b.Run("open", func(b *testing.B) {52+ measure(b, func() {53+ src, err := OpenSource(ctx, spec)54+ if err != nil {55+ b.Fatal(err)56+ }57+ if src.Rows() != bigRows {58+ b.Fatalf("%d rows", src.Rows())59+ }60+ src.Close()61+ })62+ })63+ src, err := OpenSource(ctx, spec)64+ if err != nil {65+ b.Fatal(err)66+ }67+ defer src.Close()68+ b.Run("sum-column", func(b *testing.B) {69+ measure(b, func() {70+ total := 0.071+ err := src.Scan(ctx, 0, []int{2}, func(_ int64, v []sheet.LiveCell) bool {72+ total += v[0].V.Num73+ return true74+ })75+ if err != nil {76+ b.Fatal(err)77+ }78+ })79+ b.ReportMetric(float64(bigRows)*float64(b.N)/b.Elapsed().Seconds(), "rows/s")80+ })81+ for _, o := range []struct {82+ name string83+ order SourceOrder84+ }{{"plain", SourceOrder{}}, {"sorted", sortByAmount}, {"filtered", northOnly}} {85+ var v SourceView86+ b.Run("view-"+o.name, func(b *testing.B) {87+ measure(b, func() {88+ if v != nil {89+ v.Close()90+ }91+ if v, err = src.View(ctx, o.order); err != nil {92+ b.Fatal(err)93+ }94+ })95+ b.ReportMetric(float64(v.Rows()), "rows")96+ })97+ b.Run("page-"+o.name, func(b *testing.B) {98+ r := rand.New(rand.NewPCG(1, 2))99+ measure(b, func() {100+ from := v.Rows()/2 + r.Int64N(v.Rows()/2-60)101+ nums, _, err := v.Page(ctx, from, 60)102+ if err != nil || len(nums) != 60 {103+ b.Fatalf("page at %d: %d rows, %v", from, len(nums), err)104+ }105+ })106+ })107+ v.Close()108+ }109+}110+111+// measure runs fn in the benchmark's loop, reporting the peak heap it112+// took above what was live before.113+func measure(b *testing.B, fn func()) {114+ before := liveHeap()115+ stop := samplePeak()116+ for b.Loop() {117+ fn()118+ }119+ peak := stop()120+ b.ReportMetric(float64(peak-min(peak, before))/(1<<20), "peak-MB")121+}@@ -0,0 +1,330 @@
1+package fileio2+3+import (4+ "context"5+ "fmt"6+ "os"7+ "path/filepath"8+ "slices"9+ "strings"10+ "sync"11+ "testing"12+ "time"13+14+ "github.com/parquet-go/parquet-go"15+16+ "github.com/FelineStateMachine/012/internal/sheet"17+)18+19+// sale is a row of the Parquet sources tests read.20+type sale struct {21+ ID int64 `parquet:"id"`22+ Cat *string `parquet:"cat,optional"`23+ Amount *float64 `parquet:"amount,optional"`24+ Day int32 `parquet:"day,date"`25+ Tags []string `parquet:"tags,list"`26+}27+28+var saleCats = []string{"North", "south", "East", "west", "Éire", "", "north"}29+30+// writeSales writes n rows in row groups of group rows.31+func writeSales(t testing.TB, dir string, n, group int) string {32+ t.Helper()33+ name := filepath.Join(dir, "sales.parquet")34+ f, err := os.Create(name)35+ if err != nil {36+ t.Fatal(err)37+ }38+ w := parquet.NewGenericWriter[sale](f, parquet.MaxRowsPerRowGroup(int64(group)), parquet.PageBufferSize(512))39+ day := int32(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC).Unix() / 86400)40+ rows := make([]sale, n)41+ for i := range rows {42+ r := &rows[i]43+ r.ID, r.Day = int64(i), day+int32(i%40)44+ if i%9 != 4 {45+ c := saleCats[i%len(saleCats)]46+ r.Cat = &c47+ }48+ if i%11 != 7 {49+ a := float64((i*37)%101) - 20.550+ r.Amount = &a51+ }52+ if i%5 == 0 {53+ r.Tags = []string{"a", fmt.Sprint(i)}54+ }55+ }56+ if _, err := w.Write(rows); err != nil {57+ t.Fatal(err)58+ }59+ if err := w.Close(); err != nil {60+ t.Fatal(err)61+ }62+ f.Close()63+ return name64+}65+66+// cellsText is a row as its cells show, separated by |.67+func cellsText(row []sheet.LiveCell) string {68+ parts := make([]string, len(row))69+ for i, c := range row {70+ parts[i] = sheet.FormatText(c.V, c.F)71+ }72+ return strings.Join(parts, "|")73+}74+75+// imported is a file imported as a sheet, which sources must read as76+// the import does, and the rows under its header.77+func imported(t *testing.T, name string, opt Options) (*sheet.Sheet, sheet.Rect) {78+ t.Helper()79+ res, err := Import(context.Background(), name, opt)80+ if err != nil {81+ t.Fatal(err)82+ }83+ used, _ := res.Sheet.UsedRange()84+ used.From.Row = 185+ return res.Sheet, used86+}87+88+// sheetRows are the rows of r a sheet shows, the filter's hidden ones89+// left out.90+func sheetRows(s *sheet.Sheet, r sheet.Rect) []string {91+ var out []string92+ for row := r.From.Row; row <= r.To.Row; row++ {93+ if s.RowHidden(row) {94+ continue95+ }96+ parts := make([]string, 0, r.To.Col+1)97+ for c := r.From.Col; c <= r.To.Col; c++ {98+ parts = append(parts, shown(s, sheet.Addr{Col: c, Row: row}))99+ }100+ out = append(out, strings.Join(parts, "|"))101+ }102+ return out103+}104+105+// viewRows pages through a whole view, n rows at a time.106+func viewRows(t *testing.T, v SourceView, n int) ([]int64, []string) {107+ t.Helper()108+ var nums []int64109+ var out []string110+ for from := int64(0); from < v.Rows(); from += int64(n) {111+ ns, rows, err := v.Page(context.Background(), from, n)112+ if err != nil {113+ t.Fatal(err)114+ }115+ if len(ns) != min(n, int(v.Rows()-from)) {116+ t.Fatalf("page at %d has %d rows", from, len(ns))117+ }118+ for i, r := range rows {119+ nums, out = append(nums, ns[i]), append(out, cellsText(r))120+ }121+ }122+ return nums, out123+}124+125+// orders are the sorts and filters every source is checked with,126+// against a sheet sorting and filtering the same rows.127+var orders = []SourceOrder{128+ {},129+ {Sort: []SourceSort{{Col: 1}}},130+ {Sort: []SourceSort{{Col: 1, Desc: true}}},131+ {Sort: []SourceSort{{Col: 2, Desc: true}, {Col: 1}}},132+ {Filter: []SourceFilter{{Col: 2, Cond: sheet.Condition{Op: sheet.CondGreater, Arg: "30"}}}},133+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondContains, Arg: "OR"}}}},134+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondEmpty}}}},135+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondNotEqual, Arg: "north"}}}},136+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondStartsWith, Arg: "e"}}}},137+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondEndsWith, Arg: "TH"}}}},138+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondExactly, Arg: "EAST"}}}},139+ {Filter: []SourceFilter{{Col: 2, Cond: sheet.Condition{Op: sheet.CondLessEq, Arg: "-10"}}},140+ Sort: []SourceSort{{Col: 2}}},141+ {Filter: []SourceFilter{{Col: 1, Cond: sheet.Condition{Op: sheet.CondNotEmpty}},142+ {Col: 2, Cond: sheet.Condition{Op: sheet.CondGreaterEq, Arg: "0"}}},143+ Sort: []SourceSort{{Col: 1, Desc: true}}},144+}145+146+// checkOrders checks each order's view against the sheet the file147+// imports as, sorted and filtered the same way.148+func checkOrders(t *testing.T, src Source, name string, opt Options, list []SourceOrder) {149+ t.Helper()150+ for _, o := range list {151+ s, r := imported(t, name, opt)152+ if len(o.Sort) > 0 {153+ var keys []sheet.SortKey154+ for _, k := range o.Sort {155+ keys = append(keys, sheet.SortKey{Col: k.Col, Desc: k.Desc})156+ }157+ s.SortRange(r, keys)158+ }159+ if len(o.Filter) > 0 {160+ s.CreateFilter(sheet.Rect{To: r.To})161+ for _, f := range o.Filter {162+ s.FilterColumn(f.Col, sheet.Criteria{Cond: f.Cond})163+ }164+ }165+ want := sheetRows(s, r)166+ v, err := src.View(context.Background(), o)167+ if err != nil {168+ t.Fatalf("%+v: %v", o, err)169+ }170+ _, got := viewRows(t, v, 37)171+ if !slices.Equal(got, want) {172+ i := 0173+ for i < min(len(got), len(want)) && got[i] == want[i] {174+ i++175+ }176+ t.Errorf("%+v: %d rows, want %d; from row %d\n got %q\nwant %q", o, len(got), len(want), i, head(got[i:]), head(want[i:]))177+ }178+ if err := v.Close(); err != nil {179+ t.Error(err)180+ }181+ }182+}183+184+func head(rows []string) []string { return rows[:min(len(rows), 12)] }185+186+func TestParquetSource(t *testing.T) {187+ dir := t.TempDir()188+ name := writeSales(t, dir, 1000, 128)189+ tmp := t.TempDir()190+ src, err := OpenSource(context.Background(), SourceSpec{Path: name, TempDir: tmp})191+ if err != nil {192+ t.Fatal(err)193+ }194+ defer src.Close()195+ if src.Rows() != 1000 {196+ t.Fatalf("%d rows", src.Rows())197+ }198+ var cols []string199+ for _, c := range src.Columns() {200+ cols = append(cols, fmt.Sprintf("%s:%v:%v", c.Name, c.Numeric, c.Format.Kind == sheet.FmtDate))201+ }202+ if got := strings.Join(cols, " "); got != "id:true:false cat:false:false amount:true:false day:true:true tags.list.element:false:false" {203+ t.Errorf("columns %s", got)204+ }205+ s, r := imported(t, name, Options{})206+ want := sheetRows(s, r)207+208+ // Scan from part way, two columns in another order.209+ var got []string210+ err = src.Scan(context.Background(), 500, []int{2, 0}, func(row int64, vals []sheet.LiveCell) bool {211+ got = append(got, fmt.Sprintf("%d:%s", row, cellsText(vals)))212+ return row < 510213+ })214+ if err != nil {215+ t.Fatal(err)216+ }217+ for i, g := range got {218+ parts := strings.Split(want[500+i], "|")219+ if w := fmt.Sprintf("%d:%s|%s", 500+i, parts[2], parts[0]); g != w {220+ t.Errorf("scan %q, want %q", g, w)221+ }222+ }223+ if len(got) != 11 {224+ t.Errorf("scan read %d rows", len(got))225+ }226+227+ // Fetch rows in any order, across row groups.228+ nums := []int64{999, 3, 500, 128, 127, 3}229+ rows, err := src.Fetch(context.Background(), nums, nil)230+ if err != nil {231+ t.Fatal(err)232+ }233+ for i, n := range nums {234+ if g := cellsText(rows[i]); g != want[n] {235+ t.Errorf("row %d is %q, want %q", n, g, want[n])236+ }237+ }238+ checkOrders(t, src, name, Options{}, orders)239+ if left, _ := os.ReadDir(tmp); len(left) != 0 {240+ t.Errorf("views left %d files", len(left))241+ }242+}243+244+// TestSourceSortSpills sorts in runs spilled to files and merged.245+func TestSourceSortSpills(t *testing.T) {246+ defer func(n int) { sortRunBytes = n }(sortRunBytes)247+ sortRunBytes = 4096248+ name := writeSales(t, t.TempDir(), 3000, 1000)249+ tmp := t.TempDir()250+ src, err := OpenSource(context.Background(), SourceSpec{Path: name, TempDir: tmp})251+ if err != nil {252+ t.Fatal(err)253+ }254+ checkOrders(t, src, name, Options{}, orders[1:4])255+ v, err := src.View(context.Background(), orders[2])256+ if err != nil {257+ t.Fatal(err)258+ }259+ if left, _ := os.ReadDir(tmp); len(left) != 1 {260+ t.Errorf("a view keeps %d files, want its row order", len(left))261+ }262+ if err := src.Close(); err != nil {263+ t.Fatal(err)264+ }265+ if left, _ := os.ReadDir(tmp); len(left) != 0 {266+ t.Errorf("closing left %d files", len(left))267+ }268+ if err := v.Close(); err != nil {269+ t.Errorf("closing a view after its source: %v", err)270+ }271+}272+273+// TestSourceConcurrent reads a source from several goroutines at once.274+func TestSourceConcurrent(t *testing.T) {275+ dir := t.TempDir()276+ for _, name := range []string{writeSales(t, dir, 2000, 300), writeSalesDB(t, dir, 2000, false)} {277+ src, err := OpenSource(context.Background(), SourceSpec{Path: name, Table: "sales", TempDir: dir})278+ if err != nil {279+ t.Fatal(err)280+ }281+ sum := func() float64 {282+ total := 0.0283+ err := src.Scan(context.Background(), 0, []int{2}, func(_ int64, v []sheet.LiveCell) bool {284+ total += v[0].V.Num285+ return true286+ })287+ if err != nil {288+ t.Error(err)289+ }290+ return total291+ }292+ want := sum()293+ var wg sync.WaitGroup294+ for i := range 8 {295+ wg.Go(func() {296+ if got := sum(); got != want {297+ t.Errorf("%s: sum %v, want %v", name, got, want)298+ }299+ if _, err := src.Fetch(context.Background(), []int64{int64(i * 200), 1999}, nil); err != nil {300+ t.Error(err)301+ }302+ v, err := src.View(context.Background(), SourceOrder{Sort: []SourceSort{{Col: 2}}})303+ if err != nil {304+ t.Error(err)305+ return306+ }307+ if _, _, err := v.Page(context.Background(), 1500, 60); err != nil {308+ t.Error(err)309+ }310+ v.Close()311+ })312+ }313+ wg.Wait()314+ src.Close()315+ }316+}317+318+func TestSourceKind(t *testing.T) {319+ for spec, ok := range map[SourceSpec]bool{320+ {Path: "a.parquet"}: true, {Path: "a.sqlite"}: true, {Path: "a.db"}: true,321+ {Path: "a.csv"}: false, {Path: "a.bin", Format: "Parquet"}: true, {Path: "a.parquet", Format: "CSV"}: false,322+ } {323+ if _, err := SourceKind(spec); (err == nil) != ok {324+ t.Errorf("%+v: %v", spec, err)325+ }326+ }327+ if _, err := OpenSource(context.Background(), SourceSpec{Path: filepath.Join(t.TempDir(), "none.parquet")}); err == nil {328+ t.Error("opened a file that isn't there")329+ }330+}@@ -0,0 +1,206 @@
1+package fileio2+3+import (4+ "fmt"5+ "math"6+ "strings"7+ "time"8+ "unicode/utf8"9+10+ "github.com/parquet-go/parquet-go"11+ "github.com/parquet-go/parquet-go/format"12+13+ "github.com/FelineStateMachine/012/internal/sheet"14+)15+16+// How Parquet and SQLite values become cells, shared by imports (which17+// store them through the builder) and sources (which hand them out as18+// they are). Text a file holds as a value comes out raw: an import19+// stores it as typed in its locale (textOrDate) and a source reads the20+// dates in it (typed), so both type a value the same way. Text 01221+// makes (joined lists, a blob's size) stays text.22+23+// fileCell is a value from a file, raw when it is text to be read as a24+// date where it is one.25+type fileCell struct {26+ sheet.LiveCell27+ raw bool28+}29+30+// liveNum is a number in format f; NaN and infinities, which no cell31+// holds, are #NUM!.32+func liveNum(v float64, f sheet.Format) sheet.LiveCell {33+ if math.IsNaN(v) || math.IsInf(v, 0) {34+ return sheet.LiveCell{V: sheet.ErrNum}35+ }36+ return sheet.LiveCell{V: sheet.Value{Kind: sheet.Number, Num: v}, F: f}37+}38+39+func strCell(s string) sheet.LiveCell {40+ return sheet.LiveCell{V: sheet.Value{Kind: sheet.Text, Str: s}}41+}42+43+func liveBool(b bool) sheet.LiveCell {44+ v := sheet.Value{Kind: sheet.Bool}45+ if b {46+ v.Num = 147+ }48+ return sheet.LiveCell{V: v}49+}50+51+// put stores a cell as an import does: raw text through textOrDate,52+// which reads dates in the import's locale.53+func (c fileCell) put(b *builder, a sheet.Addr) {54+ switch c.V.Kind {55+ case sheet.Empty:56+ case sheet.Number:57+ b.number(a, c.V.Num, c.F, sheet.Style{})58+ case sheet.Bool:59+ b.boolean(a, c.V.Num != 0, sheet.Style{})60+ case sheet.Text:61+ if c.raw {62+ textOrDate(b, a, c.V.Str)63+ } else {64+ b.text(a, c.V.Str, c.F, sheet.Style{})65+ }66+ default:67+ b.number(a, math.NaN(), c.F, sheet.Style{})68+ }69+}70+71+// typed is the cell as a source hands it out: raw text that is a date72+// or time (2026-09-26) as that date, as an import would store it, and73+// other text, numbers stored as text included, as text.74+func (c fileCell) typed() sheet.LiveCell {75+ if !c.raw {76+ return c.LiveCell77+ }78+ s := flatten(c.V.Str)79+ if s == "" {80+ return sheet.LiveCell{} // blank, as an import leaves it81+ }82+ if n, f, ok := sheet.ParseValue(s); ok {83+ switch f.Kind {84+ case sheet.FmtDate, sheet.FmtTime, sheet.FmtDateTime:85+ return sheet.LiveCell{V: sheet.Value{Kind: sheet.Number, Num: n}, F: f}86+ }87+ }88+ return strCell(s)89+}90+91+// parquetCells is a row's values of one column as a cell: empty for92+// null, several joined with commas (reporting that they were).93+func parquetCells(vals []parquet.Value, c parquetColumn) (cell fileCell, repeated bool) {94+ if len(vals) == 1 {95+ if vals[0].IsNull() {96+ return fileCell{}, false97+ }98+ return parquetCell(vals[0], c), false99+ }100+ parts := make([]string, 0, len(vals))101+ for _, v := range vals {102+ if !v.IsNull() {103+ parts = append(parts, parquetText(v, c))104+ }105+ }106+ if len(parts) == 0 {107+ return fileCell{}, false108+ }109+ return fileCell{LiveCell: strCell(strings.Join(parts, ", "))}, true110+}111+112+// parquetCell is one value that isn't null.113+func parquetCell(v parquet.Value, c parquetColumn) fileCell {114+ lc, ok := parquetValue(v, c)115+ if !ok {116+ return fileCell{LiveCell: strCell(parquetText(v, c)), raw: true}117+ }118+ return fileCell{LiveCell: lc}119+}120+121+// parquetValue is a value that isn't text, and false for text.122+func parquetValue(v parquet.Value, c parquetColumn) (sheet.LiveCell, bool) {123+ switch lt := c.logical.(type) {124+ case *format.DateType:125+ return liveNum(float64(v.Int32())+serialOf(time.Unix(0, 0).UTC()), sheet.Format{Kind: sheet.FmtDate}), true126+ case *format.TimestampType:127+ return liveNum(serialOf(timestamp(v.Int64(), lt.Unit)), sheet.Format{Kind: sheet.FmtDateTime}), true128+ case *format.TimeType:129+ d := timeUnit(lt.Unit) * time.Duration(timeInt(v))130+ return liveNum(d.Hours()/24, sheet.Format{Kind: sheet.FmtTime}), true131+ case *format.DecimalType:132+ if x, ok := decimal(v, lt.Scale); ok {133+ return liveNum(x, sheet.Format{}), true134+ }135+ }136+ switch c.kind {137+ case parquet.Boolean:138+ return liveBool(v.Boolean()), true139+ case parquet.Int32:140+ return liveNum(float64(v.Int32()), sheet.Format{}), true141+ case parquet.Int64:142+ return liveNum(float64(v.Int64()), sheet.Format{}), true143+ case parquet.Float:144+ return liveNum(float64(v.Float()), sheet.Format{}), true145+ case parquet.Double:146+ return liveNum(v.Double(), sheet.Format{}), true147+ case parquet.Int96:148+ // Legacy timestamps: nanoseconds of the day, then the Julian day.149+ i := v.Int96()150+ nanos := int64(i[1])<<32 | int64(i[0])151+ days := int64(i[2]) - 2440588 // the Julian day of 1970-01-01152+ return liveNum(serialOf(time.Unix(days*86400, nanos).UTC()), sheet.Format{Kind: sheet.FmtDateTime}), true153+ }154+ return sheet.LiveCell{}, false155+}156+157+// parquetFormat is the format a column's type shows its values in, and158+// whether it holds numbers.159+func parquetFormat(c parquetColumn) (sheet.Format, bool) {160+ switch c.logical.(type) {161+ case *format.DateType:162+ return sheet.Format{Kind: sheet.FmtDate}, true163+ case *format.TimestampType:164+ return sheet.Format{Kind: sheet.FmtDateTime}, true165+ case *format.TimeType:166+ return sheet.Format{Kind: sheet.FmtTime}, true167+ case *format.DecimalType:168+ return sheet.Format{}, true169+ }170+ switch c.kind {171+ case parquet.Int32, parquet.Int64, parquet.Float, parquet.Double:172+ return sheet.Format{}, true173+ case parquet.Int96:174+ return sheet.Format{Kind: sheet.FmtDateTime}, true175+ }176+ return sheet.Format{}, false177+}178+179+// sqliteCell is a database value as a cell, reporting whether it was180+// binary data that could only be described.181+func sqliteCell(v any) (cell fileCell, blob bool) {182+ switch v := v.(type) {183+ case nil:184+ return fileCell{}, false185+ case int64:186+ return fileCell{LiveCell: liveNum(float64(v), sheet.Format{})}, false187+ case float64:188+ return fileCell{LiveCell: liveNum(v, sheet.Format{})}, false189+ case bool:190+ return fileCell{LiveCell: liveBool(v)}, false191+ case time.Time:192+ f := sheet.Format{Kind: sheet.FmtDateTime}193+ if h, m, s := v.Clock(); h == 0 && m == 0 && s == 0 && v.Nanosecond() == 0 {194+ f = sheet.Format{Kind: sheet.FmtDate}195+ }196+ return fileCell{LiveCell: liveNum(serialOf(v), f)}, false197+ case string:198+ return fileCell{LiveCell: strCell(v), raw: true}, false199+ case []byte:200+ if utf8.Valid(v) {201+ return fileCell{LiveCell: strCell(string(v)), raw: true}, false202+ }203+ return fileCell{LiveCell: strCell(fmt.Sprintf("(%s)", count(len(v), "byte", "bytes")))}, true204+ }205+ return fileCell{LiveCell: strCell(fmt.Sprint(v))}, false206+}@@ -0,0 +1,288 @@
1+package fileio2+3+import (4+ "context"5+ "errors"6+ "io"7+ "os"8+ "slices"9+ "strings"10+ "sync"11+12+ "github.com/parquet-go/parquet-go"13+14+ "github.com/FelineStateMachine/012/internal/sheet"15+)16+17+// parquetSource is a Parquet file read in place. Each read opens its18+// own readers of the column chunks it needs (the file is read with19+// ReadAt), so reads on several goroutines don't meet.20+type parquetSource struct {21+ f *os.File22+ pf *parquet.File23+ groups []parquet.RowGroup24+ starts []int64 // each row group's first row, and the rows in all last25+ cols []parquetColumn26+ meta []SourceColumn27+ dir string28+29+ mu sync.Mutex30+ views []*rowFile // what views wrote, removed by Close31+}32+33+func openParquetSource(spec SourceSpec) (*parquetSource, error) {34+ f, err := os.Open(spec.Path)35+ if err != nil {36+ return nil, err37+ }38+ st, err := f.Stat()39+ if err != nil {40+ f.Close()41+ return nil, err42+ }43+ pf, err := parquet.OpenFile(f, st.Size())44+ if err != nil {45+ f.Close()46+ return nil, err47+ }48+ s := &parquetSource{f: f, pf: pf, groups: pf.RowGroups(), dir: spec.TempDir}49+ s.starts = make([]int64, len(s.groups)+1)50+ for i, g := range s.groups {51+ s.starts[i+1] = s.starts[i] + g.NumRows()52+ }53+ schema := pf.Schema()54+ for _, path := range schema.Columns() {55+ var c parquetColumn56+ if leaf, ok := schema.Lookup(path...); ok && leaf.Node != nil {57+ t := leaf.Node.Type()58+ c.kind = t.Kind()59+ c.repeated = leaf.MaxRepetitionLevel > 060+ if lt := t.LogicalType(); lt != nil {61+ c.logical = lt.Value62+ }63+ }64+ f, num := parquetFormat(c)65+ s.cols = append(s.cols, c)66+ s.meta = append(s.meta, SourceColumn{Name: strings.Join(path, "."), Format: f, Numeric: num})67+ }68+ return s, nil69+}70+71+func (s *parquetSource) Columns() []SourceColumn { return slices.Clone(s.meta) }72+73+func (s *parquetSource) Rows() int64 { return s.starts[len(s.starts)-1] }74+75+func (s *parquetSource) Close() error {76+ s.mu.Lock()77+ views := s.views78+ s.views = nil79+ s.mu.Unlock()80+ var errs []error81+ for _, v := range views {82+ errs = append(errs, v.close())83+ }84+ return errors.Join(append(errs, s.f.Close())...)85+}86+87+// group is the row group holding row.88+func (s *parquetSource) group(row int64) int {89+ i, found := slices.BinarySearch(s.starts, row)90+ if !found {91+ i--92+ }93+ return i94+}95+96+// Scan streams rows from row from on.97+func (s *parquetSource) Scan(ctx context.Context, from int64, cols []int, fn func(int64, []sheet.LiveCell) bool) error {98+ cols = allCols(cols, len(s.cols))99+ if err := checkCols(cols, len(s.cols)); err != nil {100+ return err101+ }102+ rs := s.newRows(cols)103+ defer rs.close()104+ vals := make([]sheet.LiveCell, len(cols))105+ for row := max(from, 0); row < s.Rows(); row++ {106+ if row%1024 == 0 {107+ if err := ctx.Err(); err != nil {108+ return err109+ }110+ }111+ if err := rs.read(row, vals); err != nil {112+ return err113+ }114+ if !fn(row, vals) {115+ return nil116+ }117+ }118+ return nil119+}120+121+// Fetch reads the rows numbered rows.122+func (s *parquetSource) Fetch(ctx context.Context, rows []int64, cols []int) ([][]sheet.LiveCell, error) {123+ cols = allCols(cols, len(s.cols))124+ if err := checkCols(cols, len(s.cols)); err != nil {125+ return nil, err126+ }127+ rs := s.newRows(cols)128+ defer rs.close()129+ out := make([][]sheet.LiveCell, len(rows))130+ for _, i := range fetchOrder(rows) {131+ if err := ctx.Err(); err != nil {132+ return nil, err133+ }134+ row := rows[i]135+ if row < 0 || row >= s.Rows() {136+ return nil, errors.New("no such row in the source")137+ }138+ vals := make([]sheet.LiveCell, len(cols))139+ if err := rs.read(row, vals); err != nil {140+ return nil, err141+ }142+ out[i] = vals143+ }144+ return out, nil145+}146+147+// pqRows reads rows of some columns, each through a cursor of its own.148+type pqRows struct {149+ s *parquetSource150+ curs []*pqCursor151+ buf []parquet.Value152+}153+154+func (s *parquetSource) newRows(cols []int) *pqRows {155+ rs := &pqRows{s: s}156+ for _, c := range cols {157+ rs.curs = append(rs.curs, &pqCursor{s: s, col: c, repeated: s.cols[c].repeated, group: -1})158+ }159+ return rs160+}161+162+// read reads row's values into vals.163+func (rs *pqRows) read(row int64, vals []sheet.LiveCell) error {164+ for i, c := range rs.curs {165+ var err error166+ if rs.buf, err = c.row(row, rs.buf[:0]); err != nil {167+ return err168+ }169+ cell, _ := parquetCells(rs.buf, rs.s.cols[c.col])170+ vals[i] = cell.typed()171+ }172+ return nil173+}174+175+func (rs *pqRows) close() {176+ for _, c := range rs.curs {177+ c.close()178+ }179+}180+181+// pqCursor reads one column's values a row at a time: from the page182+// readers of its chunk in the row group being read, sought to a row183+// when a read isn't the next one.184+type pqCursor struct {185+ s *parquetSource186+ col int187+ repeated bool188+ group int // the row group being read, -1 before any189+ next int64 // the row the next value starts190+ pages parquet.Pages191+ vals parquet.ValueReader192+ buf []parquet.Value193+ i, n int194+}195+196+// skipRows is how far ahead a cursor reads its way rather than seeking.197+const skipRows = 4096198+199+// row appends the values of row to out: one, or a repeated column's200+// several (none for an empty list, read as null).201+func (c *pqCursor) row(row int64, out []parquet.Value) ([]parquet.Value, error) {202+ if err := c.seek(row); err != nil {203+ return out, err204+ }205+ for first := true; ; first = false {206+ if c.i == c.n {207+ if err := c.fill(); err != nil {208+ if errors.Is(err, io.EOF) && !first {209+ break210+ }211+ return out, err212+ }213+ }214+ v := c.buf[c.i]215+ if !first && v.RepetitionLevel() == 0 {216+ break217+ }218+ if c.repeated {219+ v = v.Clone() // the next fill may reuse what it points at220+ }221+ out = append(out, v)222+ c.i++223+ }224+ c.next = row + 1225+ return out, nil226+}227+228+// seek places the cursor at row: reading forward to it when it is a229+// little ahead in the same group, else seeking the group's pages.230+func (c *pqCursor) seek(row int64) error {231+ if row == c.next && c.group >= 0 && row < c.s.starts[c.group+1] {232+ return nil // the next row, as a scan reads them233+ }234+ g := c.s.group(row)235+ if g == c.group && row >= c.next && row-c.next <= skipRows {236+ var skip []parquet.Value237+ for c.next < row {238+ var err error239+ if skip, err = c.row(c.next, skip[:0]); err != nil {240+ return err241+ }242+ }243+ return nil244+ }245+ c.close()246+ pages := c.s.groups[g].ColumnChunks()[c.col].Pages()247+ if err := pages.SeekToRow(row - c.s.starts[g]); err != nil {248+ pages.Close()249+ return err250+ }251+ c.pages, c.group, c.next, c.vals, c.i, c.n = pages, g, row, nil, 0, 0252+ return nil253+}254+255+// fill reads the next values of the group's pages, io.EOF past them.256+func (c *pqCursor) fill() error {257+ if c.buf == nil {258+ c.buf = make([]parquet.Value, 1024)259+ }260+ for {261+ if c.vals != nil {262+ n, err := c.vals.ReadValues(c.buf)263+ if n > 0 {264+ c.i, c.n = 0, n265+ return nil266+ }267+ if err != nil && !errors.Is(err, io.EOF) {268+ return err269+ }270+ c.vals = nil271+ }272+ if c.pages == nil {273+ return io.EOF274+ }275+ p, err := c.pages.ReadPage()276+ if err != nil {277+ return err278+ }279+ c.vals = p.Values()280+ }281+}282+283+func (c *pqCursor) close() {284+ if c.pages != nil {285+ c.pages.Close()286+ }287+ c.pages, c.vals, c.group, c.i, c.n = nil, nil, -1, 0, 0288+}@@ -0,0 +1,191 @@
1+package fileio2+3+import (4+ "bufio"5+ "context"6+ "errors"7+ "os"8+ "slices"9+10+ "github.com/FelineStateMachine/012/internal/sheet"11+)12+13+// Views of a Parquet file stream: building one reads the columns it14+// sorts and filters by once, writing the rows it keeps, in its order,15+// to a file of row numbers (sourcesort.go); a page reads its row16+// numbers there and fetches those rows.17+18+// View orders the file's rows as o says.19+func (s *parquetSource) View(ctx context.Context, o SourceOrder) (SourceView, error) {20+ if o.IsZero() {21+ return pqPlainView{s}, nil22+ }23+ plan, err := newViewPlan(o, len(s.cols))24+ if err != nil {25+ return nil, err26+ }27+ f, err := newRowFile(s.dir)28+ if err != nil {29+ return nil, err30+ }31+ rf := &rowFile{f: f}32+ if rf.n, err = plan.write(ctx, s, f); err != nil {33+ rf.close()34+ return nil, err35+ }36+ s.mu.Lock()37+ s.views = append(s.views, rf)38+ s.mu.Unlock()39+ return &pqView{s: s, rf: rf}, nil40+}41+42+// viewPlan is what building a streamed view reads: the columns it sorts43+// and filters by, once each, and where each key and test finds its44+// column among them.45+type viewPlan struct {46+ cols []int47+ keys []int // positions in cols of the sort's columns48+ desc []bool49+ tests []viewTest50+}51+52+type viewTest struct {53+ at int // the column's position in cols54+ pass func(sheet.Value, string) bool55+}56+57+func newViewPlan(o SourceOrder, ncols int) (*viewPlan, error) {58+ p := &viewPlan{}59+ at := func(col int) (int, error) {60+ if col < 0 || col >= ncols {61+ return 0, errors.New("the source has no such column")62+ }63+ if i := slices.Index(p.cols, col); i >= 0 {64+ return i, nil65+ }66+ p.cols = append(p.cols, col)67+ return len(p.cols) - 1, nil68+ }69+ for _, k := range o.Sort {70+ i, err := at(k.Col)71+ if err != nil {72+ return nil, err73+ }74+ p.keys, p.desc = append(p.keys, i), append(p.desc, k.Desc)75+ }76+ for _, f := range o.Filter {77+ i, err := at(f.Col)78+ if err != nil {79+ return nil, err80+ }81+ p.tests = append(p.tests, viewTest{at: i, pass: f.Cond.Test()})82+ }83+ return p, nil84+}85+86+// passes reports whether a row's values pass every test, each on the87+// value and the text it shows, as a sheet's filter tests them.88+func (p *viewPlan) passes(vals []sheet.LiveCell) bool {89+ for _, t := range p.tests {90+ c := vals[t.at]91+ if !t.pass(c.V, sheet.FormatText(c.V, c.F)) {92+ return false93+ }94+ }95+ return true96+}97+98+// write streams the source, writing the numbers of the rows the view99+// keeps to f in its order, and returns how many.100+func (p *viewPlan) write(ctx context.Context, s *parquetSource, f *os.File) (int64, error) {101+ w := bufio.NewWriterSize(f, 1<<20)102+ var n int64103+ var sorter *extSorter104+ if len(p.keys) > 0 {105+ sorter = newExtSorter(s.dir, p.desc)106+ defer sorter.remove()107+ }108+ keys := make([]sheet.LiveCell, len(p.keys))109+ var ferr error110+ err := s.Scan(ctx, 0, p.cols, func(row int64, vals []sheet.LiveCell) bool {111+ if !p.passes(vals) {112+ return true113+ }114+ if sorter == nil {115+ ferr = writeRow(w, row)116+ n++117+ return ferr == nil118+ }119+ for i, k := range p.keys {120+ keys[i] = vals[k]121+ }122+ ferr = sorter.add(row, keys)123+ return ferr == nil124+ })125+ if err = errors.Join(err, ferr); err != nil {126+ return 0, err127+ }128+ if sorter != nil {129+ if n, err = sorter.finish(w); err != nil {130+ return 0, err131+ }132+ }133+ return n, w.Flush()134+}135+136+// pqView is a sorted or filtered view of a Parquet file.137+type pqView struct {138+ s *parquetSource139+ rf *rowFile140+}141+142+func (v *pqView) Rows() int64 { return v.rf.n }143+144+func (v *pqView) Page(ctx context.Context, from int64, n int) ([]int64, [][]sheet.LiveCell, error) {145+ nums, err := v.rf.read(from, n)146+ if err != nil || len(nums) == 0 {147+ return nil, nil, err148+ }149+ rows, err := v.s.Fetch(ctx, nums, nil)150+ if err != nil {151+ return nil, nil, err152+ }153+ return nums, rows, nil154+}155+156+func (v *pqView) Close() error {157+ v.s.mu.Lock()158+ i := slices.Index(v.s.views, v.rf)159+ if i < 0 {160+ v.s.mu.Unlock()161+ return nil // closed with the source162+ }163+ v.s.views = slices.Delete(v.s.views, i, i+1)164+ v.s.mu.Unlock()165+ return v.rf.close()166+}167+168+// pqPlainView is the file's rows in its own order.169+type pqPlainView struct{ s *parquetSource }170+171+func (v pqPlainView) Rows() int64 { return v.s.Rows() }172+173+func (v pqPlainView) Page(ctx context.Context, from int64, n int) ([]int64, [][]sheet.LiveCell, error) {174+ return scanPage(ctx, v.s, from, n)175+}176+177+func (pqPlainView) Close() error { return nil }178+179+// scanPage reads n rows from row from of a source in its own order.180+func scanPage(ctx context.Context, s Source, from int64, n int) ([]int64, [][]sheet.LiveCell, error) {181+ if from < 0 || from >= s.Rows() || n <= 0 {182+ return nil, nil, nil183+ }184+ var nums []int64185+ var rows [][]sheet.LiveCell186+ err := s.Scan(ctx, from, nil, func(row int64, vals []sheet.LiveCell) bool {187+ nums, rows = append(nums, row), append(rows, slices.Clone(vals))188+ return len(nums) < n189+ })190+ return nums, rows, err191+}@@ -0,0 +1,364 @@
1+package fileio2+3+import (4+ "bufio"5+ "cmp"6+ "container/heap"7+ "encoding/binary"8+ "errors"9+ "io"10+ "math"11+ "os"12+ "slices"13+ "strings"14+15+ "github.com/FelineStateMachine/012/internal/sheet"16+)17+18+// A Parquet view's rows are a file of row numbers in the view's order,19+// 8 bytes each, which a page reads with ReadAt: a filter writes the rows20+// it lets through as it streams them, and a sort is an external merge21+// sort, runs of at most sortRunBytes sorted in memory and spilled to22+// files, then merged. Memory stays bounded whatever the row count; the23+// disk holds the runs and the order, about 8 bytes a row plus the keys.24+25+// sortRunBytes bounds the keys a sort holds in memory at once.26+var sortRunBytes = 64 << 2027+28+// sortVal is one key of a row as a sort compares it: its kind's rank29+// (numbers, text, booleans, errors, then blanks), and its number or its30+// text in lower case.31+type sortVal struct {32+ rank uint833+ num float6434+ str string35+}36+37+// blankRank is a blank's, last in either direction.38+const blankRank = 439+40+func sortValOf(v sheet.Value) sortVal {41+ switch v.Kind {42+ case sheet.Number:43+ return sortVal{rank: 0, num: v.Num}44+ case sheet.Text:45+ return sortVal{rank: 1, str: strings.ToLower(v.Str)}46+ case sheet.Bool:47+ return sortVal{rank: 2, num: v.Num}48+ case sheet.Error:49+ return sortVal{rank: 3, str: strings.ToLower(v.Str)}50+ }51+ return sortVal{rank: blankRank}52+}53+54+// sortRec is a row being sorted: its number and its keys.55+type sortRec struct {56+ row int6457+ keys []sortVal58+}59+60+// size is about what the record holds in memory.61+func (r *sortRec) size() int {62+ n := 48 + 32*len(r.keys)63+ for _, k := range r.keys {64+ n += len(k.str)65+ }66+ return n67+}68+69+// compareRecs orders rows as a sheet's sort does (sheet.compareRows):70+// key by key, blanks last in either direction, Z to A reversing the71+// rest, and ties in the source's order.72+func compareRecs(a, b *sortRec, desc []bool) int {73+ for i := range a.keys {74+ x, y := &a.keys[i], &b.keys[i]75+ if (x.rank == blankRank) != (y.rank == blankRank) {76+ if x.rank == blankRank {77+ return 178+ }79+ return -180+ }81+ d := cmp.Compare(x.rank, y.rank)82+ if d == 0 {83+ switch x.rank {84+ case 0, 2:85+ d = cmp.Compare(x.num, y.num)86+ case 1, 3:87+ d = strings.Compare(x.str, y.str)88+ }89+ }90+ if desc[i] {91+ d = -d92+ }93+ if d != 0 {94+ return d95+ }96+ }97+ return cmp.Compare(a.row, b.row)98+}99+100+// extSorter sorts rows by their keys in bounded memory.101+type extSorter struct {102+ dir string103+ desc []bool104+ recs []sortRec105+ bytes int106+ runs []string107+ limit int108+}109+110+func newExtSorter(dir string, desc []bool) *extSorter {111+ return &extSorter{dir: dir, desc: desc, limit: sortRunBytes}112+}113+114+// add takes a row's keys, spilling a run when memory is full.115+func (s *extSorter) add(row int64, keys []sheet.LiveCell) error {116+ r := sortRec{row: row, keys: make([]sortVal, len(keys))}117+ for i, k := range keys {118+ r.keys[i] = sortValOf(k.V)119+ }120+ s.recs = append(s.recs, r)121+ s.bytes += r.size()122+ if s.bytes >= s.limit {123+ return s.spill()124+ }125+ return nil126+}127+128+func (s *extSorter) sort() {129+ slices.SortFunc(s.recs, func(a, b sortRec) int { return compareRecs(&a, &b, s.desc) })130+}131+132+// spill sorts the rows in memory and writes them as a run.133+func (s *extSorter) spill() error {134+ s.sort()135+ f, err := os.CreateTemp(s.dir, "012-sort-*.run")136+ if err != nil {137+ return err138+ }139+ s.runs = append(s.runs, f.Name())140+ w := bufio.NewWriterSize(f, 1<<20)141+ for i := range s.recs {142+ writeRec(w, &s.recs[i])143+ }144+ err = errors.Join(w.Flush(), f.Close())145+ clear(s.recs)146+ s.recs, s.bytes = s.recs[:0], 0147+ return err148+}149+150+// finish writes the sorted rows' numbers to out and removes the runs.151+func (s *extSorter) finish(out io.Writer) (n int64, err error) {152+ defer s.remove()153+ if len(s.runs) == 0 {154+ s.sort()155+ for i := range s.recs {156+ if err := writeRow(out, s.recs[i].row); err != nil {157+ return n, err158+ }159+ n++160+ }161+ return n, nil162+ }163+ if len(s.recs) > 0 {164+ if err := s.spill(); err != nil {165+ return 0, err166+ }167+ }168+ return s.merge(out)169+}170+171+// remove deletes the runs.172+func (s *extSorter) remove() {173+ for _, name := range s.runs {174+ os.Remove(name)175+ }176+ s.runs = nil177+}178+179+// merge merges the runs into out.180+func (s *extSorter) merge(out io.Writer) (int64, error) {181+ h := &runHeap{desc: s.desc}182+ for _, name := range s.runs {183+ f, err := os.Open(name)184+ if err != nil {185+ h.close()186+ return 0, err187+ }188+ r := &runReader{f: f, r: bufio.NewReaderSize(f, 256<<10), keys: len(s.desc)}189+ if r.next() {190+ h.runs = append(h.runs, r)191+ } else {192+ f.Close()193+ }194+ }195+ defer h.close()196+ heap.Init(h)197+ var n int64198+ for h.Len() > 0 {199+ r := h.runs[0]200+ if err := writeRow(out, r.rec.row); err != nil {201+ return n, err202+ }203+ n++204+ if r.next() {205+ heap.Fix(h, 0)206+ } else {207+ r.f.Close()208+ heap.Pop(h)209+ }210+ }211+ return n, h.err()212+}213+214+func writeRow(w io.Writer, row int64) error {215+ var b [8]byte216+ binary.LittleEndian.PutUint64(b[:], uint64(row))217+ _, err := w.Write(b[:])218+ return err219+}220+221+// writeRec writes a record to a run: its row, then each key's rank and222+// its number or text.223+func writeRec(w *bufio.Writer, r *sortRec) {224+ var b [binary.MaxVarintLen64]byte225+ w.Write(b[:binary.PutUvarint(b[:], uint64(r.row))])226+ for _, k := range r.keys {227+ w.WriteByte(k.rank)228+ switch k.rank {229+ case 0, 2:230+ binary.LittleEndian.PutUint64(b[:8], math.Float64bits(k.num))231+ w.Write(b[:8])232+ case 1, 3:233+ w.Write(b[:binary.PutUvarint(b[:], uint64(len(k.str)))])234+ w.WriteString(k.str)235+ }236+ }237+}238+239+// runReader reads a run's records in order.240+type runReader struct {241+ f *os.File242+ r *bufio.Reader243+ keys int244+ rec sortRec245+ fail error246+}247+248+// next reads the next record, false at the end or on an error.249+func (r *runReader) next() bool {250+ row, err := binary.ReadUvarint(r.r)251+ if err != nil {252+ if !errors.Is(err, io.EOF) {253+ r.fail = err254+ }255+ return false256+ }257+ r.rec.row = int64(row)258+ r.rec.keys = r.rec.keys[:0]259+ for range r.keys {260+ k, err := r.key()261+ if err != nil {262+ r.fail = err263+ return false264+ }265+ r.rec.keys = append(r.rec.keys, k)266+ }267+ return true268+}269+270+func (r *runReader) key() (sortVal, error) {271+ rank, err := r.r.ReadByte()272+ if err != nil {273+ return sortVal{}, err274+ }275+ k := sortVal{rank: rank}276+ switch rank {277+ case 0, 2:278+ var b [8]byte279+ if _, err := io.ReadFull(r.r, b[:]); err != nil {280+ return k, err281+ }282+ k.num = math.Float64frombits(binary.LittleEndian.Uint64(b[:]))283+ case 1, 3:284+ n, err := binary.ReadUvarint(r.r)285+ if err != nil {286+ return k, err287+ }288+ b := make([]byte, n)289+ if _, err := io.ReadFull(r.r, b); err != nil {290+ return k, err291+ }292+ k.str = string(b)293+ }294+ return k, nil295+}296+297+// runHeap merges runs by their next record.298+type runHeap struct {299+ runs []*runReader300+ desc []bool301+ done []*runReader302+}303+304+func (h *runHeap) Len() int { return len(h.runs) }305+func (h *runHeap) Less(i, j int) bool {306+ return compareRecs(&h.runs[i].rec, &h.runs[j].rec, h.desc) < 0307+}308+func (h *runHeap) Swap(i, j int) { h.runs[i], h.runs[j] = h.runs[j], h.runs[i] }309+func (h *runHeap) Push(x any) { h.runs = append(h.runs, x.(*runReader)) }310+func (h *runHeap) Pop() any {311+ r := h.runs[len(h.runs)-1]312+ h.runs = h.runs[:len(h.runs)-1]313+ h.done = append(h.done, r)314+ return r315+}316+317+// err is the first error a run met.318+func (h *runHeap) err() error {319+ for _, r := range append(h.done, h.runs...) {320+ if r.fail != nil {321+ return r.fail322+ }323+ }324+ return nil325+}326+327+func (h *runHeap) close() {328+ for _, r := range h.runs {329+ r.f.Close()330+ }331+}332+333+// rowFile is a view's rows as a file of row numbers in its order.334+type rowFile struct {335+ f *os.File336+ n int64337+}338+339+// newRowFile makes the file rows are written to, in dir.340+func newRowFile(dir string) (*os.File, error) { return os.CreateTemp(dir, "012-view-*.rows") }341+342+// read returns the row numbers at positions from to from+n, as many as343+// there are.344+func (rf *rowFile) read(from int64, n int) ([]int64, error) {345+ if from < 0 || from >= rf.n || n <= 0 {346+ return nil, nil347+ }348+ n = int(min(int64(n), rf.n-from))349+ b := make([]byte, 8*n)350+ if _, err := rf.f.ReadAt(b, 8*from); err != nil {351+ return nil, err352+ }353+ out := make([]int64, n)354+ for i := range out {355+ out[i] = int64(binary.LittleEndian.Uint64(b[8*i:]))356+ }357+ return out, nil358+}359+360+// close closes and removes the file.361+func (rf *rowFile) close() error {362+ name := rf.f.Name()363+ return errors.Join(rf.f.Close(), os.Remove(name))364+}@@ -0,0 +1,209 @@
1+package fileio2+3+import (4+ "context"5+ "database/sql"6+ "fmt"7+ "os"8+ "path/filepath"9+ "slices"10+ "testing"11+12+ "github.com/FelineStateMachine/012/internal/sheet"13+)14+15+// writeSalesDB writes a database whose table sales has n rows (fewer,16+// with rowids missing, when sparse), a view of it, a WITHOUT ROWID17+// table, and a column of mixed kinds with no declared type.18+func writeSalesDB(t testing.TB, dir string, n int, sparse bool) string {19+ t.Helper()20+ name := filepath.Join(dir, fmt.Sprintf("sales-%d-%v.sqlite", n, sparse))21+ db, err := sql.Open("sqlite", name)22+ if err != nil {23+ t.Fatal(err)24+ }25+ defer db.Close()26+ tx, err := db.Begin()27+ if err != nil {28+ t.Fatal(err)29+ }30+ for _, q := range []string{31+ "CREATE TABLE sales (id INTEGER, cat TEXT, amount REAL, day TEXT, mixed)",32+ "CREATE VIEW big AS SELECT * FROM sales WHERE amount > 50",33+ "CREATE TABLE keyed (k TEXT PRIMARY KEY, v REAL) WITHOUT ROWID",34+ "CREATE TABLE other (x)",35+ } {36+ if _, err := tx.Exec(q); err != nil {37+ t.Fatal(err)38+ }39+ }40+ ins, err := tx.Prepare("INSERT INTO sales VALUES (?, ?, ?, ?, ?)")41+ if err != nil {42+ t.Fatal(err)43+ }44+ for i := range n {45+ var cat, amount, mixed any46+ if i%9 != 4 {47+ cat = saleCats[i%len(saleCats)]48+ }49+ if i%11 != 7 {50+ amount = float64((i*37)%101) - 20.551+ }52+ switch i % 6 {53+ case 0:54+ mixed = int64(i % 13)55+ case 1:56+ mixed = fmt.Sprintf("t%d", i%7)57+ case 2:58+ mixed = 2.559+ case 3:60+ mixed = "12"61+ case 4:62+ mixed = ""63+ }64+ day := fmt.Sprintf("2026-01-%02d", 1+i%28)65+ if _, err := ins.Exec(i, cat, amount, day, mixed); err != nil {66+ t.Fatal(err)67+ }68+ if _, err := tx.Exec("INSERT INTO keyed VALUES (?, ?)", fmt.Sprintf("k%05d", n-i), i); err != nil {69+ t.Fatal(err)70+ }71+ }72+ if sparse {73+ if _, err := tx.Exec("DELETE FROM sales WHERE id % 7 = 3"); err != nil {74+ t.Fatal(err)75+ }76+ }77+ if err := tx.Commit(); err != nil {78+ t.Fatal(err)79+ }80+ return name81+}82+83+// sqlOrders are orders only a SQLite source's table has columns for.84+var sqlOrders = []SourceOrder{85+ {Sort: []SourceSort{{Col: 4}}},86+ {Sort: []SourceSort{{Col: 4, Desc: true}}},87+ {Filter: []SourceFilter{{Col: 4, Cond: sheet.Condition{Op: sheet.CondGreater, Arg: "2"}}}},88+ {Filter: []SourceFilter{{Col: 4, Cond: sheet.Condition{Op: sheet.CondLess, Arg: "T3"}}}},89+ {Filter: []SourceFilter{{Col: 4, Cond: sheet.Condition{Op: sheet.CondEqual, Arg: "12"}}}},90+}91+92+func TestSQLiteSource(t *testing.T) {93+ dir := t.TempDir()94+ for _, sparse := range []bool{false, true} {95+ t.Run(fmt.Sprint("sparse=", sparse), func(t *testing.T) { checkSQLiteTable(t, writeSalesDB(t, dir, 700, sparse)) })96+ }97+}98+99+// checkSQLiteTable reads the table sales of name every way a source100+// reads, as the sheet its import makes shows it.101+func checkSQLiteTable(t *testing.T, name string) {102+ tmp := t.TempDir()103+ src, err := OpenSource(context.Background(), SourceSpec{Path: name, Table: "sales", TempDir: tmp})104+ if err != nil {105+ t.Fatal(err)106+ }107+ s, r := imported(t, name, Options{Table: "sales"})108+ want := sheetRows(s, r)109+ if src.Rows() != int64(len(want)) {110+ t.Fatalf("%d rows, want %d", src.Rows(), len(want))111+ }112+ var cols []string113+ for _, c := range src.Columns() {114+ cols = append(cols, fmt.Sprintf("%s:%v", c.Name, c.Numeric))115+ }116+ if !slices.Equal(cols, []string{"id:true", "cat:false", "amount:true", "day:false", "mixed:false"}) {117+ t.Errorf("columns %q", cols)118+ }119+ var got []string120+ err = src.Scan(context.Background(), 300, nil, func(row int64, v []sheet.LiveCell) bool {121+ got = append(got, cellsText(v))122+ return true123+ })124+ if err != nil || !slices.Equal(got, want[300:]) {125+ t.Errorf("scan from 300: %v %q", err, head(got))126+ }127+ rows, err := src.Fetch(context.Background(), []int64{int64(len(want) - 1), 0, 350}, nil)128+ if err != nil {129+ t.Fatal(err)130+ }131+ for i, n := range []int{len(want) - 1, 0, 350} {132+ if g := cellsText(rows[i]); g != want[n] {133+ t.Errorf("row %d is %q, want %q", n, g, want[n])134+ }135+ }136+ checkOrders(t, src, name, Options{Table: "sales"}, append(slices.Clone(orders), sqlOrders...))137+ if err := src.Close(); err != nil {138+ t.Fatal(err)139+ }140+ if left, _ := os.ReadDir(tmp); len(left) != 0 {141+ t.Errorf("closing left %d files", len(left))142+ }143+}144+145+// TestSQLiteSourceKinds reads a view, a WITHOUT ROWID table and a146+// query, which are paged through a copy in the work database.147+func TestSQLiteSourceKinds(t *testing.T) {148+ dir := t.TempDir()149+ name := writeSalesDB(t, dir, 300, false)150+ for _, spec := range []SourceSpec{151+ {Path: name, Table: "big"},152+ {Path: name, Table: "keyed"},153+ {Path: name, Query: "SELECT cat, sum(amount) AS total, count(*) AS n FROM sales GROUP BY cat"},154+ {Path: name, Query: "SELECT id, id FROM sales WHERE id < 40"},155+ } {156+ spec.TempDir = t.TempDir()157+ src, err := OpenSource(context.Background(), spec)158+ if err != nil {159+ t.Fatalf("%+v: %v", spec, err)160+ }161+ s, r := imported(t, name, Options{Table: spec.Table, Query: spec.Query})162+ want := sheetRows(s, r)163+ v, err := src.View(context.Background(), SourceOrder{})164+ if err != nil {165+ t.Fatal(err)166+ }167+ if _, got := viewRows(t, v, 50); !slices.Equal(got, want) {168+ t.Errorf("%+v: %q, want %q", spec, head(got), head(want))169+ }170+ sorted := []SourceOrder{{Sort: []SourceSort{{Col: 1, Desc: true}}}}171+ checkOrders(t, src, name, Options{Table: spec.Table, Query: spec.Query}, sorted)172+ src.Close()173+ if left, _ := os.ReadDir(spec.TempDir); len(left) != 0 {174+ t.Errorf("%+v: closing left %d files", spec, len(left))175+ }176+ }177+ if _, err := OpenSource(context.Background(), SourceSpec{Path: name}); err == nil {178+ t.Error("a database of several tables opened without naming one")179+ }180+ if _, err := OpenSource(context.Background(), SourceSpec{Path: name, Query: "SELECT nope FROM sales"}); err == nil {181+ t.Error("a query that fails opened")182+ }183+}184+185+// TestSQLiteSourceReadOnly checks the database file is never written.186+func TestSQLiteSourceReadOnly(t *testing.T) {187+ name := writeSalesDB(t, t.TempDir(), 200, true)188+ before, err := os.ReadFile(name)189+ if err != nil {190+ t.Fatal(err)191+ }192+ src, err := OpenSource(context.Background(), SourceSpec{Path: name, Table: "sales", TempDir: t.TempDir()})193+ if err != nil {194+ t.Fatal(err)195+ }196+ v, err := src.View(context.Background(), SourceOrder{Sort: []SourceSort{{Col: 2}}})197+ if err != nil {198+ t.Fatal(err)199+ }200+ v.Page(context.Background(), 100, 20)201+ src.Close()202+ after, _ := os.ReadFile(name)203+ if !slices.Equal(before, after) {204+ t.Error("the database changed")205+ }206+ if _, err := os.Stat(name + "-journal"); err == nil {207+ t.Error("the database has a journal")208+ }209+}@@ -0,0 +1,329 @@
1+package fileio2+3+import (4+ "context"5+ "database/sql"6+ "errors"7+ "fmt"8+ "net/url"9+ "os"10+ "path/filepath"11+ "slices"12+ "strconv"13+ "strings"14+ "sync"15+16+ "github.com/FelineStateMachine/012/internal/sheet"17+)18+19+// sqliteSource is a SQLite table, view or query read in place, through20+// connections that can only read (mode=ro, query_only), so the database21+// is never written. Its rows are in rowid order for a table, and as the22+// query or view gives them otherwise.23+//24+// A row's number is found cheaply only in a table whose rowids are 1 to25+// its row count (dense), the common case of a table only ever appended26+// to: row n is rowid n+1. Any other source is copied, when something27+// first asks for rows by number, into a table of a work database in28+// the temporary directory (sourcesqlwork.go), where it is dense; views'29+// row orders are tables there too.30+type sqliteSource struct {31+ spec SourceSpec32+ db *sql.DB33+ query string // the SELECT giving the source's rows, in order34+ table string // the table, quoted, when the source is one with rowids35+ names []string36+ meta []SourceColumn37+ rows int6438+ dense bool39+40+ mu sync.Mutex41+ work *sqlWork // made when first needed42+ base *sqlBase // where rows are found by number, once known43+}44+45+// sqliteURI is the URI opening a database file to read only.46+func sqliteURI(path string, params ...string) (string, error) {47+ abs, err := filepath.Abs(path)48+ if err != nil {49+ return "", err50+ }51+ u := url.URL{Scheme: "file", Path: filepath.ToSlash(abs), RawQuery: strings.Join(params, "&")}52+ return u.String(), nil53+}54+55+func openSQLiteSource(ctx context.Context, spec SourceSpec) (*sqliteSource, error) {56+ if _, err := os.Stat(spec.Path); err != nil {57+ return nil, err58+ }59+ uri, err := sqliteURI(spec.Path, "mode=ro", "_pragma=query_only(1)", "_pragma=busy_timeout(10000)")60+ if err != nil {61+ return nil, err62+ }63+ db, err := sql.Open("sqlite", uri)64+ if err != nil {65+ return nil, err66+ }67+ s := &sqliteSource{spec: spec, db: db}68+ if err := s.open(ctx); err != nil {69+ db.Close()70+ return nil, err71+ }72+ return s, nil73+}74+75+// open finds what the source reads, its columns and its row count.76+func (s *sqliteSource) open(ctx context.Context) error {77+ s.query = s.spec.Query78+ if s.query == "" {79+ table := s.spec.Table80+ if table == "" {81+ ts, err := tables(ctx, s.db)82+ switch {83+ case err != nil:84+ return err85+ case len(ts) == 0:86+ return errors.New("the database has no tables")87+ case len(ts) > 1:88+ return &ErrNeedTable{Tables: ts}89+ }90+ table = ts[0].Name91+ }92+ s.query = "SELECT * FROM " + quoteIdent(table)93+ if s.hasRowid(ctx, table) {94+ s.table = quoteIdent(table)95+ s.query += " ORDER BY rowid"96+ }97+ }98+ if err := s.columns(ctx); err != nil {99+ return err100+ }101+ return s.count(ctx)102+}103+104+// hasRowid reports whether table is a table with rowids (not a view or105+// a WITHOUT ROWID table).106+func (s *sqliteSource) hasRowid(ctx context.Context, table string) bool {107+ rows, err := s.db.QueryContext(ctx, "SELECT rowid FROM "+quoteIdent(table)+" LIMIT 0")108+ if err != nil {109+ return false110+ }111+ rows.Close()112+ var typ string113+ err = s.db.QueryRowContext(ctx, "SELECT type FROM sqlite_master WHERE name = ?", table).Scan(&typ)114+ return err == nil && typ == "table"115+}116+117+// count counts the rows, and for a table whether its rowids are dense.118+func (s *sqliteSource) count(ctx context.Context) error {119+ if s.table == "" {120+ err := s.db.QueryRowContext(ctx, "SELECT count(*) FROM ("+s.query+")").Scan(&s.rows)121+ return sqliteErr(err)122+ }123+ var lo, hi sql.NullInt64124+ err := s.db.QueryRowContext(ctx, "SELECT count(*), min(rowid), max(rowid) FROM "+s.table).Scan(&s.rows, &lo, &hi)125+ if err != nil {126+ return sqliteErr(err)127+ }128+ s.dense = s.rows == 0 || lo.Int64 == 1 && hi.Int64 == s.rows129+ return nil130+}131+132+// columns reads the columns' names and types: a declared type says133+// whether one holds numbers or dates; an undeclared one (a query's134+// expressions) is judged by its first rows.135+func (s *sqliteSource) columns(ctx context.Context) error {136+ rows, err := s.db.QueryContext(ctx, "SELECT * FROM ("+s.query+") LIMIT 0")137+ if err != nil {138+ return sqliteErr(err)139+ }140+ types, err := rows.ColumnTypes()141+ rows.Close()142+ if err != nil {143+ return sqliteErr(err)144+ }145+ var untyped []int146+ for i, t := range types {147+ c := SourceColumn{Name: t.Name()}148+ decl := strings.ToUpper(t.DatabaseTypeName())149+ switch {150+ case decl == "":151+ untyped = append(untyped, i)152+ case strings.Contains(decl, "DATETIME"), strings.Contains(decl, "TIMESTAMP"):153+ c.Format, c.Numeric = sheet.Format{Kind: sheet.FmtDateTime}, true154+ case strings.Contains(decl, "DATE"):155+ c.Format, c.Numeric = sheet.Format{Kind: sheet.FmtDate}, true156+ default:157+ c.Numeric = numericAffinity(decl)158+ }159+ s.names = append(s.names, t.Name())160+ s.meta = append(s.meta, c)161+ }162+ if len(untyped) > 0 {163+ return s.sample(ctx, untyped)164+ }165+ return nil166+}167+168+// numericAffinity reports whether a declared type gives a column169+// SQLite's INTEGER, REAL or NUMERIC affinity.170+func numericAffinity(decl string) bool {171+ switch {172+ case strings.Contains(decl, "CHAR"), strings.Contains(decl, "CLOB"), strings.Contains(decl, "TEXT"),173+ strings.Contains(decl, "BLOB"):174+ return false175+ }176+ return true177+}178+179+// sampleRows is how many rows judge an undeclared column's type.180+const sampleRows = 100181+182+// sample marks the columns cols numeric when their first rows hold183+// numbers and nothing else.184+func (s *sqliteSource) sample(ctx context.Context, cols []int) error {185+ num := make([]bool, len(s.meta))186+ seen := make([]bool, len(s.meta))187+ for _, c := range cols {188+ num[c] = true189+ }190+ err := s.scanSQL(ctx, "SELECT * FROM ("+s.query+") LIMIT "+strconv.Itoa(sampleRows), nil, func(vals []any) bool {191+ for _, c := range cols {192+ switch vals[c].(type) {193+ case nil:194+ case int64, float64:195+ seen[c] = true196+ default:197+ num[c] = false198+ }199+ }200+ return true201+ })202+ for _, c := range cols {203+ s.meta[c].Numeric = num[c] && seen[c]204+ }205+ return err206+}207+208+func (s *sqliteSource) Columns() []SourceColumn { return slices.Clone(s.meta) }209+210+func (s *sqliteSource) Rows() int64 { return s.rows }211+212+// selectCols is the select list reading cols, and whether the columns213+// come back as asked: a table's by name, anything else whole, picked214+// from.215+func (s *sqliteSource) selectCols(cols []int) (string, bool) {216+ if s.table == "" {217+ return "*", false218+ }219+ names := make([]string, len(cols))220+ for i, c := range cols {221+ names[i] = quoteIdent(s.names[c])222+ }223+ return strings.Join(names, ", "), true224+}225+226+// Scan streams the rows from row from on.227+func (s *sqliteSource) Scan(ctx context.Context, from int64, cols []int, fn func(int64, []sheet.LiveCell) bool) error {228+ cols = allCols(cols, len(s.names))229+ if err := checkCols(cols, len(s.names)); err != nil {230+ return err231+ }232+ list, picked := s.selectCols(cols)233+ var q string234+ var args []any235+ switch {236+ case s.table != "" && s.dense:237+ q, args = "SELECT "+list+" FROM "+s.table+" WHERE rowid > ? ORDER BY rowid", []any{from}238+ case s.table != "":239+ q, args = "SELECT "+list+" FROM "+s.table+" ORDER BY rowid LIMIT -1 OFFSET ?", []any{from}240+ default:241+ q, args = "SELECT * FROM ("+s.query+") LIMIT -1 OFFSET ?", []any{from}242+ }243+ out := make([]sheet.LiveCell, len(cols))244+ row := max(from, 0)245+ return s.scanSQL(ctx, q, args, func(vals []any) bool {246+ for i, c := range cols {247+ v := vals[i]248+ if !picked {249+ v = vals[c]250+ }251+ cell, _ := sqliteCell(v)252+ out[i] = cell.typed()253+ }254+ row++255+ return fn(row-1, out)256+ })257+}258+259+// scanSQL runs a query on the source, calling fn with each row's values260+// until it returns false.261+func (s *sqliteSource) scanSQL(ctx context.Context, q string, args []any, fn func([]any) bool) error {262+ return queryRows(ctx, s.db, q, args, fn)263+}264+265+// queryRows runs a query, calling fn with each row's values (reused)266+// until it returns false.267+func queryRows(ctx context.Context, db interface {268+ QueryContext(context.Context, string, ...any) (*sql.Rows, error)269+}, q string, args []any, fn func([]any) bool) error {270+ rows, err := db.QueryContext(ctx, q, args...)271+ if err != nil {272+ return sqliteErr(err)273+ }274+ defer rows.Close()275+ cols, err := rows.Columns()276+ if err != nil {277+ return sqliteErr(err)278+ }279+ vals := make([]any, len(cols))280+ ptrs := make([]any, len(cols))281+ for i := range vals {282+ ptrs[i] = &vals[i]283+ }284+ for rows.Next() {285+ if err := rows.Scan(ptrs...); err != nil {286+ return sqliteErr(err)287+ }288+ if !fn(vals) {289+ return nil290+ }291+ }292+ if err := rows.Err(); err != nil {293+ if ctx.Err() != nil {294+ return ctx.Err()295+ }296+ return sqliteErr(err)297+ }298+ return nil299+}300+301+// Fetch reads the rows numbered rows.302+func (s *sqliteSource) Fetch(ctx context.Context, rows []int64, cols []int) ([][]sheet.LiveCell, error) {303+ cols = allCols(cols, len(s.names))304+ if err := checkCols(cols, len(s.names)); err != nil {305+ return nil, err306+ }307+ for _, r := range rows {308+ if r < 0 || r >= s.rows {309+ return nil, fmt.Errorf("the source has no row %d", r+1)310+ }311+ }312+ b, err := s.byNumber(ctx)313+ if err != nil {314+ return nil, err315+ }316+ return b.fetch(ctx, rows, cols)317+}318+319+func (s *sqliteSource) Close() error {320+ s.mu.Lock()321+ w := s.work322+ s.work, s.base = nil, nil323+ s.mu.Unlock()324+ var err error325+ if w != nil {326+ err = w.close()327+ }328+ return errors.Join(err, s.db.Close())329+}@@ -0,0 +1,204 @@
1+package fileio2+3+import (4+ "context"5+ "fmt"6+ "strings"7+8+ "github.com/FelineStateMachine/012/internal/sheet"9+)10+11+// SQLite views push their sort and filter down to SQL: building one is12+// one INSERT ... SELECT rowid ... WHERE ... ORDER BY into a table of the13+// work database (sourcesqlwork.go), and a page reads its rowids there,14+// by position, then the rows.15+//16+// The SQL follows a sheet's sort and filter, with these differences:17+// SQLite folds case in ASCII only; text holding a date (2026-09-26),18+// which the source shows as that date, is compared and sorted as text;19+// and the text conditions (contains, starts with ...) test a number's20+// text as SQLite writes it (1e+20), where a sheet tests it as shown.21+22+// sqlStmt is a statement and its arguments.23+type sqlStmt struct {24+ q string25+ args []any26+}27+28+// View orders the rows as o says.29+func (s *sqliteSource) View(ctx context.Context, o SourceOrder) (SourceView, error) {30+ b, err := s.byNumber(ctx)31+ if err != nil {32+ return nil, err33+ }34+ if o.IsZero() {35+ return sqlPlainView{s: s, b: b}, nil36+ }37+ where, args, order, err := viewSQL(o, b.cols)38+ if err != nil {39+ return nil, err40+ }41+ s.mu.Lock()42+ w, err := s.workDB()43+ s.mu.Unlock()44+ if err != nil {45+ return nil, err46+ }47+ name := fmt.Sprintf("_012_v%d", w.n.Add(1))48+ from := b.table49+ if !b.work {50+ from = "src." + b.table51+ }52+ n, err := w.build(ctx, s.spec.Path,53+ sqlStmt{q: "CREATE TABLE " + name + " (r INTEGER)"},54+ sqlStmt{q: "INSERT INTO " + name + " (r) SELECT rowid FROM " + from + where + " ORDER BY " + order, args: args})55+ if err != nil {56+ w.db.ExecContext(context.WithoutCancel(ctx), "DROP TABLE IF EXISTS "+name)57+ return nil, err58+ }59+ return &sqlView{s: s, b: b, w: w, name: name, rows: n}, nil60+}61+62+// viewSQL is a view's WHERE clause (with its leading space), its63+// arguments and its ORDER BY, over the columns cols.64+func viewSQL(o SourceOrder, cols []string) (string, []any, string, error) {65+ var conds []string66+ var args []any67+ for _, f := range o.Filter {68+ if f.Col < 0 || f.Col >= len(cols) {69+ return "", nil, "", fmt.Errorf("the source has no column %d", f.Col+1)70+ }71+ c, a := condSQL(cols[f.Col], f.Cond)72+ conds, args = append(conds, c), append(args, a...)73+ }74+ var keys []string75+ for _, k := range o.Sort {76+ if k.Col < 0 || k.Col >= len(cols) {77+ return "", nil, "", fmt.Errorf("the source has no column %d", k.Col+1)78+ }79+ keys = append(keys, orderSQL(cols[k.Col], k.Desc))80+ }81+ where := ""82+ if len(conds) > 0 {83+ where = " WHERE " + strings.Join(conds, " AND ")84+ }85+ return where, args, strings.Join(append(keys, "rowid"), ", "), nil86+}87+88+// orderSQL orders column x as a sheet sorts: numbers, text ignoring89+// case, then the rest (booleans are numbers in SQLite, blobs last), and90+// blanks last in either direction.91+func orderSQL(x string, desc bool) string {92+ dir := ""93+ if desc {94+ dir = " DESC"95+ }96+ v := "nullif(" + x + ", '')" // a blank, NULL or '', is NULL97+ return "(" + v + " IS NULL), " +98+ "CASE typeof(" + v + ") WHEN 'null' THEN 0 WHEN 'integer' THEN 0 WHEN 'real' THEN 0 WHEN 'text' THEN 1 ELSE 2 END" + dir + ", " +99+ v + " COLLATE NOCASE" + dir100+}101+102+// condSQL is a filter's condition on column x as SQL, as103+// sheet.Condition tests a cell: text conditions on the text a value104+// shows, blank for NULL; comparisons with a number (or a date, $1,200,105+// 12%) only of numbers, with text only of text ignoring case; neither106+// matching a blank.107+func condSQL(x string, c sheet.Condition) (string, []any) {108+ shown := "lower(coalesce(CAST(" + x + " AS TEXT), ''))"109+ arg := strings.ToLower(c.Arg)110+ switch c.Op {111+ case sheet.CondEmpty:112+ return "(" + x + " IS NULL OR " + x + " = '')", nil113+ case sheet.CondNotEmpty:114+ return "(" + x + " IS NOT NULL AND " + x + " <> '')", nil115+ case sheet.CondContains:116+ return "instr(" + shown + ", ?) > 0", []any{arg}117+ case sheet.CondNotContains:118+ return "instr(" + shown + ", ?) = 0", []any{arg}119+ case sheet.CondStartsWith:120+ return "substr(" + shown + ", 1, length(?)) = ?", []any{arg, arg}121+ case sheet.CondEndsWith:122+ return "(length(?) = 0 OR substr(" + shown + ", -length(?)) = ?)", []any{arg, arg, arg}123+ case sheet.CondExactly:124+ return shown + " = ?", []any{arg}125+ case sheet.CondGreater, sheet.CondGreaterEq, sheet.CondLess, sheet.CondLessEq, sheet.CondEqual:126+ return compareSQL(x, c)127+ case sheet.CondNotEqual:128+ c.Op = sheet.CondEqual129+ q, args := compareSQL(x, c)130+ return "NOT " + q, args131+ }132+ return "1", nil133+}134+135+var compareOps = map[sheet.CondOp]string{136+ sheet.CondGreater: ">", sheet.CondGreaterEq: ">=", sheet.CondLess: "<", sheet.CondLessEq: "<=", sheet.CondEqual: "=",137+}138+139+// compareSQL compares column x with a condition's value as the sheet's140+// filter does: a number with numbers, text with text.141+func compareSQL(x string, c sheet.Condition) (string, []any) {142+ op := compareOps[c.Op]143+ if n, _, ok := sheet.ParseValue(strings.TrimSpace(c.Arg)); ok {144+ return "(typeof(" + x + ") IN ('integer', 'real') AND " + x + " " + op + " ?)", []any{n}145+ }146+ return "(typeof(" + x + ") = 'text' AND " + x + " <> '' AND upper(" + x + ") " + op + " upper(?))", []any{c.Arg}147+}148+149+// sqlView is a sorted or filtered view: its rows' rowids in a table of150+// the work database.151+type sqlView struct {152+ s *sqliteSource153+ b *sqlBase154+ w *sqlWork155+ name string156+ rows int64157+}158+159+func (v *sqlView) Rows() int64 { return v.rows }160+161+func (v *sqlView) Page(ctx context.Context, from int64, n int) ([]int64, [][]sheet.LiveCell, error) {162+ if from < 0 || from >= v.rows || n <= 0 {163+ return nil, nil, nil164+ }165+ var nums []int64166+ err := queryRows(ctx, v.w.db, "SELECT r FROM "+v.name+" WHERE rowid BETWEEN ? AND ? ORDER BY rowid",167+ []any{from + 1, from + int64(n)}, func(x []any) bool {168+ r, _ := x[0].(int64)169+ nums = append(nums, r-1)170+ return true171+ })172+ if err != nil {173+ return nil, nil, err174+ }175+ rows, err := v.b.fetch(ctx, nums, allCols(nil, len(v.b.cols)))176+ if err != nil {177+ return nil, nil, err178+ }179+ return nums, rows, nil180+}181+182+func (v *sqlView) Close() error {183+ v.s.mu.Lock()184+ defer v.s.mu.Unlock()185+ if v.s.work != v.w {186+ return nil // closed with the source187+ }188+ _, err := v.w.db.Exec("DROP TABLE IF EXISTS " + v.name)189+ return sqliteErr(err)190+}191+192+// sqlPlainView is the rows in the source's own order.193+type sqlPlainView struct {194+ s *sqliteSource195+ b *sqlBase196+}197+198+func (v sqlPlainView) Rows() int64 { return v.s.rows }199+200+func (v sqlPlainView) Page(ctx context.Context, from int64, n int) ([]int64, [][]sheet.LiveCell, error) {201+ return v.b.page(ctx, from, n, v.s.rows)202+}203+204+func (sqlPlainView) Close() error { return nil }@@ -0,0 +1,216 @@
1+package fileio2+3+import (4+ "context"5+ "database/sql"6+ "errors"7+ "fmt"8+ "os"9+ "strings"10+ "sync/atomic"11+12+ "github.com/FelineStateMachine/012/internal/sheet"13+)14+15+// A SQLite source's work database: a file of its own in the temporary16+// directory, never the source's, holding what reading rows by number17+// needs: a copy of a source whose rows can't be found by number18+// otherwise (_012_rows), and each view's rows in its order as a table of19+// rowids (_012_v1, ...), paged by that table's own rowid, so a page20+// deep in a view costs what a page at its top does. Tables are built21+// in SQLite, the source attached read-only as src; SQLite sorts in22+// temporary files, so memory stays bounded. The file goes when the23+// source closes.24+25+type sqlWork struct {26+ path string27+ db *sql.DB28+ n atomic.Int64 // views made, naming their tables29+}30+31+func newSQLWork(dir string) (*sqlWork, error) {32+ f, err := os.CreateTemp(dir, "012-source-*.db")33+ if err != nil {34+ return nil, err35+ }36+ f.Close()37+ uri, err := sqliteURI(f.Name(), "_pragma=journal_mode(OFF)", "_pragma=synchronous(OFF)",38+ "_pragma=busy_timeout(60000)", "_pragma=temp_store(1)")39+ if err != nil {40+ os.Remove(f.Name())41+ return nil, err42+ }43+ db, err := sql.Open("sqlite", uri)44+ if err != nil {45+ os.Remove(f.Name())46+ return nil, err47+ }48+ return &sqlWork{path: f.Name(), db: db}, nil49+}50+51+func (w *sqlWork) close() error {52+ err := w.db.Close()53+ for _, suffix := range []string{"", "-journal", "-wal", "-shm"} {54+ if rerr := os.Remove(w.path + suffix); rerr != nil && !errors.Is(rerr, os.ErrNotExist) {55+ err = errors.Join(err, rerr)56+ }57+ }58+ return err59+}60+61+// build runs statements on a connection of the work database with the62+// source attached as src.63+func (w *sqlWork) build(ctx context.Context, source string, stmts ...sqlStmt) (int64, error) {64+ conn, err := w.db.Conn(ctx)65+ if err != nil {66+ return 0, err67+ }68+ defer conn.Close()69+ uri, err := sqliteURI(source, "mode=ro")70+ if err != nil {71+ return 0, err72+ }73+ if _, err := conn.ExecContext(ctx, "ATTACH DATABASE ? AS src", uri); err != nil {74+ return 0, sqliteErr(err)75+ }76+ defer conn.ExecContext(context.WithoutCancel(ctx), "DETACH DATABASE src")77+ var n int6478+ for _, q := range stmts {79+ res, err := conn.ExecContext(ctx, q.q, q.args...)80+ if err != nil {81+ return 0, sqliteErr(err)82+ }83+ n, _ = res.RowsAffected()84+ }85+ return n, nil86+}87+88+// workDB is the source's work database, made when first needed. It is89+// called with s.mu held.90+func (s *sqliteSource) workDB() (*sqlWork, error) {91+ if s.work == nil {92+ w, err := newSQLWork(s.spec.TempDir)93+ if err != nil {94+ return nil, err95+ }96+ s.work = w97+ }98+ return s.work, nil99+}100+101+// sqlBase is a table whose rowids are its rows' numbers plus one: a102+// dense source table, or the copy in the work database.103+type sqlBase struct {104+ db *sql.DB105+ table string // quoted106+ cols []string // its columns, quoted107+ work bool // in the work database108+}109+110+// byNumber is where rows are found by number, copying the source into111+// the work database the first time when it must.112+func (s *sqliteSource) byNumber(ctx context.Context) (*sqlBase, error) {113+ s.mu.Lock()114+ defer s.mu.Unlock()115+ if s.base != nil {116+ return s.base, nil117+ }118+ if s.table != "" && s.dense {119+ s.base = &sqlBase{db: s.db, table: s.table, cols: quoteAll(s.names)}120+ return s.base, nil121+ }122+ w, err := s.workDB()123+ if err != nil {124+ return nil, err125+ }126+ if _, err := w.build(ctx, s.spec.Path, sqlStmt{q: "CREATE TABLE _012_rows AS " + s.query}); err != nil {127+ w.db.ExecContext(context.WithoutCancel(ctx), "DROP TABLE IF EXISTS _012_rows")128+ return nil, err129+ }130+ var names []string131+ err = queryRows(ctx, w.db, "SELECT name FROM pragma_table_info('_012_rows') ORDER BY cid", nil, func(v []any) bool {132+ names = append(names, fmt.Sprint(v[0]))133+ return true134+ })135+ if err != nil {136+ return nil, err137+ }138+ s.base = &sqlBase{db: w.db, table: "_012_rows", cols: quoteAll(names), work: true}139+ return s.base, nil140+}141+142+func quoteAll(names []string) []string {143+ out := make([]string, len(names))144+ for i, n := range names {145+ out[i] = quoteIdent(n)146+ }147+ return out148+}149+150+// list is the select list of cols after the rowid.151+func (b *sqlBase) list(cols []int) string {152+ parts := make([]string, 0, len(cols)+1)153+ parts = append(parts, "rowid")154+ for _, c := range cols {155+ parts = append(parts, b.cols[c])156+ }157+ return strings.Join(parts, ", ")158+}159+160+// fetchChunk is how many rowids one query asks for.161+const fetchChunk = 500162+163+// fetch reads the rows numbered rows, in the order given.164+func (b *sqlBase) fetch(ctx context.Context, rows []int64, cols []int) ([][]sheet.LiveCell, error) {165+ got := make(map[int64][]sheet.LiveCell, len(rows))166+ for i := 0; i < len(rows); i += fetchChunk {167+ part := rows[i:min(i+fetchChunk, len(rows))]168+ marks := strings.Repeat("?, ", len(part))169+ args := make([]any, len(part))170+ for j, r := range part {171+ args[j] = r + 1172+ }173+ q := "SELECT " + b.list(cols) + " FROM " + b.table + " WHERE rowid IN (" + marks[:len(marks)-2] + ")"174+ if err := b.collect(ctx, q, args, got); err != nil {175+ return nil, err176+ }177+ }178+ out := make([][]sheet.LiveCell, len(rows))179+ for i, r := range rows {180+ out[i] = got[r]181+ }182+ return out, nil183+}184+185+// page reads n rows from row from on.186+func (b *sqlBase) page(ctx context.Context, from int64, n int, rows int64) ([]int64, [][]sheet.LiveCell, error) {187+ if from < 0 || from >= rows || n <= 0 {188+ return nil, nil, nil189+ }190+ got := map[int64][]sheet.LiveCell{}191+ q := "SELECT " + b.list(allCols(nil, len(b.cols))) + " FROM " + b.table + " WHERE rowid BETWEEN ? AND ?"192+ if err := b.collect(ctx, q, []any{from + 1, from + int64(n)}, got); err != nil {193+ return nil, nil, err194+ }195+ var nums []int64196+ var out [][]sheet.LiveCell197+ for r := from; r < min(from+int64(n), rows); r++ {198+ nums, out = append(nums, r), append(out, got[r])199+ }200+ return nums, out, nil201+}202+203+// collect runs a query of a rowid and values, keeping each row's204+// values by its number.205+func (b *sqlBase) collect(ctx context.Context, q string, args []any, got map[int64][]sheet.LiveCell) error {206+ return queryRows(ctx, b.db, q, args, func(v []any) bool {207+ cells := make([]sheet.LiveCell, len(v)-1)208+ for i, x := range v[1:] {209+ c, _ := sqliteCell(x)210+ cells[i] = c.typed()211+ }212+ id, _ := v[0].(int64)213+ got[id-1] = cells214+ return true215+ })216+}@@ -9,7 +9,6 @@ import (
99 "os"1010 "strings"1111 "time"12- "unicode/utf8"1312 1413 _ "modernc.org/sqlite" // registers the "sqlite" driver1514 @@ -45,6 +44,9 @@ func openSQLite(ctx context.Context, name string) (*sql.DB, error) {
4544 // sqliteErr shortens the driver's errors, e.g. "file is not a database4645 // (26)" for a file that isn't SQLite.4746 func sqliteErr(err error) error {47+ if err == nil {48+ return nil49+ }4850 msg := err.Error()4951 if i := strings.Index(msg, ": "); i >= 0 && strings.HasPrefix(msg, "SQL logic error") {5052 msg = msg[i+2:]@@ -194,7 +196,9 @@ func readSQLiteRows(ctx context.Context, rows *sql.Rows, b *builder, ncols, tota
194196 return 0, 0, sqliteErr(err)195197 }196198 for c, v := range vals {197- if putValue(b, sheet.Addr{Col: c, Row: row}, v) {199+ cell, blob := sqliteCell(v)200+ cell.put(b, sheet.Addr{Col: c, Row: row})201+ if blob {198202 blobs++199203 }200204 }@@ -207,38 +211,6 @@ func readSQLiteRows(ctx context.Context, rows *sql.Rows, b *builder, ncols, tota
207211 return row, blobs, nil208212 }209213 210-// putValue stores a database value, reporting whether it was binary data211-// that could only be described.212-func putValue(b *builder, a sheet.Addr, v any) (blob bool) {213- switch v := v.(type) {214- case nil:215- case int64:216- b.number(a, float64(v), sheet.Format{}, sheet.Style{})217- case float64:218- b.number(a, v, sheet.Format{}, sheet.Style{})219- case bool:220- b.boolean(a, v, sheet.Style{})221- case time.Time:222- f := sheet.Format{Kind: sheet.FmtDateTime}223- if h, m, s := v.Clock(); h == 0 && m == 0 && s == 0 && v.Nanosecond() == 0 {224- f = sheet.Format{Kind: sheet.FmtDate}225- }226- b.number(a, serialOf(v), f, sheet.Style{})227- case string:228- textOrDate(b, a, v)229- case []byte:230- if utf8.Valid(v) {231- textOrDate(b, a, string(v))232- return false233- }234- b.text(a, fmt.Sprintf("(%s)", count(len(v), "byte", "bytes")), sheet.Format{}, sheet.Style{})235- return true236- default:237- b.text(a, fmt.Sprint(v), sheet.Format{}, sheet.Style{})238- }239- return false240-}241-242214 // textOrDate stores database text as text, except dates and times such243215 // as 2026-09-26, which become dates as they would when typed. Numbers244216 // stored as text (zip codes, IDs) stay text.@@ -152,6 +152,8 @@ func eval(n Node, get lookup) Value {
152152 var v Value153153 if get.lift > 0 && f.arrays != takesArrays {154154 v = liftCall(f, n.Args, get)155+ } else if sv, ok := get.streamed(f, n.Args); ok {156+ v = sv155157 } else {156158 want := get.wantArr157159 get.wantArr = f.arrays != liftScalar@@ -183,6 +185,8 @@ func evalOther(n Node, get lookup) Value {
183185 return get.invoke(n)184186 case liftArg:185187 return n.one(get)188+ case constArg:189+ return n.v186190 }187191 return value.ErrValue188192 }@@ -227,8 +227,14 @@ func xlookup(args []Node, get lookup) Value {
227227 return value.ErrValue228228 }229229 q := lineSeq(look, get, look.cols == 1)230- order := q.order(search < 0)231- best := -1230+ best, streamed := -1, false231+ if get.stream && q.sparse {232+ best, streamed = q.seekStream(key, mode, search < 0)233+ }234+ var order []int235+ if !streamed {236+ best, order = -1, q.order(search < 0)237+ }232238 for _, i := range order {233239 v := look.at(i)234240 if lookupEqual(key, v, mode == 2) {@@ -191,6 +191,9 @@ func sumIfTerms(args []Node, get lookup, add func(float64)) *Value {
191191 return &value.ErrValue192192 }193193 }194+ if streamable(get, rng, sum) {195+ return sumStream(get, sum, []matrix{rng}, []criterion{newCriterion(cv)}, add)196+ }194197 return sumMasked(sum, applyCriteria(get, []matrix{rng}, []criterion{newCriterion(cv)}, sum), add)195198 }196199 @@ -204,6 +207,8 @@ func sumIfsTerms(args []Node, get lookup, add func(float64)) *Value {
204207 return err205208 case len(ms) == 0 || ms[0].rows != sum.rows || ms[0].cols != sum.cols:206209 return &value.ErrValue210+ case streamable(get, append(ms, sum)...):211+ return sumStream(get, sum, ms, cs, add)207212 }208213 return sumMasked(sum, applyCriteria(get, ms, cs, sum), add)209214 }@@ -218,6 +223,9 @@ func averageIf(args []Node, get lookup) Value {
218223 if len(args) > 2 {219224 avg = matrixArg(args[2], get).resized(rng.rows, rng.cols, get)220225 }226+ if streamable(get, rng, avg) && avg.rows == rng.rows && avg.cols == rng.cols {227+ return averageStream(get, avg, []matrix{rng}, []criterion{newCriterion(cv)})228+ }221229 return averageMasked(avg, applyCriteria(get, []matrix{rng}, []criterion{newCriterion(cv)}, avg))222230 }223231 @@ -229,6 +237,8 @@ func averageIfs(args []Node, get lookup) Value {
229237 return *err230238 case len(ms) == 0 || ms[0].rows != avg.rows || ms[0].cols != avg.cols:231239 return value.ErrValue240+ case streamable(get, append(ms, avg)...):241+ return averageStream(get, avg, ms, cs)232242 }233243 return averageMasked(avg, applyCriteria(get, ms, cs, avg))234244 }@@ -238,6 +248,9 @@ func countIfs(args []Node, get lookup) Value {
238248 if err != nil {239249 return *err240250 }251+ if streamable(get, ms...) {252+ return countStream(get, ms, cs)253+ }241254 k := applyCriteria(get, ms, cs)242255 n := len(k.pass)243256 if k.tailPass {@@ -248,6 +261,9 @@ func countIfs(args []Node, get lookup) Value {
248261 249262 func countBlank(args []Node, get lookup) Value {250263 m := matrixArg(args[0], get)264+ if streamable(get, m) {265+ return blankStream(get, m)266+ }251267 isBlank := func(v Value) bool { return v.Kind == value.Empty || v.Kind == value.Text && v.Str == "" }252268 cells := cellsOf(get, m)253269 n := 0@@ -274,6 +290,9 @@ func sumProductTerms(args []Node, get lookup, term func(fs []float64)) *Value {
274290 return &value.ErrValue275291 }276292 }293+ if streamable(get, ms...) {294+ return productStream(get, ms, term)295+ }277296 cells := cellsOf(get, ms...)278297 fs := make([]float64, len(ms))279298 for _, pc := range cells {@@ -40,6 +40,12 @@ type Book interface {
4040 // value is an error when there is none to give: Pending while it is4141 // on its way, ErrNoRemote when nothing answers them.4242 Ask(call RemoteCall) (RemoteAnswer, Value)43+ // Paged reports whether the sheet is a paged source (stream.go),44+ // whose ranges a function is given through Stream.45+ Paged(sheet string) bool46+ // Stream answers a call over paged sources, with an array when it47+ // computes several values: Pending while the answer is on its way.48+ Stream(call StreamCall) (Value, *Array)4349 }4450 4551 // Reader is what a function is given to read the cells its arguments@@ -49,8 +55,13 @@ type Reader struct {
4955 book Book5056 depth *int // how deeply evaluation is nested; see the engine's evaluate.go5157 dense bool // read every address of a range; see NewReader52- bufs []*scanBuf53- level int // how many of bufs are in use by range reads in progress58+ // stream is set on a Reader computing a StreamCall, over a Book that59+ // streams paged sources: functions over aligned ranges read them a60+ // window of rows at a time (streamed.go) rather than gathering the61+ // positions of their cells.62+ stream bool63+ bufs []*scanBuf64+ level int // how many of bufs are in use by range reads in progress5465 5566 evalState5667 // arena holds the arrays and LAMBDAs of the formulas being evaluated@@ -61,6 +61,9 @@ func (b *fakeBook) Bounds(string, Rect) (Rect, bool, bool) { return Rect{},
6161 func (b *fakeBook) RangeAgg(string, Rect) (Agg, *Value, bool) { return Agg{}, nil, false }6262 func (b *fakeBook) Fold(_ string, _ Rect, s Agg) (Agg, *Value) { return s, nil }6363 func (b *fakeBook) Ask(RemoteCall) (RemoteAnswer, Value) { return RemoteAnswer{}, ErrNoRemote }64+func (b *fakeBook) Paged(string) bool { return false }65+func (b *fakeBook) Stream(StreamCall) (Value, *Array) { return value.ErrValue, nil }66+6467 func newFakeReader(cells map[Addr]Value) (*Reader, *fakeBook) {6568 b := &fakeBook{cells: cells}6669 depth := 0@@ -0,0 +1,210 @@
1+package functions2+3+import (4+ "fmt"5+ "strings"6+7+ "github.com/FelineStateMachine/012/internal/formula"8+ "github.com/FelineStateMachine/012/internal/value"9+)10+11+// Calls over paged sources. A linked source (the engine's paged sheets:12+// a Parquet file or a SQLite table read in place) is too big to hold,13+// so a function given one of its ranges isn't evaluated where the14+// formula is: the Reader describes the call, its source ranges and the15+// values of its other arguments, as a StreamCall and asks the Book for16+// the answer, which is worked out in the background by EvalStream over17+// a Book that streams the source, and kept until the file changes.18+// Until it is known the cell shows Loading…, as a JEV question does.19+//20+// The functions in streaming read any size in one pass and hold only21+// what they compute; any other reads what its ranges hold, so it is22+// given a source range only while that holds no more cells than the23+// budget (max-cells), and past it answers #VALUE! saying why.24+25+// StreamCall is a call of a function some of whose arguments are ranges26+// of paged sheets. It isn't comparable: Key identifies it by content.27+type StreamCall struct {28+ Fn string29+ // Dec is set when the call is computed in decimal (decimal.go).30+ Dec bool31+ Args []StreamArg32+}33+34+// StreamArg is an argument of a StreamCall: a range of a paged sheet35+// (Sheet set, as the engine names the source), or a value.36+type StreamArg struct {37+ Sheet string38+ R Rect39+ V Value40+}41+42+// StreamAnswer is what a StreamCall computes: a value, or an array of43+// them, and for an error that needs one, why.44+type StreamAnswer struct {45+ V Value46+ A *Array47+ Why string48+}49+50+// Key identifies the call by its content.51+func (c StreamCall) Key() string {52+ var b strings.Builder53+ b.WriteString(c.Fn)54+ if c.Dec {55+ b.WriteString("#dec")56+ }57+ for _, a := range c.Args {58+ b.WriteByte('\x1f')59+ if a.Sheet != "" {60+ fmt.Fprintf(&b, "r%s\x1e%d:%d:%d:%d", a.Sheet, a.R.From.Col, a.R.From.Row, a.R.To.Col, a.R.To.Row)61+ continue62+ }63+ fmt.Fprintf(&b, "v%d\x1e%v\x1e%s", a.V.Kind, a.V.Num, a.V.Str)64+ }65+ return b.String()66+}67+68+// streaming are the functions that read a source range of any size in69+// one pass, holding only what they compute: aggregates, criteria70+// functions and lookups.71+var streaming = map[string]bool{72+ "SUM": true, "AVERAGE": true, "COUNT": true, "COUNTA": true, "MIN": true, "MAX": true, "PRODUCT": true,73+ "COUNTIF": true, "COUNTIFS": true, "SUMIF": true, "SUMIFS": true, "AVERAGEIF": true, "AVERAGEIFS": true,74+ "COUNTBLANK": true, "SUMPRODUCT": true,75+ "MATCH": true, "XLOOKUP": true, "VLOOKUP": true, "HLOOKUP": true, "INDEX": true, "ROWS": true, "COLUMNS": true,76+}77+78+// Streams reports whether the function named name reads a source of79+// any size (see streaming).80+func Streams(name string) bool { return streaming[name] }81+82+// StreamingFuncs lists the functions that read a source of any size, in83+// alphabetical order, for the docs and messages.84+func StreamingFuncs() []string {85+ var out []string86+ for _, f := range Funcs() {87+ if streaming[f.Name] {88+ out = append(out, f.Name)89+ }90+ }91+ return out92+}93+94+// streamed asks the Book for the call of f with args when one of its95+// arguments is a range of a paged sheet, reporting whether it asked. A96+// call whose other arguments aren't single values (a range of another97+// sheet, an array) is evaluated as usual, reading the source through98+// the Book's cells, which it holds only up to the budget.99+func (rd *Reader) streamed(f *FuncDef, args []Node) (Value, bool) {100+ if rd.stream || f.remote != nil || f.binds != formula.BindNone || !rd.pagedArg(args) {101+ return Value{}, false102+ }103+ call := StreamCall{Fn: f.Name, Dec: funcs[f.Name] != f, Args: make([]StreamArg, len(args))}104+ for i, a := range args {105+ switch r := rd.refOf(a).(type) {106+ case formula.Range:107+ if r.Sheet == "" || !rd.book.Paged(r.Sheet) {108+ return Value{}, false109+ }110+ call.Args[i] = StreamArg{Sheet: r.Sheet, R: r.Rect}111+ continue112+ }113+ v := eval1(a, rd)114+ if v.Kind == value.Array {115+ return Value{}, false116+ }117+ call.Args[i] = StreamArg{V: v}118+ }119+ v, a := rd.book.Stream(call)120+ if a != nil {121+ return rd.arrayValue(a), true122+ }123+ return v, true124+}125+126+// pagedArg reports whether an argument is a range of a paged sheet.127+func (rd *Reader) pagedArg(args []Node) bool {128+ for _, a := range args {129+ if r, ok := rd.refOf(a).(formula.Range); ok && r.Sheet != "" && rd.book.Paged(r.Sheet) {130+ return true131+ }132+ }133+ return false134+}135+136+// constArg is a StreamCall's value argument, as a node.137+type constArg struct{ v Value }138+139+// EvalStream computes c over book, a Book that reads the paged sheets140+// its ranges name by streaming them: in the background, where reading141+// takes as long as the source is big. A function not in streaming is142+// given the source's ranges only up to budget cells.143+func EvalStream(c StreamCall, book Book, budget int) StreamAnswer {144+ f, ok := LookupFunc(c.Fn)145+ if !ok {146+ return StreamAnswer{V: value.ErrName}147+ }148+ if twin, ok := decFuncs()[c.Fn]; ok && c.Dec {149+ f = twin150+ }151+ if why := overBudget(c, book, budget); why != "" {152+ return StreamAnswer{V: value.ErrValue, Why: why}153+ }154+ args := make([]Node, len(c.Args))155+ const fixed = formula.AbsCol | formula.AbsRow156+ for i, a := range c.Args {157+ if a.Sheet != "" {158+ args[i] = formula.Range{Rect: a.R, Sheet: a.Sheet, Abs: [2]formula.Abs{fixed, fixed}}159+ } else {160+ args[i] = constArg{a.V}161+ }162+ }163+ depth := 0164+ rd := NewReader(book, &depth, false)165+ rd.stream = true166+ v := EvalCell(formula.Call{Fn: f, Args: args}, rd, Addr{})167+ return StreamAnswer{V: v, A: rd.Spilled()}168+}169+170+// overBudget says why c can't be computed, when it names a function171+// that doesn't stream over more cells of its sources than budget.172+func overBudget(c StreamCall, book Book, budget int) string {173+ if streaming[c.Fn] {174+ return ""175+ }176+ cells, most := 0, 0177+ biggest := ""178+ for _, a := range c.Args {179+ if a.Sheet == "" {180+ continue181+ }182+ b, any, _ := book.Bounds(a.Sheet, a.R)183+ if !any {184+ continue185+ }186+ n := (b.To.Row - b.From.Row + 1) * (b.To.Col - b.From.Col + 1)187+ if cells += n; n > most {188+ most, biggest = n, a.Sheet189+ }190+ }191+ if cells <= budget {192+ return ""193+ }194+ return fmt.Sprintf("%s would hold %s cells of %s, past max-cells (%s): SUMIFS, XLOOKUP and the like stream them instead",195+ c.Fn, grouped(cells), biggest, grouped(budget))196+}197+198+// grouped is n with thousands separators: 12,000,000.199+func grouped(n int) string {200+ s := fmt.Sprint(n)201+ for i := len(s) - 3; i > 0 && s[i-1] != '-'; i -= 3 {202+ s = s[:i] + "," + s[i:]203+ }204+ return s205+}206+207+// Const is a node that evaluates to v: what the engine binds a name to208+// while what it names can't be read yet (a source being opened shows209+// Loading…).210+func Const(v Value) Node { return constArg{v} }@@ -0,0 +1,225 @@
1+package functions2+3+import "github.com/FelineStateMachine/012/internal/value"4+5+// Functions over aligned ranges computing a StreamCall (stream.go). On6+// a sheet they gather the positions of the cells their ranges hold and7+// visit those (masked.go), which over a source of ten million rows8+// would hold ten million positions. Computing a StreamCall they read9+// their ranges instead a window of rows at a time, every cell of the10+// window's rows, blank or not, testing and adding as they go, so they11+// hold one window whatever the source's size. Blank cells meet12+// criteria or add nothing as they would when accounted for at once, so13+// the results are the same.14+15+// streamRows is how many rows of each range a window holds.16+const streamRows = 204817+18+// streamable reports whether same-sized matrices are read a window at19+// a time: computing a StreamCall, every one a range of a sheet that20+// exists.21+func streamable(get lookup, ms ...matrix) bool {22+ if !get.stream {23+ return false24+ }25+ for _, m := range ms {26+ if !m.ref || m.blank.Kind == value.Error {27+ return false28+ }29+ }30+ return true31+}32+33+// windows calls fn with each window of the rows of same-sized ranges34+// that may hold data, the values of each range's window row by row (the35+// slices are reused), until fn returns false. It returns how many cells36+// of each range the windows cover; the rest are blank in all of them.37+func windows(get lookup, ms []matrix, fn func(vals [][]Value) bool) int {38+ rows, cols := 0, ms[0].cols39+ for _, m := range ms {40+ rows = max(rows, min(m.dataRows, m.rows))41+ }42+ bufs := make([][]Value, len(ms))43+ for r0 := 0; r0 < rows; r0 += streamRows {44+ n := min(streamRows, rows-r0)45+ for i, m := range ms {46+ bufs[i] = m.window(get, r0, n, bufs[i])47+ }48+ if !fn(bufs) {49+ break50+ }51+ }52+ return rows * cols53+}54+55+// window reads n rows of the range m from its row r0 into buf, every56+// cell of them, blanks as blank values.57+func (m matrix) window(get lookup, r0, n int, buf []Value) []Value {58+ size := n * m.cols59+ if cap(buf) < size {60+ buf = make([]Value, size)61+ }62+ buf = buf[:size]63+ clear(buf)64+ top := m.origin.Row + r065+ r := Rect{From: Addr{Col: m.origin.Col, Row: top}, To: Addr{Col: m.origin.Col + m.cols - 1, Row: top + n - 1}}66+ get.cells(m.sheet, r, func(a Addr, v Value) bool {67+ buf[(a.Row-top)*m.cols+a.Col-m.origin.Col] = v68+ return true69+ })70+ return buf71+}72+73+// passAll reports whether the cell at i of each window meets its74+// criterion.75+func passAll(vals [][]Value, cs []criterion, i int) bool {76+ for j, c := range cs {77+ if !c.test(vals[j][i]) {78+ return false79+ }80+ }81+ return true82+}83+84+// blanksPass reports whether blank cells meet every criterion.85+func blanksPass(cs []criterion) bool {86+ for _, c := range cs {87+ if !c.test(Value{}) {88+ return false89+ }90+ }91+ return true92+}93+94+// countStream is COUNTIFS over windows.95+func countStream(get lookup, ms []matrix, cs []criterion) Value {96+ n := 097+ covered := windows(get, ms, func(vals [][]Value) bool {98+ for i := range vals[0] {99+ if passAll(vals, cs, i) {100+ n++101+ }102+ }103+ return true104+ })105+ if blanksPass(cs) {106+ n += ms[0].size() - covered107+ }108+ return num(float64(n))109+}110+111+// sumStream calls add with the numbers of val where the criteria on ms112+// pass, over windows, or returns the first error there.113+func sumStream(get lookup, val matrix, ms []matrix, cs []criterion, add func(float64)) *Value {114+ var err *Value115+ windows(get, append([]matrix{val}, ms...), func(vals [][]Value) bool {116+ for i, v := range vals[0] {117+ if !passAll(vals[1:], cs, i) {118+ continue119+ }120+ switch v.Kind {121+ case value.Error:122+ err = &v123+ return false124+ case value.Number:125+ add(v.Num)126+ }127+ }128+ return true129+ })130+ return err131+}132+133+// averageStream is AVERAGEIFS over windows.134+func averageStream(get lookup, val matrix, ms []matrix, cs []criterion) Value {135+ sum, n := 0.0, 0136+ if e := sumStream(get, val, ms, cs, func(f float64) { sum += f; n++ }); e != nil {137+ return *e138+ }139+ if n == 0 {140+ return value.ErrDiv0141+ }142+ return num(sum / float64(n))143+}144+145+// blankStream is COUNTBLANK over windows.146+func blankStream(get lookup, m matrix) Value {147+ n := 0148+ covered := windows(get, []matrix{m}, func(vals [][]Value) bool {149+ for _, v := range vals[0] {150+ if v.Kind == value.Empty || v.Kind == value.Text && v.Str == "" {151+ n++152+ }153+ }154+ return true155+ })156+ return num(float64(n + m.size() - covered))157+}158+159+// productStream calls term with the factors of each entry of160+// same-sized ranges, over windows, as sumProductTerms does.161+func productStream(get lookup, ms []matrix, term func(fs []float64)) *Value {162+ var err *Value163+ fs := make([]float64, len(ms))164+ windows(get, ms, func(vals [][]Value) bool {165+ for i := range vals[0] {166+ for j := range ms {167+ switch v := vals[j][i]; v.Kind {168+ case value.Error:169+ err = &v170+ return false171+ case value.Number, value.Bool:172+ fs[j] = v.Num173+ default:174+ fs[j] = 0175+ }176+ }177+ term(fs)178+ }179+ return true180+ })181+ return err182+}183+184+// seekStream is XLOOKUP's search of a sparse line computing a185+// StreamCall: one pass over the cells it holds, forwards whichever way186+// the search goes, keeping what the search in order would find (the187+// first exact match, or the last searching backwards; else the closest188+// value, the first or last of equals) without listing the entries. ok189+// is false when a blank would match the key, which the search in order190+// handles.191+func (q *seq) seekStream(key Value, mode int, reverse bool) (int, bool) {192+ wild := mode == 2193+ if lookupEqual(key, q.blank, wild) {194+ return 0, false195+ }196+ exact, best := -1, -1197+ var bestV Value198+ c := q.walk()199+ defer c.close()200+ for a, ok := c.next(); ok; a, ok = c.next() {201+ i, v := q.index(a), q.value(a)202+ switch {203+ case lookupEqual(key, v, wild):204+ if !reverse {205+ return i, true206+ }207+ exact = i208+ case exact >= 0 || mode == 0 || wild || v.Kind != key.Kind:209+ case compare(v, key)*mode > 0 && (best < 0 || closer(v, bestV, mode, reverse)):210+ best, bestV = i, v211+ }212+ }213+ if exact >= 0 {214+ return exact, true215+ }216+ return best, true217+}218+219+// closer reports whether v is closer to the key than the best so far,220+// in XLOOKUP's next smaller (-1) or next larger (1) mode: equals keep221+// the first met, or the last searching backwards.222+func closer(v, best Value, mode int, reverse bool) bool {223+ d := compare(v, best) * mode224+ return d < 0 || reverse && d == 0225+}@@ -18,7 +18,7 @@ type Description struct {
1818 // SheetDescription is one tab: a sheet or a notebook.1919 type SheetDescription struct {2020 Name string `json:"name"`21- Kind string `json:"kind"` // "sheet", "notebook" or "pivot"21+ Kind string `json:"kind"` // "sheet", "notebook", "pivot" or "source"2222 Shown bool `json:"shown"`2323 Hidden bool `json:"hidden"`2424 // Used is the range from A1 to the last cell with contents, "" when@@ -59,7 +59,7 @@ type TableDescription struct {
5959 // the sheet.6060 type RegionDescription struct {6161 Name string `json:"name"`62- Kind string `json:"kind"` // "output" or "linked"62+ Kind string `json:"kind"` // "output", "linked" or "source"6363 Range string `json:"range"`6464 File string `json:"file,omitempty"` // a linked region's file6565 }@@ -137,6 +137,10 @@ func describeSheet(s *sheet.Sheet, shown bool) SheetDescription {
137137 if s.IsNotebook() {138138 d.Kind, d.Notebook = "notebook", describeNotebook(s)139139 }140+ if info, ok := s.Source(); ok {141+ d.Kind, d.Rows, d.Cols = "source", info.Shape.Rows+1, len(info.Shape.Cols)142+ d.Header, d.Columns = 1, append([]string{}, info.Shape.Cols...)143+ }140144 return d141145 }142146 @@ -165,6 +169,9 @@ func describeRegion(s *sheet.Sheet, r sheet.Region) RegionDescription {
165169 if r.Linked() {166170 d.Kind, d.File = "linked", r.File.Path167171 }172+ if r.File.Paged {173+ d.Kind, d.File = "source", r.File.Path174+ }168175 return d169176 }170177 @@ -46,7 +46,7 @@ func TestDescribeFixture(t *testing.T) {
4646 if err := json.Unmarshal(js.Bytes(), &back); err != nil {4747 t.Fatal(err)4848 }49- if len(back.Sheets) != len(d.Sheets) || back.Sheets[0].Charts[2].Type != "pie" || back.Sheets[4].Notebook[1].Name != "files" {49+ if len(back.Sheets) != len(d.Sheets) || back.Sheets[0].Charts[2].Type != "pie" || back.Sheets[6].Notebook[1].Name != "files" {5050 t.Errorf("JSON round trip lost something: %+v", back.Sheets[0])5151 }5252 var nu bytes.Buffer@@ -81,9 +81,12 @@ func describeParts(b *strings.Builder, s SheetDescription) {
8181 fmt.Fprintf(b, " table %s %s %s\n", t.Name, t.Range, strings.Join(t.Columns, ", "))8282 }8383 for _, r := range s.Regions {84- if r.Kind == "linked" {84+ switch r.Kind {85+ case "linked":8586 fmt.Fprintf(b, " linked %s %s following %s\n", r.Name, r.Range, r.File)86- } else {87+ case "source":88+ fmt.Fprintf(b, " source %s %s reading %s in place\n", r.Name, r.Range, r.File)89+ default:8790 fmt.Fprintf(b, " output %s %s\n", r.Name, r.Range)8891 }8992 }@@ -8,6 +8,7 @@
88 package headless99 1010 import (11+ "context"1112 "errors"1213 "fmt"1314 "io"@@ -52,6 +53,7 @@ func Open(path string, create bool) (*File, error) {
5253 return nil, fmt.Errorf("%s: %w", path, err)5354 }5455 SyncOutputs(f.Book)56+ f.ReadSources(context.Background())5557 return f, nil5658 }5759 @@ -0,0 +1,36 @@
1+package headless2+3+import (4+ "context"5+ "path/filepath"6+ "strings"7+8+ "github.com/FelineStateMachine/012/internal/paged"9+ "github.com/FelineStateMachine/012/internal/sheet"10+)11+12+// ReadSources answers what the workbook's formulas and pivot tables ask13+// of its linked sources, reading them to the end before it returns, as14+// the screen does in the background. Only sources in the workbook's15+// folder or below are read: a path out of it (absolute, or through ..)16+// shows why it wasn't, as the screen asks first about such paths in a17+// workbook from another computer.18+func (f *File) ReadSources(ctx context.Context) {19+ w := f.Book20+ if len(w.Sources()) == 0 {21+ return22+ }23+ dir := filepath.Dir(f.Path)24+ h := paged.NewHost(sheet.MaxCells())25+ defer h.Close()26+ var inside []paged.Linked27+ for _, l := range paged.Links(w, func(p string) string { return p }, "") {28+ if p := l.Spec.Path; filepath.IsAbs(p) || p == ".." || strings.HasPrefix(p, ".."+string(filepath.Separator)) {29+ w.SetSourceShape(l.Name, sheet.SourceShape{}, "Not read: outside the workbook's folder")30+ continue31+ }32+ l.Spec.Path = filepath.Join(dir, l.Spec.Path)33+ inside = append(inside, l)34+ }35+ h.Settle(ctx, w, inside)36+}@@ -0,0 +1,54 @@
1+package headless2+3+import (4+ "database/sql"5+ "os"6+ "path/filepath"7+ "testing"8+9+ "github.com/FelineStateMachine/012/internal/sheet"10+)11+12+func TestOpenReadsSources(t *testing.T) {13+ dir := t.TempDir()14+ db, err := sql.Open("sqlite", filepath.Join(dir, "n.sqlite"))15+ if err != nil {16+ t.Fatal(err)17+ }18+ for _, q := range []string{"CREATE TABLE n (x INTEGER)", "WITH RECURSIVE c(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM c WHERE i < 100) INSERT INTO n SELECT i FROM c"} {19+ if _, err := db.Exec(q); err != nil {20+ t.Fatal(err)21+ }22+ }23+ db.Close()24+ w := sheet.NewBook()25+ s := w.Sheet(0)26+ if _, err := w.AddSource("n", sheet.LinkSource{Path: "n.sqlite", Table: "n"}, s); err != nil {27+ t.Fatal(err)28+ }29+ if _, err := w.AddSource("far", sheet.LinkSource{Path: "../far.sqlite"}, s); err != nil {30+ t.Fatal(err)31+ }32+ s.Set(sheet.Addr{}, "=SUM(n[x])")33+ s.Set(sheet.Addr{Row: 1}, "=ROWS(far)")34+ path := filepath.Join(dir, "book.012")35+ out, err := os.Create(path)36+ if err != nil {37+ t.Fatal(err)38+ }39+ if err := w.Write(out); err != nil {40+ t.Fatal(err)41+ }42+ out.Close()43+ f, err := Open(path, false)44+ if err != nil {45+ t.Fatal(err)46+ }47+ got := f.Book.Sheet(0)48+ if v := got.Value(sheet.Addr{}); v.Num != 5050 {49+ t.Errorf("SUM over the source: %v", v)50+ }51+ if why := got.ExplainError(sheet.Addr{Row: 1}); why != "far: Not read: outside the workbook's folder" {52+ t.Errorf("a source outside the folder: %q", why)53+ }54+}@@ -35,6 +35,7 @@ type Stream struct {
3535 partial []byte // a line not yet ended3636 tail *fileio.Tail3737 values [][]byte38+ printed int // values printed in all; guarded by mu3839 header sheet.LiveRow3940 rows []sheet.LiveRow4041 total int // rows printed in all; guarded by mu@@ -145,9 +146,6 @@ func (s *Stream) read(got []byte, ended bool, u *Update) {
145146 if len(lines) == 0 {146147 return147148 }148- s.mu.Lock()149- s.values = keepLast(append(s.values, lines...), s.Keep)150- s.mu.Unlock()151149 res, err := s.tail.Feed(append(bytes.Join(lines, []byte("\n")), '\n'))152150 if err != nil {153151 u.Err = err.Error()@@ -156,7 +154,9 @@ func (s *Stream) read(got []byte, ended bool, u *Update) {
156154 s.header = res.Header157155 u.Header = res.Header158156 }159- s.mu.Lock()157+ s.mu.Lock() // the values and the counts change together (Snapshot)158+ s.values = keepLast(append(s.values, lines...), s.Keep)159+ s.printed += len(lines)160160 s.total += len(res.Rows)161161 s.mu.Unlock()162162 s.rows = keepLast(append(s.rows, res.Rows...), s.Keep)@@ -210,5 +210,34 @@ func (s *Stream) Rows() int {
210210 func (s *Stream) NUON() []byte {211211 s.mu.Lock()212212 defer s.mu.Unlock()213- return append(append([]byte("["), bytes.Join(s.values, []byte(", "))...), ']')213+ return list(s.values)214+}215+216+// list is values as a NUON list.217+func list(values [][]byte) []byte {218+ return append(append([]byte("["), bytes.Join(values, []byte(", "))...), ']')219+}220+221+// Snapshot is what a stream has printed, read at one moment.222+type Snapshot struct {223+ NUON []byte // the values kept, as NUON gives them224+ // More are the values printed after the first seen (Snapshot's225+ // argument), as a NUON list, so what shows the output can add them226+ // to what it shows rather than read it all again; nil when some of227+ // them are kept no more.228+ More []byte229+ // Values and Rows are how many values and rows have been printed.230+ Values, Rows int231+}232+233+// Snapshot is what the stream has printed so far, and what of it came234+// after the first seen values.235+func (s *Stream) Snapshot(seen int) Snapshot {236+ s.mu.Lock()237+ defer s.mu.Unlock()238+ snap := Snapshot{NUON: list(s.values), Values: s.printed, Rows: s.total}239+ if n := s.printed - seen; n >= 0 && n <= len(s.values) {240+ snap.More = list(s.values[len(s.values)-n:])241+ }242+ return snap214243 }@@ -0,0 +1,247 @@
1+package paged2+3+import (4+ "cmp"5+ "context"6+ "slices"7+8+ "github.com/FelineStateMachine/012/internal/functions"9+ "github.com/FelineStateMachine/012/internal/sheet"10+)11+12+// book is the functions.Book a job computes a call through: the sources13+// the call reads, each as a sheet named by the source's name, whose14+// first row is its columns' names and the rest its rows. Ranges are15+// streamed, a chunk at a time, through a cursor that goes on where the16+// last chunk ended, so a function reading a column once reads the file17+// once; a cell is fetched with its row, and the last chunk's cells are18+// at hand, for lookups that walk a column and read what they pass.19+type book struct {20+ ctx context.Context21+ srcs map[string]snapshot22+ err error // the first a source gave: the answer is then that23+24+ cursors map[cursorKey]*cursor25+ chunk []sheet.Addr // the last chunk Scan read, row by row26+ chunkV []sheet.Value27+ chunkOf string28+ rows map[rowKey][]sheet.LiveCell // rows fetched, a few29+}30+31+type cursorKey struct {32+ src string33+ c0, c1 int34+}35+36+type rowKey struct {37+ src string38+ row int39+}40+41+// maxRows bounds the rows fetched a book keeps.42+const maxRows = 25643+44+func (b *book) close() {45+ for _, c := range b.cursors {46+ b.stopCursor(c)47+ }48+}49+50+func (b *book) src(name string) (snapshot, bool) {51+ s, ok := b.srcs[key(name)]52+ return s, ok53+}54+55+// table is the source's table as a sheet holds it, header included.56+func (s snapshot) table() sheet.Rect {57+ return sheet.Rect{To: sheet.Addr{Col: len(s.shape.Cols) - 1, Row: s.shape.Rows}}58+}59+60+// clip is the part of r the source's table holds.61+func (s snapshot) clip(r sheet.Rect) (sheet.Rect, bool) {62+ t := s.table()63+ if len(s.shape.Cols) == 0 {64+ return sheet.Rect{}, false65+ }66+ out := sheet.Rect{67+ From: sheet.Addr{Col: max(r.From.Col, t.From.Col), Row: max(r.From.Row, t.From.Row)},68+ To: sheet.Addr{Col: min(r.To.Col, t.To.Col), Row: min(r.To.Row, t.To.Row)},69+ }70+ return out, out.From.Col <= out.To.Col && out.From.Row <= out.To.Row71+}72+73+func (s snapshot) header(col int) sheet.Value {74+ return sheet.Value{Kind: sheet.Text, Str: s.shape.Cols[col]}75+}76+77+// Cell is a cell's value: a column's name, a value of its row, or blank78+// outside the table.79+func (b *book) Cell(name string, a sheet.Addr) sheet.Value {80+ s, ok := b.src(name)81+ if !ok {82+ return sheet.ErrRef83+ }84+ if !s.table().Contains(a) {85+ return sheet.Value{}86+ }87+ if a.Row == 0 {88+ return s.header(a.Col)89+ }90+ if b.chunkOf == key(name) {91+ if i, found := slices.BinarySearchFunc(b.chunk, a, addrOrder); found {92+ return b.chunkV[i]93+ }94+ }95+ row, ok := b.fetch(s, a.Row-1)96+ if !ok || a.Col >= len(row) {97+ return sheet.Value{}98+ }99+ return row[a.Col].V100+}101+102+func addrOrder(x, y sheet.Addr) int {103+ return cmp.Or(cmp.Compare(x.Row, y.Row), cmp.Compare(x.Col, y.Col))104+}105+106+// fetch is the source's row numbered row, every column.107+func (b *book) fetch(s snapshot, row int) ([]sheet.LiveCell, bool) {108+ k := rowKey{key(s.name), row}109+ if r, ok := b.rows[k]; ok {110+ return r, true111+ }112+ got, err := s.h.Fetch(b.ctx, []int64{int64(row)}, nil)113+ if err != nil {114+ b.fail(err)115+ return nil, false116+ }117+ if b.rows == nil || len(b.rows) >= maxRows {118+ b.rows = map[rowKey][]sheet.LiveCell{}119+ }120+ if len(got) == 0 {121+ return nil, false122+ }123+ b.rows[k] = got[0]124+ return got[0], true125+}126+127+func (b *book) fail(err error) {128+ if b.err == nil && b.ctx.Err() == nil {129+ b.err = err130+ }131+}132+133+// Scan reads the cells of r holding something from from on, as134+// functions.Book says: a chunk of a cursor streaming the rows.135+func (b *book) Scan(name string, r sheet.Rect, from sheet.Addr, addrs []sheet.Addr, vals []sheet.Value) int {136+ s, ok := b.src(name)137+ if !ok {138+ return -1139+ }140+ part, ok := s.clip(r)141+ if !ok || from.Row > part.To.Row {142+ return 0143+ }144+ if from.Row < part.From.Row || from.Row == part.From.Row && from.Col < part.From.Col {145+ from = part.From146+ }147+ n := 0148+ b.chunk, b.chunkV, b.chunkOf = b.chunk[:0], b.chunkV[:0], key(name)149+ emit := func(a sheet.Addr, v sheet.Value) bool {150+ if v.Kind == sheet.Empty {151+ return true152+ }153+ addrs[n] = a154+ if vals != nil {155+ vals[n] = v156+ }157+ b.chunk, b.chunkV = append(b.chunk, a), append(b.chunkV, v)158+ n++159+ return n < len(addrs) && !(vals != nil && v.Kind == sheet.Error)160+ }161+ if from.Row == 0 {162+ for c := from.Col; c <= part.To.Col; c++ {163+ if !emit(sheet.Addr{Col: c, Row: 0}, s.header(c)) {164+ return n165+ }166+ }167+ from = sheet.Addr{Col: part.From.Col, Row: 1}168+ }169+ if from.Row > part.To.Row {170+ return n171+ }172+ c := b.cursor(s, part, from)173+ c.read(part, emit)174+ if c.cur == nil && c.err != nil {175+ b.fail(c.err) // the scan ended failing176+ }177+ return n178+}179+180+// Bounds is the part of r the source's table holds.181+func (b *book) Bounds(name string, r sheet.Rect) (sheet.Rect, bool, bool) {182+ s, ok := b.src(name)183+ if !ok {184+ return sheet.Rect{}, false, false185+ }186+ part, any := s.clip(r)187+ return part, any, true188+}189+190+// RangeAgg shares nothing: each call reads its ranges once.191+func (b *book) RangeAgg(string, sheet.Rect) (functions.Agg, *sheet.Value, bool) {192+ return functions.Agg{}, nil, false193+}194+195+// Fold adds the cells of r to agg, streaming the source's rows.196+func (b *book) Fold(name string, r sheet.Rect, agg functions.Agg) (functions.Agg, *sheet.Value) {197+ s, ok := b.src(name)198+ if !ok {199+ return agg, agg.Add(sheet.ErrRef, false)200+ }201+ part, ok := s.clip(r)202+ if !ok {203+ return agg, nil204+ }205+ var e *sheet.Value206+ if part.From.Row == 0 {207+ for c := part.From.Col; c <= part.To.Col && e == nil; c++ {208+ e = agg.Add(s.header(c), false)209+ }210+ part.From.Row = 1211+ }212+ if e != nil || part.From.Row > part.To.Row {213+ return agg, e214+ }215+ cols := make([]int, 0, part.To.Col-part.From.Col+1)216+ for c := part.From.Col; c <= part.To.Col; c++ {217+ cols = append(cols, c)218+ }219+ last := int64(part.To.Row - 1)220+ err := s.h.Scan(b.ctx, int64(part.From.Row-1), cols, func(row int64, vals []sheet.LiveCell) bool {221+ for _, v := range vals {222+ if v.V.Kind != sheet.Empty {223+ if e = agg.Add(v.V, false); e != nil {224+ return false225+ }226+ }227+ }228+ return row < last229+ })230+ if err != nil {231+ b.fail(err)232+ }233+ return agg, e234+}235+236+// Ask has no answers: sources don't ask JEV.237+func (b *book) Ask(functions.RemoteCall) (functions.RemoteAnswer, sheet.Value) {238+ return functions.RemoteAnswer{}, functions.ErrNoRemote239+}240+241+// Paged is false: a job reads its sources as sheets.242+func (b *book) Paged(string) bool { return false }243+244+// Stream isn't asked: nothing is paged here.245+func (b *book) Stream(functions.StreamCall) (sheet.Value, *functions.Array) {246+ return sheet.ErrValue, nil247+}@@ -0,0 +1,88 @@
1+package paged2+3+import (4+ "fmt"5+6+ "github.com/FelineStateMachine/012/internal/functions"7+ "github.com/FelineStateMachine/012/internal/sheet"8+)9+10+// rangeValues is a range of a source read whole, as an array: what an11+// operator or a function reading it some other way is given, no bigger12+// than budget cells.13+func (b *book) rangeValues(name string, r sheet.Rect, budget int) sheet.SourceAnswer {14+ s, ok := b.src(name)15+ if !ok {16+ return sheet.SourceAnswer{V: sheet.ErrRef}17+ }18+ part, ok := s.clip(r)19+ if !ok {20+ return sheet.SourceAnswer{A: &functions.Array{Rows: 1, Cols: 1}}21+ }22+ rows, cols := part.To.Row-part.From.Row+1, part.To.Col-part.From.Col+123+ if rows > budget/cols {24+ return sheet.SourceAnswer{V: sheet.ErrValue, Why: fmt.Sprintf(25+ "Reading %s whole takes %s cells, more than max-cells (%s): SUM, COUNTIFS, XLOOKUP and the like read a source of any size",26+ s.name, grouped(rows*cols), grouped(budget))}27+ }28+ a := functions.NewArray(rows, cols)29+ addrs, vals := make([]sheet.Addr, 1024), make([]sheet.Value, 1024)30+ for from := part.From; ; {31+ n := b.Scan(name, part, from, addrs, vals)32+ for i := range n {33+ a.Set(addrs[i].Row-part.From.Row, addrs[i].Col-part.From.Col, vals[i])34+ }35+ if n < len(addrs) && (n == 0 || vals[n-1].Kind != sheet.Error) {36+ break37+ }38+ from = after(part, addrs[n-1])39+ }40+ return sheet.SourceAnswer{A: a}41+}42+43+// after is the cell after a in r, row by row.44+func after(r sheet.Rect, a sheet.Addr) sheet.Addr {45+ if a.Col < r.To.Col {46+ return sheet.Addr{Col: a.Col + 1, Row: a.Row}47+ }48+ return sheet.Addr{Col: r.From.Col, Row: a.Row + 1}49+}50+51+// pivot gathers a pivot table's groups over the source, streaming the52+// columns it reads.53+func (b *book) pivot(name string, p sheet.Pivot) sheet.SourceAnswer {54+ s, ok := b.src(name)55+ if !ok {56+ return sheet.SourceAnswer{V: sheet.ErrRef}57+ }58+ cols := sheet.PivotCols(p)59+ width := len(s.shape.Cols)60+ for _, c := range cols {61+ if c >= width {62+ return sheet.SourceAnswer{V: sheet.ErrRef, Why: fmt.Sprintf("%s has no column %s", s.name, sheet.ColName(c))}63+ }64+ }65+ row := make([]sheet.LiveCell, width)66+ g, err := sheet.GatherPivot(p, s.shape, func(each func(int, []sheet.LiveCell) bool) error {67+ return s.h.Scan(b.ctx, 0, cols, func(n int64, vals []sheet.LiveCell) bool {68+ for i, c := range cols {69+ row[c] = vals[i]70+ }71+ return each(int(n), row)72+ })73+ })74+ if err != nil {75+ b.fail(err)76+ return sheet.SourceAnswer{V: sheet.ErrRef}77+ }78+ return sheet.SourceAnswer{Pivot: g}79+}80+81+// grouped is n with thousands separators: 12,000,000.82+func grouped(n int) string {83+ s := fmt.Sprint(n)84+ for i := len(s) - 3; i > 0; i -= 3 {85+ s = s[:i] + "," + s[i:]86+ }87+ return s88+}@@ -0,0 +1,160 @@
1+package paged2+3+import (4+ "context"5+6+ "github.com/FelineStateMachine/012/internal/sheet"7+)8+9+// A cursor streams a range's rows for the book's Scan, a chunk at a10+// time: a goroutine of its own scans the source and hands the rows over11+// in batches, so the source's scan runs on while functions read what12+// it read, and taking a row costs a slice index.13+14+// batchRows is how many rows a batch holds.15+const batchRows = 409616+17+// cursor is a range's rows from where the last chunk ended.18+type cursor struct {19+ at sheet.Addr // where the next chunk starts20+ width int21+ batches chan []sheet.LiveCell22+ free chan []sheet.LiveCell23+ cancel context.CancelFunc24+ err error // what the scan failed with, read once batches is closed25+ cur []sheet.LiveCell26+ i int // the next row of cur27+ row []sheet.LiveCell // the row being read28+ have bool29+}30+31+// cursor is the cursor over the columns of part from from on: the one32+// the last chunk left there, or a new one.33+func (b *book) cursor(s snapshot, part sheet.Rect, from sheet.Addr) *cursor {34+ k := cursorKey{key(s.name), part.From.Col, part.To.Col}35+ if c := b.cursors[k]; c != nil {36+ if c.at == from {37+ return c38+ }39+ b.stopCursor(c)40+ }41+ cols := make([]int, 0, part.To.Col-part.From.Col+1)42+ for c := part.From.Col; c <= part.To.Col; c++ {43+ cols = append(cols, c)44+ }45+ c := startCursor(b.ctx, s, from, cols)46+ if b.cursors == nil {47+ b.cursors = map[cursorKey]*cursor{}48+ }49+ b.cursors[k] = c50+ return c51+}52+53+// startCursor scans the source s's columns cols from the sheet row of54+// from on.55+func startCursor(ctx context.Context, s snapshot, from sheet.Addr, cols []int) *cursor {56+ ctx, cancel := context.WithCancel(ctx)57+ c := &cursor{at: from, width: len(cols), batches: make(chan []sheet.LiveCell, 2), free: make(chan []sheet.LiveCell, 3), cancel: cancel}58+ go func() {59+ defer close(c.batches)60+ batch := c.take()61+ err := s.h.Scan(ctx, int64(from.Row-1), cols, func(_ int64, vals []sheet.LiveCell) bool {62+ if batch = append(batch, vals...); len(batch) < batchRows*c.width {63+ return true64+ }65+ if !c.send(ctx, batch) {66+ return false67+ }68+ batch = c.take()69+ return true70+ })71+ if len(batch) > 0 {72+ c.send(ctx, batch)73+ }74+ if ctx.Err() == nil {75+ c.err = err76+ }77+ }()78+ return c79+}80+81+// take is a batch to fill: one read already, or a new one.82+func (c *cursor) take() []sheet.LiveCell {83+ select {84+ case b := <-c.free:85+ return b[:0]86+ default:87+ return make([]sheet.LiveCell, 0, batchRows*c.width)88+ }89+}90+91+// send hands a batch over, reporting false once the cursor is stopped.92+func (c *cursor) send(ctx context.Context, batch []sheet.LiveCell) bool {93+ select {94+ case c.batches <- batch:95+ return true96+ case <-ctx.Done():97+ return false98+ }99+}100+101+// next is the next row, with false after the last.102+func (c *cursor) next() ([]sheet.LiveCell, bool) {103+ if c.i*c.width >= len(c.cur) {104+ if c.cur != nil {105+ select {106+ case c.free <- c.cur:107+ default:108+ }109+ }110+ b, ok := <-c.batches111+ if !ok {112+ c.cur = nil113+ return nil, false114+ }115+ c.cur, c.i = b, 0116+ }117+ row := c.cur[c.i*c.width : (c.i+1)*c.width]118+ c.i++119+ return row, true120+}121+122+// stopCursor stops c's scan, noting what it failed with.123+func (b *book) stopCursor(c *cursor) {124+ c.cancel()125+ for range c.batches {126+ }127+ if c.err != nil {128+ b.fail(c.err)129+ }130+}131+132+// read gives emit the cells of part from the cursor on, until emit133+// returns false or the part ends.134+func (c *cursor) read(part sheet.Rect, emit func(sheet.Addr, sheet.Value) bool) {135+ for c.at.Row <= part.To.Row {136+ if !c.have {137+ row, ok := c.next()138+ if !ok {139+ c.at.Row = part.To.Row + 1140+ return141+ }142+ c.row, c.have = row, true143+ }144+ more := true145+ for more && c.at.Col <= part.To.Col {146+ v := c.row[c.at.Col-part.From.Col].V147+ a := c.at148+ c.at.Col++149+ more = emit(a, v)150+ }151+ if c.at.Col > part.To.Col {152+ // The row is read: the next chunk starts at the next one's153+ // first cell, as the Reader asks for it.154+ c.at, c.have = sheet.Addr{Col: part.From.Col, Row: c.at.Row + 1}, false155+ }156+ if !more {157+ return158+ }159+ }160+}@@ -0,0 +1,327 @@
1+// Package paged reads linked sources in place: Parquet files and SQLite2+// tables or queries too big for any grid, linked as tables on tabs of3+// their own (the engine's source.go). Its Host answers what a4+// workbook's formulas and pivot tables ask of them (sheet.SourceHost)5+// by streaming the sources in the background, keeping the answers until6+// a source's file changes; its Pages are the rows a source's tab shows,7+// read a page at a time as it scrolls.8+//9+// The Host and Pages belong to the goroutine that owns the workbook (the10+// UI's). What reads files runs as Jobs on other goroutines, over what11+// the Job took with it, and hands its result back to be stored, so the12+// owner never waits on a file.13+package paged14+15+import (16+ "context"17+ "os"18+ "strings"19+ "time"20+21+ "github.com/FelineStateMachine/012/internal/fileio"22+ "github.com/FelineStateMachine/012/internal/sheet"23+)24+25+// Linked is a source as the workbook links it, its file resolved.26+type Linked struct {27+ Name string28+ Spec fileio.SourceSpec29+}30+31+// Opener opens what a spec names: fileio.OpenSource, or a stand-in.32+type Opener func(ctx context.Context, spec fileio.SourceSpec) (fileio.Source, error)33+34+// Host answers a workbook's questions about its linked sources.35+type Host struct {36+ // Budget is how many cells a function that holds what it reads may37+ // read of a source, and a range read whole may have: max-cells.38+ Budget int39+ // Open opens a source; fileio.OpenSource unless replaced.40+ Open Opener41+ // Debounce is how long a changed file must hold still before it's42+ // read again.43+ Debounce time.Duration44+ // Now is the clock; tests replace it.45+ Now func() time.Time46+47+ srcs map[string]*source // by name key48+ answers map[string]answer // by question key49+ asked map[string]bool // questions queued or running50+ fresh []*Job // jobs not yet handed out51+}52+53+// source is a linked source as the host has it.54+type source struct {55+ name string56+ spec fileio.SourceSpec57+ h fileio.Source // nil until opened58+ shape sheet.SourceShape59+ err string60+ gen int // counts openings; answers of earlier ones are stale61+ opening bool // a job is opening it62+ read stamp // the file when opened63+ seen stamp // the file when last polled, different from read64+ seenAt time.Time65+}66+67+// stamp is what tells a file changed.68+type stamp struct {69+ size int6470+ mod time.Time71+}72+73+// answer is a stored answer, with the generation of each source it read.74+type answer struct {75+ a sheet.SourceAnswer76+ gens map[string]int77+}78+79+// NewHost is a host with budget max-cells.80+func NewHost(budget int) *Host {81+ return &Host{Budget: budget, Open: fileio.OpenSource, Debounce: 300 * time.Millisecond, Now: time.Now,82+ srcs: map[string]*source{}, answers: map[string]answer{}, asked: map[string]bool{}}83+}84+85+func key(name string) string { return strings.ToLower(name) }86+87+// Link makes the host's sources the workbook's: new ones and those88+// reading something else are opened, those gone are closed. It returns89+// whether anything changed.90+func (h *Host) Link(list []Linked) bool {91+ changed := false92+ keep := map[string]bool{}93+ for _, l := range list {94+ k := key(l.Name)95+ keep[k] = true96+ s := h.srcs[k]97+ if s != nil && s.spec == l.Spec {98+ continue99+ }100+ if s != nil && s.h != nil {101+ s.h.Close()102+ }103+ gen := 0104+ if s != nil {105+ gen = s.gen + 1106+ }107+ s = &source{name: l.Name, spec: l.Spec, gen: gen}108+ h.srcs[k] = s109+ h.reopen(s)110+ changed = true111+ }112+ for k, s := range h.srcs {113+ if !keep[k] {114+ if s.h != nil {115+ s.h.Close()116+ }117+ delete(h.srcs, k)118+ changed = true119+ }120+ }121+ return changed122+}123+124+// reopen queues a job opening s.125+func (h *Host) reopen(s *source) {126+ s.opening = true127+ h.fresh = append(h.fresh, &Job{kind: jobOpen, name: s.name, spec: s.spec, gen: s.gen, open: h.Open})128+}129+130+// Reload reads the source named name again, as when its file changed:131+// what was worked out of it is worked out again.132+func (h *Host) Reload(name string) {133+ s := h.srcs[key(name)]134+ if s == nil || s.opening {135+ return136+ }137+ if s.h != nil {138+ s.h.Close()139+ s.h = nil140+ }141+ s.gen++142+ h.reopen(s)143+}144+145+// Close lets every source go.146+func (h *Host) Close() {147+ for _, s := range h.srcs {148+ if s.h != nil {149+ s.h.Close()150+ }151+ }152+ clear(h.srcs)153+}154+155+// Shape is what the host found of the source named name: its shape, or156+// why it can't be read, and whether it has been opened.157+func (h *Host) Shape(name string) (sheet.SourceShape, string, bool) {158+ s := h.srcs[key(name)]159+ if s == nil || s.h == nil && s.err == "" {160+ return sheet.SourceShape{}, "", false161+ }162+ return s.shape, s.err, true163+}164+165+// Source is the opened source named name, for its tab's pages.166+func (h *Host) Source(name string) (fileio.Source, int, bool) {167+ s := h.srcs[key(name)]168+ if s == nil || s.h == nil {169+ return nil, 0, false170+ }171+ return s.h, s.gen, true172+}173+174+// Answer returns the answer to q, or false while it isn't known, queuing175+// q to be worked out (sheet.SourceHost).176+func (h *Host) Answer(q sheet.SourceQuestion) (sheet.SourceAnswer, bool) {177+ k := q.Key()178+ if a, ok := h.answers[k]; ok && h.current(a.gens) {179+ return a.a, true180+ }181+ if h.asked[k] {182+ return sheet.SourceAnswer{}, false183+ }184+ j := &Job{kind: jobAnswer, q: q, key: k, budget: h.Budget, srcs: map[string]snapshot{}}185+ for _, name := range q.Sources() {186+ s := h.srcs[key(name)]187+ switch {188+ case s == nil:189+ return sheet.SourceAnswer{V: sheet.ErrRef, Why: "There's no linked source named " + name}, true190+ case s.err != "":191+ return sheet.SourceAnswer{V: sheet.ErrRef, Why: s.name + ": " + s.err}, true192+ case s.h == nil:193+ return sheet.SourceAnswer{}, false // opening: asked again once it's open194+ }195+ j.srcs[key(name)] = snapshot{name: s.name, h: s.h, shape: s.shape, gen: s.gen}196+ }197+ h.asked[k] = true198+ h.fresh = append(h.fresh, j)199+ return sheet.SourceAnswer{}, false200+}201+202+// current reports whether answers read with the sources' generations203+// gens are still the sources'.204+func (h *Host) current(gens map[string]int) bool {205+ for k, g := range gens {206+ if s := h.srcs[k]; s == nil || s.gen != g {207+ return false208+ }209+ }210+ return true211+}212+213+// Jobs hands out the jobs queued since the last call, to run in the214+// background, each once.215+func (h *Host) Jobs() []*Job {216+ out := h.fresh217+ h.fresh = nil218+ return out219+}220+221+// Pending reports whether jobs are queued or running.222+func (h *Host) Pending() bool { return len(h.fresh) > 0 || len(h.asked) > 0 || h.opening() }223+224+func (h *Host) opening() bool {225+ for _, s := range h.srcs {226+ if s.opening {227+ return true228+ }229+ }230+ return false231+}232+233+// Done is what storing a job's result tells the workbook.234+type Done struct {235+ // Keys are the questions answered.236+ Keys []string237+ // Opened names a source opened, or failing to, with its shape.238+ Opened string239+ Shape sheet.SourceShape240+ Err string241+}242+243+// Store keeps what the job j found, run and back on the owner's244+// goroutine, and says what the workbook is to be told.245+func (h *Host) Store(j *Job) Done {246+ if j.kind == jobOpen {247+ return h.storeOpen(j)248+ }249+ delete(h.asked, j.key)250+ gens := map[string]int{}251+ for k, snap := range j.srcs {252+ gens[k] = snap.gen253+ }254+ if !h.current(gens) {255+ return Done{} // a source changed while it ran: asked again256+ }257+ h.answers[j.key] = answer{a: j.ans, gens: gens}258+ return Done{Keys: []string{j.key}}259+}260+261+func (h *Host) storeOpen(j *Job) Done {262+ s := h.srcs[key(j.name)]263+ if s == nil || s.gen != j.gen {264+ if j.h != nil {265+ j.h.Close()266+ }267+ return Done{}268+ }269+ s.opening = false270+ if s.h != nil && s.h != j.h {271+ s.h.Close()272+ }273+ s.h, s.shape, s.err, s.read = j.h, j.shape, "", j.stamp274+ if j.err != nil {275+ s.h, s.shape, s.err = nil, sheet.SourceShape{}, describe(j.err)276+ }277+ h.forget(key(s.name))278+ return Done{Opened: s.name, Shape: s.shape, Err: s.err}279+}280+281+// forget drops the answers that read the source with key k.282+func (h *Host) forget(k string) {283+ for qk, a := range h.answers {284+ if _, ok := a.gens[k]; ok {285+ delete(h.answers, qk)286+ }287+ }288+}289+290+// describe is an error as the context line says it.291+func describe(err error) string {292+ if os.IsNotExist(err) {293+ return "the file isn't there"294+ }295+ return err.Error()296+}297+298+// Poll looks at each source's file, once they're open, and queues the299+// ones that changed and have held still for Debounce to be read again,300+// reporting whether it queued any. A file that goes away is noticed the301+// same way.302+func (h *Host) Poll() bool {303+ now, queued := h.Now(), false304+ for _, s := range h.srcs {305+ if s.opening {306+ continue307+ }308+ st, err := os.Stat(s.spec.Path)309+ cur := stamp{}310+ if err == nil {311+ cur = stamp{st.Size(), st.ModTime()}312+ }313+ switch {314+ case cur == s.read:315+ s.seen = stamp{}316+ continue317+ case cur != s.seen:318+ s.seen, s.seenAt = cur, now319+ continue320+ case now.Sub(s.seenAt) < h.Debounce:321+ continue322+ }323+ h.Reload(s.name) // jobs reading it fail, and their answers are dropped324+ queued = true325+ }326+ return queued327+}@@ -0,0 +1,272 @@
1+package paged2+3+import (4+ "context"5+ "os"6+ "path/filepath"7+ "strings"8+ "testing"9+ "time"10+11+ "github.com/parquet-go/parquet-go"12+13+ "github.com/FelineStateMachine/012/internal/fileio"14+ "github.com/FelineStateMachine/012/internal/sheet"15+)16+17+// sale is a row of the Parquet files the tests link.18+type sale struct {19+ ID int64 `parquet:"id"`20+ Cat *string `parquet:"cat,optional"`21+ Amount *float64 `parquet:"amount,optional"`22+ Day int32 `parquet:"day,date"`23+ Tags []string `parquet:"tags,list"`24+}25+26+var cats = []string{"North", "south", "East", "west", "", "north"}27+28+// writeSales writes n rows to name, amounts shifted by shift, in row29+// groups of 64 rows.30+func writeSales(t testing.TB, name string, n int, shift float64) {31+ t.Helper()32+ f, err := os.Create(name)33+ if err != nil {34+ t.Fatal(err)35+ }36+ w := parquet.NewGenericWriter[sale](f, parquet.MaxRowsPerRowGroup(64))37+ day := int32(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC).Unix() / 86400)38+ rows := make([]sale, n)39+ for i := range rows {40+ r := &rows[i]41+ r.ID, r.Day = int64(i), day+int32(i%40)42+ if i%9 != 4 {43+ c := cats[i%len(cats)]44+ r.Cat = &c45+ }46+ if i%11 != 7 {47+ a := float64((i*37)%101) - 20.5 + shift48+ r.Amount = &a49+ }50+ if i%5 == 0 {51+ r.Tags = []string{"a", "b"}52+ }53+ }54+ if _, err := w.Write(rows); err != nil {55+ t.Fatal(err)56+ }57+ if err := w.Close(); err != nil {58+ t.Fatal(err)59+ }60+ if err := f.Close(); err != nil {61+ t.Fatal(err)62+ }63+}64+65+// linked is a workbook holding the file at name imported on Sheet1, as66+// the table ref, and linked as the source sales, settled.67+func linked(t *testing.T, name string, budget int) (*sheet.Workbook, *Host) {68+ t.Helper()69+ res, err := fileio.Import(context.Background(), name, fileio.Options{})70+ if err != nil {71+ t.Fatal(err)72+ }73+ w := res.Sheet.Book()74+ s := w.Sheet(0)75+ if err := w.RenameSheet(s, "Data"); err != nil {76+ t.Fatal(err)77+ }78+ b, _ := s.UsedRange()79+ if err := s.CreateTable("ref", b); err != nil {80+ t.Fatal(err)81+ }82+ if _, err := w.AddSource("sales", sheet.LinkSource{Path: name}, s); err != nil {83+ t.Fatal(err)84+ }85+ h := NewHost(budget)86+ h.Settle(context.Background(), w, Links(w, same, ""))87+ return w, h88+}89+90+// sameFormulas are formulas that read the source as {T}: each gives the91+// same over the source as over the table ref holding the same rows.92+var sameFormulas = []string{93+ "=SUM({T}[amount])", "=AVERAGE({T}[amount])", "=COUNT({T}[amount])", "=COUNTA({T}[cat])",94+ "=MIN({T}[amount])", "=MAX({T}[amount])", "=COUNTA({T}[tags])",95+ `=COUNTIF({T}[cat],"north")`, `=COUNTIF({T}[amount],">10")`, `=COUNTIFS({T}[cat],"North",{T}[amount],"<0")`,96+ `=SUMIF({T}[cat],"East",{T}[amount])`, `=SUMIFS({T}[amount],{T}[cat],"west",{T}[id],">100")`,97+ `=AVERAGEIF({T}[cat],"south",{T}[amount])`, `=AVERAGEIFS({T}[amount],{T}[cat],"<>East")`,98+ "=COUNTBLANK({T}[cat])", "=SUMPRODUCT({T}[amount],{T}[id])", `=COUNTIF({T}[day],">="&DATE(2026,1,20))`,99+ "=MATCH(17,{T}[id],0)", "=MATCH(50.5,{T}[amount],0)", "=XLOOKUP(42,{T}[id],{T}[amount])",100+ `=XLOOKUP("east",{T}[cat],{T}[id],"none",0,-1)`, "=XLOOKUP(10,{T}[amount],{T}[id],,1)",101+ "=XLOOKUP(10,{T}[amount],{T}[id],,-1,-1)", `=XLOOKUP("nowhere",{T}[cat],{T}[id],"none")`,102+ "=VLOOKUP(7,{T},3,FALSE)", "=INDEX({T}[amount],10)", "=ROWS({T})", "=COLUMNS({T}[#All])",103+ "=MEDIAN({T}[amount])", "=SUM({T}[amount]*2)", "=ROUND(SUM({T}[amount]),2)",104+}105+106+func TestFormulasOverASource(t *testing.T) {107+ dir := t.TempDir()108+ name := filepath.Join(dir, "sales.parquet")109+ writeSales(t, name, 5000, 0)110+ w, h := linked(t, name, 100000)111+ s := w.Sheet(0)112+ for i, f := range sameFormulas {113+ at := sheet.Addr{Col: 10, Row: i}114+ ref := sheet.Addr{Col: 11, Row: i}115+ if err := s.Set(at, strings.ReplaceAll(f, "{T}", "sales")); err != nil {116+ t.Fatalf("%s: %v", f, err)117+ }118+ if err := s.Set(ref, strings.ReplaceAll(f, "{T}", "ref")); err != nil {119+ t.Fatalf("%s: %v", f, err)120+ }121+ }122+ h.Settle(context.Background(), w, Links(w, same, ""))123+ for i, f := range sameFormulas {124+ got, want := s.Value(sheet.Addr{Col: 10, Row: i}), s.Value(sheet.Addr{Col: 11, Row: i})125+ if testing.Verbose() {126+ t.Logf("%s: %v", f, got)127+ }128+ if got.Kind == sheet.Error && !strings.Contains(f, "nowhere") {129+ t.Errorf("%s: %v", f, got)130+ }131+ if got != want {132+ t.Errorf("%s: %v over the source, %v over the sheet", f, got, want)133+ }134+ }135+}136+137+func TestPendingUntilAnswered(t *testing.T) {138+ dir := t.TempDir()139+ name := filepath.Join(dir, "sales.parquet")140+ writeSales(t, name, 100, 0)141+ w, h := linked(t, name, 1000)142+ s := w.Sheet(0)143+ a := sheet.Addr{Col: 10}144+ if err := s.Set(a, "=SUM(sales[amount])"); err != nil {145+ t.Fatal(err)146+ }147+ if v := s.Value(a); !sheet.IsPending(v) {148+ t.Fatalf("before the answer: %v, want Loading…", v)149+ }150+ jobs := h.Jobs()151+ if len(jobs) != 1 {152+ t.Fatalf("%d jobs queued, want 1", len(jobs))153+ }154+ jobs[0].Run(context.Background())155+ Apply(w, h.Store(jobs[0]))156+ if v := s.Value(a); v.Kind != sheet.Number {157+ t.Fatalf("after the answer: %v", v)158+ }159+ // Asked again, the answer is kept: no job.160+ if err := s.Set(sheet.Addr{Col: 11}, "=SUM(sales[amount])+1"); err != nil {161+ t.Fatal(err)162+ }163+ if jobs := h.Jobs(); len(jobs) != 0 {164+ t.Errorf("%d jobs for a kept answer", len(jobs))165+ }166+}167+168+func TestPastTheBudget(t *testing.T) {169+ dir := t.TempDir()170+ name := filepath.Join(dir, "sales.parquet")171+ writeSales(t, name, 300, 0)172+ w, h := linked(t, name, 100)173+ s := w.Sheet(0)174+ med, sum := sheet.Addr{Col: 10}, sheet.Addr{Col: 10, Row: 1}175+ s.Set(med, "=MEDIAN(sales[amount])")176+ s.Set(sum, "=SUM(sales[amount])")177+ h.Settle(context.Background(), w, Links(w, same, ""))178+ if v := s.Value(med); v != sheet.ErrValue {179+ t.Errorf("MEDIAN past the budget: %v, want #VALUE!", v)180+ }181+ if why := s.ExplainError(med); !strings.Contains(why, "max-cells") {182+ t.Errorf("MEDIAN past the budget explained as %q", why)183+ }184+ if v := s.Value(sum); v.Kind != sheet.Number {185+ t.Errorf("SUM streams past the budget: %v", v)186+ }187+}188+189+func TestPivotOverASource(t *testing.T) {190+ dir := t.TempDir()191+ name := filepath.Join(dir, "sales.parquet")192+ writeSales(t, name, 500, 0)193+ w, h := linked(t, name, 1000)194+ ref := w.Sheet(0)195+ src, _ := w.LookupSource("sales")196+ p := sheet.Pivot{197+ Rows: []sheet.PivotGroup{{Col: 1}},198+ Columns: []sheet.PivotGroup{{Col: 3}},199+ Values: []sheet.PivotValue{{Col: 2, Summarize: sheet.SumBy}, {Col: 2, Summarize: sheet.CountUniqueBy}},200+ RowTotals: true, ColumnTotals: true,201+ }202+ b, _ := ref.UsedRange()203+ want, err := w.CreatePivot(ref, b, "want", p)204+ if err != nil {205+ t.Fatal(err)206+ }207+ got, err := w.CreatePivot(src.Sheet, sheet.Rect{To: sheet.Addr{Col: 4, Row: sheet.MaxRows - 1}}, "got", p)208+ if err != nil {209+ t.Fatal(err)210+ }211+ h.Settle(context.Background(), w, Links(w, same, ""))212+ wb, _ := want.UsedRange()213+ gb, _ := got.UsedRange()214+ if wb != gb {215+ t.Fatalf("pivot over the source covers %v, over the sheet %v", gb, wb)216+ }217+ for row := wb.From.Row; row <= wb.To.Row; row++ {218+ for col := wb.From.Col; col <= wb.To.Col; col++ {219+ a := sheet.Addr{Col: col, Row: row}220+ if g, x := got.Value(a), want.Value(a); g != x && !(col == 0 && row == 0) {221+ t.Errorf("%s: %v over the source, %v over the sheet", a, g, x)222+ }223+ }224+ }225+}226+227+func TestSourceChanges(t *testing.T) {228+ dir := t.TempDir()229+ name := filepath.Join(dir, "sales.parquet")230+ writeSales(t, name, 100, 0)231+ w, h := linked(t, name, 1000)232+ s := w.Sheet(0)233+ a := sheet.Addr{Col: 10}234+ s.Set(a, "=SUM(sales[amount])")235+ h.Settle(context.Background(), w, Links(w, same, ""))236+ before := s.Value(a)237+ now := time.Now()238+ h.Now = func() time.Time { return now }239+ writeSales(t, name, 100, 1)240+ if h.Poll() {241+ t.Fatal("read again before the file held still")242+ }243+ now = now.Add(time.Second)244+ if !h.Poll() {245+ t.Fatal("a changed file isn't read again")246+ }247+ h.Settle(context.Background(), w, Links(w, same, ""))248+ after := s.Value(a)249+ if after.Kind != sheet.Number || after.Num-before.Num < 80 {250+ t.Errorf("SUM %v before the file changed, %v after", before, after)251+ }252+}253+254+func TestMissingSource(t *testing.T) {255+ w := sheet.NewBook()256+ s := w.Sheet(0)257+ if _, err := w.AddSource("gone", sheet.LinkSource{Path: filepath.Join(t.TempDir(), "gone.parquet")}, s); err != nil {258+ t.Fatal(err)259+ }260+ a := sheet.Addr{}261+ s.Set(a, "=SUM(gone[x])")262+ NewHost(1000).Settle(context.Background(), w, Links(w, same, ""))263+ if v := s.Value(a); v != sheet.ErrRef {264+ t.Errorf("a missing source: %v, want #REF!", v)265+ }266+ if why := s.ExplainError(a); !strings.Contains(why, "isn't there") {267+ t.Errorf("explained as %q", why)268+ }269+}270+271+// same is a path as it is: the tests' files are named whole.272+func same(p string) string { return p }Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.