files/following.md is the page for linked files; the architecture describes linked regions and the change stream of LiveOps, limits.md what following costs, and the roadmap moves streaming files to Shipped, leaving a nushell region following a pipeline as the step before shared viewing.
- commit
- cba59c6d4ecc5150ceff0661b1abdbf34087adf0
- parent
- 766b432fe975b2bc5e417013efe3e7190e9518c2
- author
- Dami <dami.etoile@gmail.com>
- kind
- docs risk score 0.0 of 3, Jev's estimate
| M | ROADMAP.md | 3 | +- |
| M | docs/ | 48 | ++++++++++++- |
| M | docs/ | 25 | ++++++ |
| M | docs/ | 1 | + |
| M | docs/ | 7 | +- |
| A | docs/ | 122 | ++++++++++++++++++++++++++++++ |
| M | docs/ | 6 | +- |
@@ -47,7 +47,7 @@ log macros record. Each step is useful on its own.
4747 4848 | Step | Result | Size |4949 |---|---|---|50-| Streaming files: a sheet or linked range follows a file as it grows (CSV, TSV, NDJSON, NUON; `tail -f` style) or is rewritten, and a nushell region can follow a streaming pipeline, with rows arriving live and no full reload | Logs and feeds as live sheets; the change stream multiplayer needs | M |50+| A nushell region follows a streaming pipeline through the linked regions' live sources (`live.Source`), rows arriving as the pipeline writes them | Pipelines as live sheets | S |5151 | Shared viewing over SSH: several `012 serve` sessions open the same workbook; one edits, the others follow live with their own cursor, scroll and theme; presence shows who is where | Watch-along and review, one writer | M |5252 | Shared editing over SSH: every session edits, the server orders operations (no CRDT needed while one server holds the workbook), per-user undo, presence and edit ownership shown in the grid, saves by the server | Multiplayer 012 over SSH | L |5353 @@ -116,6 +116,7 @@ stream above rather than a design of their own.
116116 117117 - Import CSV, TSV, JSON, NUON, XLSX, SQLite, Parquet and Lotus `.wk1`; export CSV, TSV, JSON, NUON, XLSX and SQLite; import locations; save-as and overwrite checks: [Files](docs/files/README.md)118118 - `.012` files read and written as a stream, cells straight into and out of the store: [The .012 format](docs/files/format.md#reading-and-writing)119+- Following files: a linked region follows a CSV, TSV, JSON lines or NUON file as it grows and any importable file as it's rewritten, keeping every row or the last ones, its rows arriving as the change stream: [Following files](docs/files/following.md)119120 120121 **Upkeep**121122 @@ -19,6 +19,7 @@ internal/numfmt number formats, rounding, General, date serials
1919 internal/locale the locales: separators, date order, currency, formula separators2020 internal/fileio import and export: CSV, TSV, XLSX, SQLite, Parquet, Lotus .wk1, JSON, NUON; tables on streams2121 internal/nuon nushell's object notation: typed values, tables read a row at a time, written back22+internal/live the sources of linked regions: files followed as they grow or are rewritten2223 internal/chart chart layout, text rendering and kitty image encoding2324 internal/jev the API key's resolution, answer cache and TypeSafe client2425 internal/keyring the OS credential store the API key lives in@@ -212,6 +213,17 @@ style.
212213 format) keeps only its formatting and note. A blocked anchor shows213214 `#REF!` with the reason. `Set` refuses spilled cells; files keep only214215 the anchor.216+- **Linked regions.** A sheet may hold regions whose rows come from217+ outside the workbook (`linked.go`, `live.go`): an anchor, the source's218+ definition (path, format, table or query, the rows to keep), and the219+ cells it has written, spilled cells as an array's are, so they are220+ refused by `Set`, constants to formulas, values to copies and exports,221+ and never saved (the file keeps the definition, `linkfile.go`). Rows222+ arrive as live operations, the change stream below; placing a cell in223+ a region (a format, undo) keeps the region's value. The definitions224+ are replaced whole on every change, so undo steps keep them; a region225+ undo puts back, or one moved by inserted lines, is emptied and marked226+ stale, and the UI reads its source again.215227 - **Rules.** A sheet's conditional formats and data validation216228 (`rules.go`, `condfmt.go`, `validation.go`) are lists of rules on217229 ranges, replaced whole on every change so undo steps keep them as they@@ -230,6 +242,30 @@ style.
230242 answers from a cache and queues new questions, and the UI sends them as231243 background commands.232244 245+### The change stream246+247+What reads a linked region's source hands the workbook `sheet.LiveOp`s,248+each applied at once by `Workbook.ApplyLive`:249+250+| Field | Holds |251+|---|---|252+| `Link` | the region, by ID |253+| `At` | when the source changed |254+| `Reset` | the rows replace every row under the header; otherwise they follow the last |255+| `Header` | the first row, when it is new or changed |256+| `Rows` | data rows: values with the formats they arrived in (`LiveCell`) |257+| `Err`, `Note` | why the source can't be read, what it left out |258+259+An op is applied outside the undo history, as a spill is written: the260+window drops the oldest rows (read back from the region's cells and261+written again one row up), only the cells that changed are written, and262+what reads them recalculates, pivots and spills included; inside an open263+step (a macro run) they recalculate when it ends. An op carries whole264+values and names nothing but its region, so applying the same ops in the265+same order to the same workbook makes the same cells: it is what a266+session following another's over `012 serve` would be sent, beside the267+steps `Batch` records. `OnLive` times each op for telemetry.268+233269 ### Functions234270 235271 `internal/functions` holds the function library: the `FuncDef` table@@ -376,6 +412,7 @@ draw. The components:
376412 | sheet tabs | `tabstrip.Strip` | where each sheet was left, the tab strip's scroll and layout (package `tabstrip`); what clicks on it do (`tabstrip.go`) |377413 | mouse | `mouseState` | drags, hover, double clicks, the fill handle (`mouse.go`, `fill.go`) |378414 | import | `transfer.Transfer`, `pipeState` | the import in progress, its progress display and cancelling (package `transfer`); choosing and placing imports (`transfer.go`, `importplace.go`); standard input read as a sheet, and what a pipeline gets on quitting (`pipe.go`) |415+| linked files | `followState` | a source (`live.Source`) for each linked region, polled one poll at a time on a command every `live.Interval` and reconciled with the workbook after every update, so undo, opening a file and unlinking need nothing of their own; trust in files outside a workbook's folder (`follow.go`); Data > Linked file and Import's Follow the file (`linked.go`) |379416 | macros | `recorder`, `macroState` | a recording in progress (`macrorec.go`); a macro running, trust in the file's macros (`macrorun.go`); what scripts act on (`macrohost.go`, `macrohostnav.go`); Data > Macros and the manager (`macro.go`, `macromanage.go`) |380417 | others | `clipboard`, `trace`, `chartState`, `jevRunner`, `terminal`, `session` | what Ctrl+V pastes, a trace being shown, chart commands' target, JEV questions in flight, what the terminal supports and the chart images sent to it, what outlasts the file open (the `:` history, whether keys can be held: `keyboard.go`) |381418 @@ -470,7 +507,16 @@ counts what didn't fit. They stream: CSV and TSV are read record by
470507 record, NUON and JSON a row at a time (`nuon.Reader`, which yields rows471508 from a pipe as they arrive), Parquet a batch of rows at a time, SQLite a row at a time, and472509 Parquet files and SQLite tables stop at the sheet's last row, taking the473-number of rows left out from the file. XLSX is read by 012's own510+number of rows left out from the file. A `Tail` reads CSV, TSV, JSON and511+NUON as they arrive in pieces, for a file followed as it grows: the512+importers' readers run on a goroutine of its own over a pipe that waits513+for the next piece, so a record cut off at the end of one waits for the514+rest, and rows come out typed as an import types them (`tail.go`).515+`internal/live` follows files with `os.Stat` alone, reading what a516+growing file gained through a `Tail`, and any other format again whole,517+through its importer, once it holds still (`live.File`); a `live.Source`518+is anything that yields a table's rows over time, which a nushell region519+following a pipeline would be too. XLSX is read by 012's own474520 SpreadsheetML reader on `archive/zip` and `encoding/xml`: the workbook,475521 shared strings (kept end to end in one buffer) and styles first, then476522 each worksheet a token at a time, a row at a time, expanding shared@@ -38,6 +38,7 @@ it lags, and past a second it stalls.
3838 | Find, filter, sort, fill | Filter or sort 8191 rows: 1 to 32 ms; find over 213 k cells: 38 ms; fill 8192 rows: 4 ms | Replace all over 213 k cells: 280 ms | | Per-cell string conversion and regexp |3939 | SSH sessions (`012 serve`) | 50 sessions typing at once: frames within one frame interval (p95 8.7 ms), 1.3 MiB per session plus its sheets (upper bound) | | More than `--max-sessions` (8 by default): turned away | Bubble Tea's 120 fps pacing; per session, the terminal's cell buffers |4040 | Pivot tables | A pivot over 8191 rows recomputes after an edit to its source in 0.8 to 2.5 ms | | Results past the grid (the pivot shows #REF!) | Grouping the source rows |41+| Following files | A log growing by 10,000 rows a second, formulas over it: each update (about 2,500 rows, four times a second) applied in 7 to 10 ms keeping every row, 1.6 to 2.2 ms keeping the last 1000, 14 ms the last 10,000; frames 2 to 3 ms at 200 x 60 | | Rows past `max-cells`: left out, with a note | Writing the rows and recalculating what reads them; a window rewrites its rows on every update |4142 | Arrays and spills | FILTER, SORT or UNIQUE over 8192 rows: about 0.1 ms each per edit; 1000 of them spilling 516 k cells, an edit recomputing 500: 52 ms | Full recalculation of those 1000: 106 ms | An array past `maxArray` values (2,097,152) stored: `#VALUE!`; a spill past the sheet's edge or `max-cells` cells: `#REF!` | Computing each array's values; writing only the spilled cells that changed |4243 4344 ## Sheet size@@ -347,6 +348,30 @@ rows of more than 16,384 cells; part names that are absolute, climb with
347348 `..` or hold a backslash. encoding/xml expands no external or declared348349 entities, so entity bombs fail as unknown entities.349350 351+## Following files352+353+`BenchmarkFollow` (`internal/ui/followstress_test.go`) follows a CSV354+log while another goroutine appends 10,000 rows a second to it for three355+seconds, with `=SUM(D:D)`, `=COUNTIF(B:B,"error")` and an AVERAGE over356+the region, polling as the UI does and drawing a 200 x 60 frame after357+each update:358+359+| Rows kept | Update applied (slowest) | Frame (slowest) | Rows a second taken in |360+|---|---|---|---|361+| Every row (30,000) | 7.5 to 10.5 ms | 2.3 to 2.7 ms | 8,100 to 8,300 |362+| The last 1000 | 1.6 to 2.2 ms | 2.9 ms | 8,300 to 8,400 |363+| The last 10,000 | 13.7 to 14.2 ms | 2.2 to 2.8 ms | 8,000 to 8,100 |364+365+(Rows a second counts to the last row read, after the writer stopped;366+the writer itself manages a little under 10,000 with its pauses.) An367+update costs the rows it brings, and with a window the rows it keeps:368+the window's rows move up one row each on every update, so a window of369+100,000 rows under a fast log would take about 140 ms an update. A370+followed file costs one `os.Stat` four times a second while it doesn't371+change; a growing file is read a megabyte a poll at most, so a large372+file loads over several frames. Rows are held once, in the region's373+cells; a window reads the rows it keeps back from them.374+350375 ## Undo351376 352377 Each step keeps the cells it changed as they were, in the form the@@ -92,6 +92,7 @@ there.
9292 | Warnings | Text on the context line or status line (`Invalid: ...`, `Circular reference`) |9393 | Entries failing validation | A dotted underline, and `Invalid:` with the rule on the context line |9494 | Values an array spilled | Italic, and `Spilled from B2` on the context line |95+| Linked files' rows, following, paused or failing | Italic; `●`, `‖` or `!` beside the tab's name and on the context line, with the state in words (`Following`, `Paused`, the error) |9596 | Pivot table results | Their headings (`SUM of Units`, `Grand Total`), and a note in words when an edit is refused |9697 | Protected ranges | A question in words before an edit (`A1:B2 is protected.`) |9798 | Notes | A `▝` in the cell's top-right corner |@@ -25,11 +25,13 @@ it back in its format.
2525 | SQLite | Pick a table or view, or type a query; a header row names the columns | The sheet shown or the selection as a table, first row as column names; a table of that name is replaced |2626 | Parquet | Every column, with dates and timestamps; lists joined with commas | |2727 | Lotus 1-2-3 `.wk1`, `.wks` | Numbers, labels with their alignment, formats, column widths, formulas translated (references, operators, `@SUM`, `@AVG`, `@IF`, `@ROUND`, `@PMT` and 60 more) or kept as values | |28-| JSON `.json` | A list of records (or records one after another, as NDJSON): keys become the header row, numbers, booleans and text keep their types, nested lists and records are their text | The sheet shown as a list of records named by its first row, numbers as numbers |28+| JSON `.json`, `.ndjson`, `.jsonl` | A list of records (or records one after another, as NDJSON): keys become the header row, numbers, booleans and text keep their types, nested lists and records are their text | The sheet shown as a list of records named by its first row, numbers as numbers |2929 | Nushell `.nuon` | A nushell table with its types: file sizes, durations and dates become numbers in the Size, Duration and Date time formats; see [Nushell](../terminal/nushell.md#types) | The sheet shown as a nushell table, first row as column names, types kept by the cells' formats |3030 3131 Formats other than XLSX import as one sheet named after the file (or the32-SQLite table). A table can also come in on standard input, and go back32+SQLite table). A file can also be followed instead, its table in a33+linked region that takes in new rows as the file grows or is rewritten:34+see [Following files](following.md). A table can also come in on standard input, and go back3335 out on standard output: see [Nushell and pipelines](../terminal/nushell.md).3436 3537 File > Import asks where the data goes, as Sheets' Import location does@@ -64,4 +66,5 @@ own 8,192 by 256. See [Bounds of support](../contributing/limits.md#imports) for
6466 |---|---|6567 | [Excel files](excel.md) | What comes in from and goes out to `.xlsx` |6668 | [Saving](saving.md) | Atomic saves, overwrite checks, unsaved work under `012 serve` |69+| [Following files](following.md) | Linked regions that follow a file as it grows or is rewritten, like `tail -f` |6770 | [The .012 format](format.md) | The JSON format, its versions and fields |@@ -0,0 +1,122 @@
1+---2+title: "Following files"3+sidebar_position: 54+---5+6+# Following files7+8+A file can be linked rather than imported: its table shows in a linked9+region of a sheet that follows the file, as `tail -f` follows a log. Rows10+appended to a CSV, TSV, JSON lines or NUON file come in as they're11+written; any file 012 imports (XLSX, SQLite, Parquet too) is read again12+when it's rewritten. Formulas, charts, pivot tables and rules over the13+region recalculate as rows arrive.14+15+## Linking a file16+17+- **Data > Linked file > Link a table** links a file at the active cell,18+ which must be empty. The picker lists the files around, as File >19+ Import's does, or takes a path.20+- **File > Import**, then **Follow the file** among the import locations,21+ follows the file in a new sheet after the one shown, named after the22+ file. On a new, empty spreadsheet File > Import opens the file instead,23+ as it always does; link it with Data > Linked file there.24+25+For a SQLite database with several tables, a picker asks which one. Then26+a question on the context line asks how many rows to keep:27+28+| Key | Keeps |29+|---|---|30+| Enter | Every row, up to `max-cells` ([Configuration](../reference/config.md#max-cells)); past it, the rows that don't fit are left out and the context line says so |31+| L | The last rows, as many as typed (1,000 to start with): older rows drop as new ones arrive. The first row, the file's header, stays |32+33+## The region34+35+The first row of the region is the file's first row (a NUON or JSON36+file's column names, as an import names them); the rows under it are the37+file's rows, typed as an import types them. The region is the file's:38+39+- its cells are read-only; typing into one, pasting over it or clearing40+ it says so. Formatting them (bold, number formats, colors) is fine and41+ stays on the cell as rows go by;42+- its values show in italics, as an array's spilled values do, and43+ copying them copies values;44+- it grows to the right and down as the file does. Where it would write45+ over a cell holding something, it stops and says which cell; clear the46+ cell and use Data > Linked file > Read again;47+- inserting or deleting rows or columns before it moves it, and it reads48+ the file again there; deleting its first cell's row or column unlinks49+ it.50+51+The sheet's tab carries a mark, and the context line says what the52+region under the pointer is doing:53+54+| Mark | Context line |55+|---|---|56+| `●` | `● Following app.csv 1,204 rows, updated 09:14:17`; with a window, `(the last 1,000)` and how many older rows dropped |57+| `‖` | `‖ Paused app.csv ...`: Data > Linked file > Follow is off |58+| `!` | `! app.csv: the file isn't there`, or why the file can't be read or its rows written. A region with nothing to show has `#REF!` in its first cell |59+60+## How a file is followed61+62+012 checks each linked file four times a second with the operating63+system's file information alone (size, modification time, identity), so64+a file that doesn't change costs one check.65+66+- **Text tables that grow** (CSV, TSV, `.json`, `.ndjson`, `.jsonl`,67+ NUON): the bytes appended since the last check are read and their rows68+ added. A line still being written (or a NUON value, or a quoted CSV69+ field spanning lines) waits until it's whole. A file shorter than what70+ was read (truncated), another file at the same path (rotated, as71+ `logrotate` does), or one whose last bytes read differ (rewritten in72+ place) is read again from its start. A large file comes in a megabyte at a time, the screen73+ staying live between.74+- **Files that are rewritten** (XLSX, SQLite, Parquet, and `.wk1`): the75+ file is read again whole when its size or modification time changes,76+ once they've held still for 0.3 s, so a file being written isn't read77+ half done. An XLSX file's sheet shown when it was saved is the one read.78+79+Rows arrive outside the undo history, as an array's spilled values do:80+undo takes back edits, not the file's rows, and rows arriving don't mark81+the spreadsheet modified.82+83+## Pause, read again, unlink84+85+Data > Linked file acts on the region under the pointer, or the sheet's86+only one:87+88+| Command | Does |89+|---|---|90+| Follow | Pauses following (the region keeps its rows) or follows again, catching up |91+| Read again | Reads the file again, whole |92+| Rows to keep | Asks again how many rows to keep; the file is read again |93+| Unlink | Keeps the rows as ordinary cells, values with their formats, and stops following; undo links the file again |94+95+## Saving and opening96+97+A spreadsheet keeps what each region reads, never its rows (see98+[The .012 format](format.md#linked-files)): the file's path relative to99+the spreadsheet's folder (absolute when the file is elsewhere and was100+linked by an absolute path), its format, the SQLite table or query, and101+how many rows to keep. Saving in another folder keeps the paths naming102+the same files. Opening the spreadsheet reads the files again and follows103+them; a file that isn't there shows in its region, and is read once it104+is.105+106+Under `012 serve`, linked files resolve inside the served folder, as107+every file name a session types does ([Serving over SSH](../terminal/ssh.md));108+a spreadsheet linking a file outside it shows why in the region.109+110+## Files from elsewhere111+112+Following reads files, so a spreadsheet made on another computer asks113+once before following files outside its own folder (an absolute path,114+or one through `..`): the context line names them, Enter follows them115+and records this computer as trusted in the spreadsheet (`linkOrigin`,116+saved with it), Esc leaves their regions empty. Files in the spreadsheet's117+folder or below, and spreadsheets linked on this computer, never ask.118+Macros are trusted the same way, separately119+([Macros](../sheets/macros.md#macros-from-other-computers)).120+121+What following costs, measured, is in122+[Bounds of support](../contributing/limits.md#following-files).@@ -103,8 +103,10 @@ open, and in their docs: sorting by several columns
103103 ([Freeze, sort and filter](../sheets/sort-filter.md#sort)), conditional formatting and data validation104104 ([rules panel](../sheets/rules.md#conditional-formatting)), pivot tables105105 ([pivot editor](../sheets/pivots.md)), protected ranges106-([Notes and protection](../sheets/notes-protection.md#protected-sheets-and-ranges)) and charts107-([Charts](../sheets/charts.md)). In Data > Named ranges, Enter goes to a106+([Notes and protection](../sheets/notes-protection.md#protected-sheets-and-ranges)), charts107+([Charts](../sheets/charts.md)) and linked files (Data > Linked file, where108+Enter keeps every row and L the last ones:109+[Following files](../files/following.md)). In Data > Named ranges, Enter goes to a108110 range, F2 renames or repoints it and Ctrl+D deletes it.109111 110112 ## Files, menus and help