cat / 012
clone

a spreadsheet for the terminal: lotus 1-2-3 looks, google sheets keys and formulas, one go binary

main
012 / docs / files / sources.md
sources.md 10.4K rendered
1---2title: "Linked sources"3sidebar_position: 84---56# Linked sources78A Parquet file, or a SQLite table or query, too big for any grid links9as a source: a read-only table on a tab of its own, read in place10rather than imported. The tab shows a window of the rows as it11scrolls, with the row numbers and the scrollbar over every row, and12formulas and pivot tables read the whole source by streaming it. A13source of ten million rows opens at once and takes a few MB.1415![A source of ten million rows on its tab, sorted by amount, the scrollbar's thumb halfway down](../media/source-tab-dark.png#gh-dark-mode-only)16![A source of ten million rows on its tab, sorted by amount, the scrollbar's thumb halfway down](../media/source-tab-light.png#gh-light-mode-only)1718## Linking a source1920![Ten million rows of Parquet linked as a source, sorted by amount in the background, then read whole by formulas on Sheet1: a count with thousands separators, an AVERAGEIFS, the last day as a date and a MEDIAN](../media/sources.gif)2122**Data > Linked file > Link a source** lists the Parquet files and23SQLite databases in the folder, or takes a path. For a database a24second list asks which table or view to read, or **A query…** to read25what a `SELECT` returns.2627The source gets a tab after the one shown, named after the file28(`sales` for `sales.parquet`), with `▦` before its name. The source is29a [table](../sheets/tables.md) of that name, which formulas on any sheet30read as a table's:3132| Formula | Reads |33|---|---|34| `=SUM(sales[amount])` | a column's values, every row |35| `=ROWS(sales)` | the rows under the header |36| `=COUNTA(nu.sales)` | the whole table, header row included |37| `=sales!C5` | one cell: row 1 is the header, so row 5 is the source's fourth row |3839Renaming the tab keeps the table's name. Undo takes the link back. A40source's rows may run past the grid's last row, 1,048,576: its table's41name reaches them all, where a reference typed as `C5` stops at the42grid's edge.4344## The tab4546Row 1 is the header, the columns' names, and stays put as the rows47scroll under it. The rows are numbered as the tab's rows, so the48source's first row is row 2, and `sales!C2` is its first amount. The49scrollbar at the right edge is as long as the share of the rows shown,50and as far down as they are.5152| Key or mouse | Does |53|---|---|54| Arrows, Tab | Move the active cell |55| Page Up, Page Down, the wheel | Scroll |56| Home, End | The first or last column |57| Ctrl+Up, Ctrl+Down | The first or last row |58| Ctrl+Home, Ctrl+End | The first cell, the last |59| A click on the scrollbar, or dragging its thumb | Jumps there |60| Ctrl+G, F5 | Goes to a row, or a cell such as `C5000000` |61| Ctrl+C | Copies the active cell |6263The rows come from the file a page (128 rows) at a time, in the64background, the pages either side of the window read ahead, so65scrolling finds them there. A row on its way shows `…`. The name box66shows the active cell as the tab names it, and the formula bar its67value; the context line says what the source is:68`▦ sales.parquet 10,000,000 rows, 4 columns`. The mode is `SOURCE`,69and typing, pasting or formatting there says the tab is read-only.7071## Sorting and filtering7273The tab's rows sort and filter as a sheet's do, without changing the74file:7576- **Data > Sort sheet A to Z** (or **Z to A**) sorts by the active77 column; blanks go last either way.78- **Data > Create a filter** or **Filter by column** asks for a79 condition on the active column: the conditions of a sheet's80 [filter](../sheets/sort-filter.md#filter). There is no list of values81 to tick: a source's are too many to list. Each column's condition82 applies at once.83- **Data > Remove filter** drops the conditions, and **Data > Linked84 file > Original order** shows the source's own rows in its own order.8586Each is an undo step and is saved with the workbook. Formulas read the87source in its own order, every row, as formulas read the rows a sheet's88filter hides.8990Ordering reads the whole source once, in the background, while the91context line says `sorting and filtering…`; the tab then shows the92first of the rows. SQLite does the work: the sort and the conditions93become SQL, the rows' order is kept in a temporary database beside the94system's temporary files, and the database itself is never written. A95Parquet file is streamed: a filter is one pass over the columns it96tests, and a sort writes sorted runs to temporary files and merges97them, so memory stays a few MB and up to about 100 MB while sorting,98whatever the size. Either way a page anywhere in the ordered rows99costs what one at the top does:100101```mermaid102flowchart TD103 order[a sort or a filter] -->|once, in the background| view[the rows' order: SQL into a temporary table, or streamed into temporary files]104 scroll[the tab scrolls] --> want[the pages around the window]105 view --> want106 want -->|a page not read| read[read 128 rows at their positions]107 read --> kept[kept, the least used let go past 48 pages]108 kept --> frame[the next frame draws them]109```110111In SQLite a text condition compares the text as SQLite writes it, so a112number reads as `1e+20` where the sheet shows `1E+20`, a date held as113text (`2026-09-26`) sorts and compares as that text, and letters beyond114ASCII are compared by case.115116## Formulas over a source117118![Formulas on Sheet1 over two million trips: a count with thousands separators, sums of fares with their two decimals, the last day as a date, and the median fare, the pointer on it](../media/source-formulas-dark.png#gh-dark-mode-only)119![Formulas on Sheet1 over two million trips: a count with thousands separators, sums of fares with their two decimals, the last day as a date, and the median fare, the pointer on it](../media/source-formulas-light.png#gh-light-mode-only)120121A function given a source's range is worked out in the background, by122reading the source; its cell shows `Loading…` until then. The answer123is kept until the file changes, so another formula asking the same, or124the same formula recalculating, costs nothing.125126These read a source of any size, holding only what they compute:127128| Kind | Functions |129|---|---|130| Aggregates | SUM, AVERAGE, COUNT, COUNTA, MIN, MAX, PRODUCT |131| Criteria | COUNTIF, COUNTIFS, SUMIF, SUMIFS, AVERAGEIF, AVERAGEIFS, COUNTBLANK, SUMPRODUCT |132| Lookups | MATCH, XLOOKUP, VLOOKUP, HLOOKUP, INDEX, ROWS, COLUMNS |133| Ranks | RANK, RANK.EQ |134| Order statistics of one range | MEDIAN, PERCENTILE, PERCENTILE.INC, PERCENTILE.EXC, QUARTILE, QUARTILE.INC, QUARTILE.EXC, LARGE, SMALL, MODE, MODE.SNGL |135136The first four read the source once. An order statistic sorts the137range's numbers on disk, beside the system's temporary files, and138reads them back in order once: a MEDIAN of ten million rows takes139about two seconds, 50 MB of memory and 160 MB of disk while it runs140([Bounds of support](../contributing/limits.md#linked-sources)). Its141range is one argument, its others single values:142`MEDIAN(sales[amount])` and `PERCENTILE(sales[amount],0.9)` stream,143where `MEDIAN(sales[amount],sales[tax])` holds what it reads.144145A result keeps what its column means, as a formula over a sheet takes146its inputs' format: SUM, AVERAGE, MIN, MAX and the lookups show in the147format the column's type gives it. In Parquet that's a DECIMAL with148its scale's decimals, a DATE as a date, a TIMESTAMP as a date and149time, a TIME as a time and an INTERVAL as a duration; in SQLite the150declared type: DATE, DATETIME or TIMESTAMP, TIME, DECIMAL(10,2) or151NUMERIC(10,2) with its decimals, and MONEY as currency. A count over a152source (COUNT, COUNTA, COUNTIF, COUNTIFS, COUNTBLANK, ROWS, COLUMNS)153shows as a whole number with the locale's thousands separators:154`2,000,000`. A format given to the cell wins over either.155156Any other function (STDEV, SORT, FILTER, UNIQUE, TEXTJOIN) holds what157it reads, and is given a source's range as long as that holds no more158than `max-cells` cells ([Configuration](../reference/config.md#max-cells));159past it the formula shows `#VALUE!`. So does an operator over a160source's range (`=SUM(sales[amount]*2)`) and a function whose other161arguments are ranges of a sheet or arrays:162`SUMPRODUCT((sales[cat]="north")*sales[amount])` computes an array as163long as the source, where `SUMIFS(sales[amount],sales[cat],"north")`164streams. The context line says how far to raise `max-cells` for it,165and F1 the rest ([Values and errors](../formulas/README.md#values-and-errors)):166167![The fares' STDEV past max-cells: F1 shows its whole explanation in a box beside the cell, to raise max-cells to 2,000,000 or use a function that streams](../media/source-formulas-why-dark.png#gh-dark-mode-only)168![The fares' STDEV past max-cells: F1 shows its whole explanation in a box beside the cell, to raise max-cells to 2,000,000 or use a function that streams](../media/source-formulas-why-light.png#gh-light-mode-only)169170## Pivot tables171172**Data > Pivot table** on a source's tab makes a173[pivot table](../sheets/pivots.md) over the whole source, and **Data >174Frequency table (column stats)** one of the active column. The source's rows are175grouped in the background, `Loading…` showing in the pivot table's176first cell until they are, and again whenever the file changes. A177filter of a pivot table over a source takes a condition, not values178ticked.179180## When the file changes181182012 looks at each source's file four times a second, with the183operating system's file information alone. Once the file has changed184and held still for 0.3 s, the source is read again: the tab, and every185formula and pivot table reading it. **Data > Linked file > Read the186source again** does the same at once. A file that isn't there shows187`!` after the tab's name and the reason on the context line, and the188formulas reading it show `#REF!` with it.189190## Saving and sharing191192The workbook keeps the link, never the rows: the file's path relative193to the workbook's folder, the table or query, and the tab's order194([The .012 format](format.md#linked-sources)). Opening the workbook195reads the sources again. A workbook from another computer asks before196reading files outside its own folder, as for197[linked files](following.md#files-from-elsewhere).198199`012 get`, `recalc` and `export` ([Scripts](scripts.md)) read the200sources in the workbook's folder or below, to the end, before they201answer. Under `012 serve` sources are read in the served folder, and in202a [shared workbook](../terminal/ssh.md#sharing-a-workbook) the room reads them, as it follows linked203files, while each participant scrolls a tab of their own.204205What sources cost, measured on ten million rows, is in206[Bounds of support](../contributing/limits.md#linked-sources).